case procountor: zephyr test tool deployment

Zephyr Test Tool Deployment | Procountor Knowit Oy, Jussi Rautio 6.10.2015

PAGE 2 [CONFIDENTIAL] THIS IS KNOWIT 6.10.2015 www.knowit.fi

6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.0 3

Senior Quality Consultant, Service Productization Responsible, Service Manager, Test Tool Technical support, Test Tool Deployments and Migrations…

• Speaks, train, coach and mentor about

• How to take new testing tool in use and best practices.

• User trainings for testers, test managers, product owners and quality managers

• Metrics and KPI`s benchmarking and design

• Data Quality

• Process enhancements

• Etc.

JUSSI RAUTIO, KNOWIT

www.knowit.fi

KNOWIT

Scandinavian company that is listed on Nordic

Exchange Stockholm

Knowit offering in Finland:

• Test and quality management

• Solutions for digital business and information

management

• Development and maintenance of client-

specific applications

6.10.2015 © Copyright Knowit Oy 2015 | Confidential | Version 1.0 4 www.knowit.fi

02.09.15 [CONFIDENTIAL] THIS IS KNOWIT PAGE 5

• Leading provider of cloud accounting software

• Used by over 10 000 companies and over 400

accounting offices

• Available in Finland, Norway, Sweden and

Denmark

02.09.15 [CONFIDENTIAL] THIS IS KNOWIT PAGE 6

PROCOUNTOR ATLASSIAN ENVIRONMENT


www.ambientia.net 7

Crowd

JIRA Fisheye Confluence

Refi

ne

d w

iki

JIR

A a

gile

Zep

hyr

for

JIR

A

Tim

es

hee

t re

po

rts

Cru

cib

le

Confluence • Social Intranet

• Knowledge Management

JIRA • Requirement management

• Product development (scrum)

• Test management

Fisheye/Crucible • Source code browsing

• Code review

Crowd • User management

*500

*50 *100 *250

Hosted by: www.ambientia.net

www.knowit.fi

10/6/2015 © Copyright Knowit Oy 2014 | Confidential | Version 1.0 8

Problem Description

www.knowit.fi

PROCOUNTOR NEED

• Procountor is a rapidly growing firm

• New people are hired constantly

• Testing tool and processes need to be updated to meet the new demands

• Knowit was asked to enhance Procountor testing


TEST PROCESS ANALYSIS

Knowit test process analysis for Procountor

Means of analysis were:

• Workshops

• Interviews

Two weeks after the release development teams are very busy on fixing bugs. Why?

• Too tight schedule

• Fixed content

• Not enough time for regression testing


TEST PROCESS ANALYSIS CONCLUSION

Insufficient regression testing time ->

Whole team approach should be introduced ->

Testing tasks are in the different testing tool ->

Better tool to enforce task sharing ->

Jira ensures already a common way of working ->

Test tool Zephyr was chosen to be used in regression testing

because of seamless integration to Jira among other things


© Copyright Knowit Oy 2014 | Confidential | Version 1.0 12

Introducing the Test Tool

www.knowit.fi

TEST TOOL DEPLOYMENT PRINCIPLES

Fail fast

• If tool is not suited for you, don’t waste your time, look other way

Start light

• Put only necessary test data in to system in pilot phase

• Do only minimum tweaking to the tool at the beginning

Embrace the change

• Give tool a chance


www.knowit.fi

Enhanced

Quality

and

Effectivity

Create a usage model for test tool

Test tool in pilot use for regression testing

Usage model tweaking

Test tool in use for wider testing

Test tool in use internationally

Continuous improvement activities

Integrations to other systems

Test tool used through organization if Finland

Virtualization & automation

GENERAL ABOUT ZEPHYR

• Brings test management and execution functionalities to Jira

New issue type Test

• Possibility to have as an add on for Jira or as an own product

• First Version released in the beginning of 2012

• Latest version 2.6.0 Released on 09.2015

Characteristics

• Easy to use for Jira users

• Customizable

• No test case version control

• If lots of add-on’s are in use the update process of Jira and Zephyr becomes challenging

• No folder structure


THINGS TO DO BEFORE DEPLOYMENT 1/3

• Fields for tests must be agreed

• default fields can be used at start

• Screens must be agreed

• Tests workflow should be done when testing process with the new tool is mature enough

• User groups and rights should be put in order

• For example who can modify test executions should be restricted

• Procountor has one tester per group so there is no need to restrict user rights



Labels must be decided

• Label is users self chosen tag for tests

• Labels should be agreed so that their amount would not explode

• Labels could be for example: Functional testing, Regression testing, Usability, Load…



Components must be decided

• Components are user defined parts of software

• For example: Accounting, User Management, Billing..


TEST STATUSES

One part of test process definition is specifying test statuses

• Statuses can be modified based on test process

Default test statuses of Zephyr are

• PASS

• FAIL

• WIP

• BLOCKED

• UNEXECUTED


LINKING TESTS TO USER STORIES

Tests can be linked to User Stories

• “Is blocked by” -link statuses are NOT affecting if the test can be run or not

Linking should be done always so changes in User Story can be followed easily

• Changes in user story are NOT sending signal for re-testing


TEST CASE DIVISION

1. By Project

• Project is a way to division test cases in a very high level. Procountor has only one Project

2. By Version

• Fix Versions –field can be used to division tests. Test Summary Screen shows tests by version

automatically

3. By Component

• Test can be attached to a component (multivalue field.) Test Summary shows also these.

4. By Label

• Anyone can create new labels so this should be agreed in organization to keep it controlled


CYCLES – TEST ROUNDS

Tests are executed in Cycles and one test can be in many Cycle

Tests can be taken to test Cycles either by:

• One at a time

• By Filter

• By Component

• By Label


NAMING AND DIVISION OF CYCLES

Cycle name should present their division and support reporting.

Cycle names can be for example:

• Regression test for version xx (Procountor)

• Sprint

• Week

• Tester

• Group

• Combination of above

…


TEST TOOL CONCEPT OWNER

Someone from the organization should be named as a

test tool concept owner after the test tool deployment

Test tool concept owner tasks:

• Follow test tool efficiency continuously

• Follow test tool development roadmap

• Introduce new test tool features to organization

• Gather user requests for test tool customization and integrations


MEASURING OF TEST TOOL EFFICIENCY

Test tool efficiency should be followed in during the long period of time

Has the quality of the product enhanced?

• Number of bugs found in production

• Customer satisfaction

Has the efficiency increased?

• Is the tool really used?

• What is the gut feeling? Has the tool helped testing? Is regression testing easier now?

• How wide has the tool usage spread?

• Has the hurry after the release decreased?

• What is the usability of the tool?


FUTURE SCENARIO FOR PROCOUNTOR

Testing during development could be done with exploratory testing

• Link exploratory testing session to User Story

• Leaves a trail what has been already tested


• Actor: intended user of the system

• Purpose: the theme of the charter including what particular objective the actor wants to

achieve, i.e., the test conditions

• Setup: what needs to be in place in order to start the test execution

• Priority: relative importance of this charter, based on the priority of the associated user story or

the risk level

• Reference: specifications (e.g., user story), risks, or other information sources

• Data: whatever data is needed to carry out the charter

• Activities: a list of ideas of what the actor may want to do with the system and what would be

interesting to test (both positive and negative tests)

• Oracle notes: how to evaluate the product to determine correct results

• Variations: alternative actions and evaluations to complement the ideas described under

activities

www.knowit.fi

EXPLORATORY TESTING – TEST SESSION


Session based testing aims to combine accountability and exploratory testing

Test sessions:

• Based on charter

• Uninterrupted session from 60 to 120 minutes

• Evaluation of the functionality or characteristics

• Each session is reported in detail

Test Design

Test Execution

Test Results

Learn Test Charter

Test Session

Test Design

Test Execution

Test Results

Learn Test Charter

Test Session

Test Design

Test Execution

Test Results

Learn Test Charter

Test Session

Session-Based Test Management

www.knowit.fi

TOOLS FOR EXPLORATORY TESTING

Record Steps to Reproduce a Problem

• Windows Out Of the Box tool to take screenshots

• Creates screenshots on each click and typing and creates one file from the recording

• Free in all Windows versions

JIRA Capture

• Web –application to create issues easily

• Records Test Session length, issues and tester notes

• Needs to be purchased


THANK YOU!

QUESTIONS?

www.knowit.fi

Knowit Oy Tehtaankatu 27-29 D 00150 Helsinki tel. +358 40 3400600 www.knowit.fi

case procountor: zephyr test tool deployment

Technology