reliable relevant metrics to the right audience - manual testing whitepaper
DESCRIPTION
'What cannot be measured cannot be managed” is the guiding philosophy behind testing metrics, a phenomenon that promises to deliver business efficiencies beyond just improving quality. Measurement helps with planning, tracking and managing the software project and enables organizations to objectively assess qualityTRANSCRIPT
INDIUM SOFTWAREAn Independent Software Testing Firm
Reliable, Relevant Metrics to the Right
Audience
“Business Assurance through Quality Assurance”
© 2014 Indium Software
2
ENHANCING SOFTWARE QUALITY
2
Agenda
Background Bridging the gap Classification
Sponsors Delivery Organization Continuous Process Improvement
Benefits Conclusion
© 2014 Indium Software
3
ENHANCING SOFTWARE QUALITY
3
Purpose
Metrics provide the following three basic functions:
Capture : Capture metrics
Communicate: Communicate to the stakeholders
Improve: Improve based on the size and direction of gaps identified
© 2014 Indium Software
4
ENHANCING SOFTWARE QUALITY
4
Metrics…. A plenty !!!!
Irrelevant metrics Delivers irrelevant value to stakeholders Distracts human capital from what is
important Leads to conflict and confusion
© 2014 Indium Software
5
ENHANCING SOFTWARE QUALITY
5
Metrics – Lifecycle, Importance and Categorization
Capture
Control
Improve
How are these metrics correlated ? To whom does it add value ? Is it a one time compilation ?
Gathered metrics makes sense only to the intended audience Relevant metrics to the relevant people makes metrics a real
value addition
Sponsors
Delivery Organization
Continuous Process
Improvement Team
© 2014 Indium Software
6
ENHANCING SOFTWARE QUALITY
6
For the Sponsors…
Customer satisfaction index Quality of the service On time delivery Business requirement understanding Communication
ROI Gain percentage
Q4 Q1 Q2 Q30%
20%
40%
60%
80%
100%
80%70%
82% 86%
Customer Satisfaction Index
CSI
6 12 180
20406080
100120140160
63%
92%
135%
Return On Investment
Months
Perc
enta
ge
Tool cost $20,000 Training $5,000 Machine $3,000 Months ATDT ATAT ATMT Investment Gain ROI
$ $ $ $ $ %6 30000 5760 11520 75280 122400 62.59
12 30000 7200 14400 79600 153000 92.2118 30000 9600 19200 86800 204000 135.02
© 2014 Indium Software
7
ENHANCING SOFTWARE QUALITY
7
For the Delivery Organization
Cycle time The period required to complete one
cycle of an operation Highlights streamlined process Turnaround time to the customer
requisitions
Defect Leakage # of Defects found in the field Gives a Caution towards missed Defects
Apr May Jun Jul Aug Sep Oct0
10002000300040005000600070008000
012345678910
800 950 927
2240 1890
4290
6728
47487
89
7 7 7 7 7
Cycle Time
Executions Days
Month
Exec
ution
s
Days
0%
5%
10%6%
0% 0%
9%8%
9%
5% 5%3%
Defect Leakage
Release
Per
cen
tag
e
Reduced Cycle time for more number of executions
Defect leakage in different releases
© 2014 Indium Software
8
ENHANCING SOFTWARE QUALITY
8
For the Delivery Organization
Schedule Variance Delayed start Application Down time Error in planning test phase Logging more bugs
Defect Catch Rate Defects captured by the test cases Depicts the test case efficiency
Schedule variance = Actual days - Planned days / Planned days
Defect Catch Rate
R10.12R10.13R10.14R10.15R10.16 R11.1 R11.2 R11.3 R11.40
2
4
6
8
10
12
7 78
7 7 7 7 7 7
89
109
8 87 7 7
Schedule Variance
Planned Actual
Release
Day
s
80%
85%
90%
95%
100%
89.0%
93.0%96.0%
99.0%97.0% 96.0% 96.5% 97.5% 98.0%
Defect Catch Rate
Release
per
cen
tag
e
© 2014 Indium Software
9
ENHANCING SOFTWARE QUALITY
9
For Continuous Process Improvement team
Business process improvements can be realized through the following objectives: Increased process throughput Consistent output quality Predictable process costs Increased re-use of existing processes Reduced time of sending business information from one process to another process
Process related Metrics Schedule variance % Defect leakage Defect Catch Rate Review efficiency
Streamlined delivery in 7 days due to zero schedule variance
R10.12R10.13R10.14R10.15R10.16 R11.1 R11.2 R11.3 R11.40
2000
4000
6000
8000
0
2
4
6
8
10
12
800 950 927
2240 1890
4290
6728
47485467
7 78
7 7 7 7 7 7
89
109
8 87 7 7
Schedule Variance
Executions Planned Actual
Release
Exe
cuti
on
s
Day
s
© 2014 Indium Software
10
ENHANCING SOFTWARE QUALITY
10
For Continuous Process Improvement team
Defect Leakage Caution towards defects missed Root cause for the defects
Defect Catch Rate Improved test cases Increased test coverage
0%
5%
10%
6%
0% 0%
9%8%
9%
5% 5%
3%2%
0%
Defect Leakage
Release
Perc
enta
ge
Defect leakage percentage – Downward trend
Defect Catch Rate – Upward trend
84%86%88%90%92%94%96%98%
100%
89.0%
93.0%
96.0%
99.0%97.0%96.0%96.5%97.5%98.0%98.0%99.0%
Defect Catch Rate
Release
perc
enta
ge
© 2014 Indium Software
11
ENHANCING SOFTWARE QUALITY
11
Benefits of categorization
Relevance Relationship between the data points Cause and effect correlation between the observed data points Pointers to future planning and continuous improvements
© 2014 Indium Software
12
ENHANCING SOFTWARE QUALITY
12
Conclusion
Analytics and quantitative measurement gains credence in the realm of quality management specifically around software testing
The value of the metrics gathered makes sense only if it serves the intended audience Classification of metrics relevant to the Sponsors, Delivery Organization and Continuous Process
Improvement team An extensive collation of historical testing data in a defined framework
© 2014 Indium Software
13
ENHANCING SOFTWARE QUALITY
13
Thank You
USA: [email protected]
EMEA & APAC: [email protected]@indiumsoft.com
INDIA: [email protected]
SunnyvaleSuite 210, 1250 Oakmead ParkwaySunnyvale, CA – 94085.Phone: +1(408) 501-8844Fax: +1(408) 501-8808
Atlanta Crown Office Suites1870 The Exchange, Suite 100 Atlanta, GA 30339Phone: +1 (678) 608-0809
New DelhiF-1/5 Okhla Industrial Area Phase-I, New Delhi – 110 020.Phone: +91-11-6613 0400
BengaluruSRS Plaza, No. 246, 2nd & 3rd Floor13th Cross, Indira Nagar 2nd StageBengaluru – 560 038.Phone: +91-80-4914 7777
ChennaiVDS House41, 2nd Floor, Cathedral RoadChennai – 600 086.Phone: +91-44-2811 6329 / 6330Fax: +91-44-4210 4033