5 Core Metrics to GuideTesting in Your Endgames
Bob Galen
RGalen Consulting Group, LLC
www.rgalen.com [email protected]
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 2
Outline
Notion of QA teams as GUIDES not reporters
Metrics Analysis Question (MAQ) development
5 Key Metrics Found vs. Fixed
High Priority
Project Keywords
Defect Transition Progress
Product Functional Area Distribution
with MAQ, Guide Points and PM Guidance
Wrap-up and Q&A
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 3
Introduction
We (Testers) need to adjust from a Detecting
Reporting
“Safety net” or “Gating” perspective or role
To one including all of the above, but more soserving as a project Guide. Providing more - Deep data insights and recommendations
Bottleneck and efficiency improvement guidance
Risk detection and resource application guidance
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 4
Metrics Analysis QuestionsOrigin
STQE May/June 2001 article by Anna Allison –Meaningful Metrics
Looking for the meaning behind the metrics orproject data
Determining a list of questions to ask whenviewing or observing project data trends
Can simply ask. Increase your curiosity and giveyourself a license to inquire.
Always looking to provide higher level guidanceto the project team
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 5
Metrics Analysis QuestionsExamples
1. Has there been a change in QA personnelassignments that could account for fewer bugsbeing found in the last cycle?
Vacations, time-off, illness?
Another project starting up?
2. We’ve seen a drastic increase in priority 1defects in this last release. Did the developersdo something?
Add a new component, fix some related defects?
Are we testing something for the first time?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 6
Metrics Analysis QuestionsExamples – cont.
3. What kinds of bugs are being found?
Severity mix? Priority mix?
What is the trending? Did we expect this?
Are these only code bugs or other things?Relationships in counts and trending?
4. Is the software really stabilizing / maturing?
Where specifically – everywhere? Exceptions?
Same trends? Anything troubling or getting worse?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 7
Metrics Analysis QuestionsWhy?
Why ask? Why probe?
Provide insight for improved management decision-making
Alert team to risks
Provide clarity and insight into the REAL state of theproduct & project
Because you have the ability to do so AND thefunctional breadth to do it well
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 8
5 Core Metrics to Guide Testing in YourEndgames
1. Found vs. Fixed
2. High Priority
3. Project Keywords
4. Defect Transition Progress
5. Product Functional Area Distribution
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 9
Defects Found vs. FixedTrending - Maturation
Find vs. Fixed Chart
0
20
40
60
80
100
120
140
160
1/1/
2003
1/8/
2003
1/15
/2003
1/22
/2003
1/29
/2003
2/5/
2003
2/12
/2003
2/19
/2003
2/26
/2003
3/5/
2003
3/12
/2003
3/19
/2003
3/26
/2003
Project weeks
#ofD
efe
cts
New Closed Deferred Cum-New Cum-Closed
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 10
Defects Found vs. FixedTrending - Maturation
Simple Defect Trending
0
10
20
30
40
50
60
C1,p
1p2
p3
p4
C2,p
1p2
p3
p4
C3, p
1p2
C3_1
, p1
p2
p3
C3_2
, p1
p2
1 Week Test Iterations
#ofDefe
cts
Open High-Priority Regressions
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 11
Defects Found vs. FixedTrending - Maturation
Found – new, vs. Fixed – repaired, closed,duplicate, not-a-bug, works-as-designed, etc.
MAQ
What are the general trends? Found down? Fix up?
Is the trending stabilizing for the project or test cycle?
Are we approaching a release point? (cumulativetrends intersect, with period of low / no find andstability)
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 12
Defects Found vs. FixedTrending - Maturation
Key Guide Points
Carefully view trending to take into considerationorganizational or other anomalies
Looking for equilibrium or regression trends (progress& stability vs. backsliding)
Feedback to Project Management
Release readiness predictions
Triage / closure concerns
Intermittent trend influences – identifying root causes
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 13
High PriorityTrending – Robustness & Risk
High Priority vs. Total Open Chart
0
20
40
60
80
100
120
1/1/
2003
1/8/
2003
1/15
/2003
1/22
/2003
1/29
/2003
2/5/
2003
2/12
/2003
2/19
/2003
2/26
/2003
3/5/
2003
3/12
/200
3
3/19
/2003
3/26
/2003
Project weeks
#ofD
efe
cts
Open P0/P1 Open P2 Open P6 Tot-Prio-Open Tot-Open
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 14
High PriorityTrending – Robustness & Risk
Specifically defined set of defects indicating highpriority (defects vs. enhancements) monitored asa significant group
MAQ
What is the overall trending – relative to position withinthe Endgame release cycle?
Stable downward curve? Is there any spiking orobvious trend regression?
Defects vs. Enhancements – scope creep?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 15
High PriorityTrending – Robustness & Risk
Key Guide Points High priority defects should decline and stabilize early
in testing iterations
Continuous and abrupt downward trending
Shouldn’t occur midway to late in Endgame
Feedback to PM Any “spiking” in trending usually indicates a systemic
regression of some sort
Late in the Endgame implies lack of robustness andincreasing risk
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 16
Project KeywordsTrending – PM Workflow Impact
Project Keyword (Open) Trends
0
5
10
15
20
25
30
35
40
1/1/20
03
1/8/20
03
1/15
/2003
1/22
/2003
1/29
/2003
2/5/20
03
2/12
/2003
2/19
/2003
2/26
/2003
3/5/20
03
3/12
/2003
3/19
/2003
3/26
/2003
Project weeks
#ofDefe
cts
Blocking Must-Enhancement Must-Beta Must-GA
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 17
Project KeywordsTrending – PM Workflow Impact
Attaching keywords to defects, allowing fortargeting repairs towards your iterative releaseplanning scheme. Normally driven by triage andchange control.
MAQ What is the trending of “must have” repairs for
individual project milestones?
What are the priorities within each targeted milestonerelease?
Will we make it? And what needs to be deferred?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 18
Project KeywordsTrending – PM Workflow Impact
Key Guide Points
Define meaningful keywords aligned with majorproject milestones
Target features, enhancements, and repairs towardsthese milestones
Feedback to PM
Blocking issues
Trending relative to milestones, example Code Freeze
Deferral guidance (priority, impact)
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 19
Defect State TransitionsTrending – Development Capacity
Defects State Transitions / Days
0
5
10
15
20
25
Jan 1-15 Jan 16-31 Feb 1-14 Feb 15-28 Mar 1-15 Mar 16-30
Project weeks
#ofD
ays
Days in New Days in Working Days in Verify Days New-to-Closed
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 20
Defect State TransitionsTrending – Development Capacity
Reviewing defect activity state transitions todetermine team capacity (efficiency & capacity)
MAQ
What is the typical triage / assigning time (Triage)?
What is the typical repair turnaround time (Dev)?
What is the typical verification time (Test)?
What are the trending and relationships amongst theteam workflows? Improving or not? Performanceproblems – specific bottlenecks?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 21
Defect State TransitionsTrending – Development Capacity
Key Guide Points As the project progresses, triage should reduce from
5-7 days average to < day average
Development time should decrease as they move fromconstruction iterations to repair, polish & maintenance
Verification time relates to overall testing cycle time,early feedback preferred. Regression rates come into play
# of days moving from New -> Closed indicates overallteam capacity Should be factored into iteration release plans.
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 22
Defect State TransitionsTrending – Development Capacity
Feedback to PM
Triage backlog & workflow bottlenecks – trackingaverage time to New -> Close
Filter out anomalies
Development-to-testing equilibrium
Either function overloaded, suggest adjustments. Moredevelopers, more testers, help each other, adjust goals.
Factor functional and team capabilities into iterativerelease planning.
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 23
Open Defects per Functional AreaTrending – Pareto (80:20 Rule) Guidance
Sample Pareto Chart
30
25
15
10 10
530
55
7080
90100
0
5
10
15
20
25
30
35
UI Mware Parsing SOAP Reports Help
Defe
cts
0
20
40
60
80
100
120
# Bugs
Cum %
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 24
Open Defects per Functional AreaTrending – Pareto (80:20 Rule) Guidance
Open Defects per Functional Area
0
5
10
15
20
25
30
Jan 1-15 Jan 16-31 Feb 1-14 Feb 15-28 Mar 1-15 Mar 16-30
Project weeks
#ofD
efe
cts
Install & Config Internal files Dbase Reporting
R-time analysis Off-line analysis GUI Help & docs
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 25
Open Defects per Functional AreaTrending – Pareto (80:20 Rule) Guidance
Partitioning the AUT into meaningfulcomponents, then diligently mapping defects tocomponent yields valuable Pareto insights
MAQ Different trends within components? Is it expected
behavior? (Late vs. early maturation)
Highest risk, lowest risk areas? By defect count,priority or regression?
Where to focus process (inspections & unit testing)and testing?
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 26
Open Defects per Functional AreaTrending – Pareto (80:20 Rule) Guidance
Key Guide Points Correct decomposition is the first challenge –
horizontally, then vertically
Requires triage insights into root cause / locale anddiligent categorization
Powerful guiding mechanism
Feedback to PM (and Development, Testing) High & Low risk components – testing direction
Individualized trending and maturation rates
How component rates map to overall project goals
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 27
Wrap-upHistorical Data
Maintain historical data active in your DTS
My observations are that:
Cycle times repeat (iterations, testing duration)
Settle times repeat (stabilization for releases of similarcomposition – new, maintenance, patch)
Bandwidth capabilities repeat for similarly sized teamsand efforts
Defect injection rates and regression rates repeat forsimilar efforts
Learn to compare and leverage the patterns
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 28
Wrap-upTheme
If there is a theme to this talk its –
Look beneath the data
Be inquisitive - ask questions of yourself and others
Find the root relationships
Share your concerns, thoughts, guidance with thebroader team
Take and expect action
In other words, become an Endgame Guide foryour team
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 29
Thank you for taking the time!
Questions?
Contact Info:RGalen Consulting Group, L.L.C.
PO Box 865, Cary, NC 27512
919-272-0719
www.rgalen.com [email protected]
Spring 2007RGalen Consulting Group, LLC
Copyright © 2007 30
References
www.stickyminds.com Search for “MetricsAnalysis Questions”. Stick Minds article link-http://www.stickyminds.com/getfile.asp?ot=XML&id=5029&fn=Smzr1XDD3368filelistfilename1%2Epdf
Software Endgames: Eliminating Defects,Controlling Change, and the Countdown toOn-Time Delivery published by DorsetHouse in December 2004. www.rgalen.comfor order info, misc. related presentations,papers, and link to a Yahoo group focusedon Endgames – please join us.
http://www.stpmag.com/issues/stp-2005-
03.pdf