defect metrics
TRANSCRIPT
Defect Management & Metrics
Society for Software Quality
April 24, 2010
021209.01SSQ Presentation © 2010 – Holl Consulting
Who Is This Guy?• 30 years in software
and hardware/software Quality
• 19 years in Quality & Process Management
• 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit
• Past SSQ PresidentQUALI YT
QSS
021209.01SSQ Presentation © 2010 – Holl Consulting
Primary Relevant Experience• Designed Intuit’s corporate defect
management process
• Assisted in rollout to ~1800 engineers
• Guided its evolution for six years
• As Corporate Software Metrics Leader, fostered quantitative project management and process improvement
• Made mistakes and learned from them
021209.01SSQ Presentation © 2010 – Holl Consulting
Agenda
• Summary of Previous Talk
• Defect Management Metrics
• Metrics for Project Management
• Metrics for Process Improvement
• Advanced Metric Models
021209.01SSQ Presentation © 2010 – Holl Consulting
Last time…
• We reviewed cultural issues related to defect management
• We went over a defect management process:
– Policies
– Workflow/States
– Fields
021209.01SSQ Presentation © 2010 – Holl Consulting
Data Collection
• At different states in the workflow, different data are collected
• Most data are collected at submit time
• Additional data are entered by the developer upon resolution
• And finally by the tester upon verification
021209.01SSQ Presentation © 2010 – Holl Consulting
Agenda
Summary of Previous Talk
• Defect Management Metrics
• Metrics for Project Management
• Metrics for Process Improvement
• Advanced Metric Models
021209.01SSQ Presentation © 2010 – Holl Consulting
When Does A Defect Count?It depends on your goals• Defect Management: To track defects for resolution in
order to release products– Just log functional problems found in test; deviations from the
requirements that the customer will see; defects that you don’t want to fall through the cracks.
• Defect Analysis: To improve the process or predict– If your goal is to get better (learn) or predict the future, you need
to capture much more information.
– Without early lifecycle data (requirements, etc.) predictive models aren’t that reliable
World-class companies don’t just manage defects – they analyze information.
021209.01SSQ Presentation © 2010 – Holl Consulting
Data Collection Costs• Estimate Assumptions:
– 18,000 defects
– ½ the defects are corrected (9,000)
– Data collection time: 10 minutes (validated at HP)
– Average defect cost: 16 hours (data from Michael Fagan)
– Data collection reduces defects 5%
Time to record (18k *10m) 3,000 hrs
Time to correct (9k *16h) 144,000 hrs
Defect reduction (9k * 5%)
450 defects
Time savings (450 * 16h) 7,200 hrs
ROI 240%
Sampling is valid, butsampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected.
This doesn’t quantify the value of being able to predict quality and schedules
021209.01SSQ Presentation © 2010 – Holl Consulting
Simple Measures
• Defect cost =
• Fix rate =
• Duration =
Last two include queue time, so they depend on workloadIt’s helpful to understand the standard deviation tooBonus: By type, root cause, injection phase
* Excel NetWorkDays is a useful function
Σ (Total Resolution Effort)
Number of resolved defects
Number of resolved defects
Period of time (week, month, etc.)
Σ((Closed Date) – (Submit Date))*
Number of resolved defects
021209.01SSQ Presentation © 2010 – Holl Consulting
Relative Defect Costs
• If you record injection and discovery phases and effort, you can compute relative costs
• Example:
– Cost of a requirements defect = 1
– Cost of a design defect = 10
– Cost of a coding defect = 100
– Cost of a production defect = 1000
021209.01SSQ Presentation © 2010 – Holl Consulting
Similar to Phase Containment• Start with matrix of injection and discovery
• Enter the average rework (or duration) costs in each cell (resolved defects only)
Injection Phase
Discovery Phase
Req's Design Code Test Production
Requirements 2.4
Design 15.3 4.1
Code 22.7 9.5 1.2
Testing 43.8 18.5 4.3 1.2
Production 85.3 38.5 18.3 2.2 1
021209.01SSQ Presentation © 2010 – Holl Consulting
Similar to Phase Containment
• Normalize by dividing each cell by the Injection=Discovery value
Injection Phase
Discovery Phase
Req's Design Code Test Production
Requirements 1.00
Design 6.38 1.00
Code 9.46 2.32 1.00
Testing 18.25 4.51 3.58 1.00
Production 35.54 9.39 15.25 1.83 1.00
021209.01SSQ Presentation © 2010 – Holl Consulting
Logging Defects• This is not
How many defects leak?
• This is
What percent of known defects are recorded?
• If you only start logging defects in test, you’re missing over half of your data
• Try to capture data from reviews/inspections and developer unit tests
021209.01SSQ Presentation © 2010 – Holl Consulting
Prioritizing Defects• This is not about how to set priority, but…
• Priority means the most important defects get resources first
• To measure this, we need to know the queue time (from “submit” to “work starts”)
• Therefore, we need an “Accepted” state, or something to note when work began
• Why can’t we use Submit to Verify duration?
021209.01SSQ Presentation © 2010 – Holl Consulting
Aging Defects
• Time in Resolver queue
• Time in Verifier queue
• Alternately, use durations minus effort as a proxy
Average Queue Time
0
2
4
6
8
10
12
0-TestStopper
1 2 3 4 5
Priority
Day
s in
Qu
eue
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Backlog• Many groups don’t count defects left over from
prior releases; they only focus on defects found and not fixed in the current project
• This is dangerous because a product defect backlog can grow unmanaged
• ALL of the open defects are released, not just the new ones
• When producing the report of open defects, query for all open defects in the product
021209.01SSQ Presentation © 2010 – Holl Consulting
The CalculationWeek
1
2
3
4
Total number of defects
– Number closed before the week started
– Number that were opened after the week ended
= Number open in the week
021209.01SSQ Presentation © 2010 – Holl Consulting
Calculating Backlog in Excel
• Query Submit Date and Closed Date
021209.01SSQ Presentation © 2010 – Holl Consulting
Calculating Backlog in Excel
This uses named ranges, where
D.Num.CRs = Total number of CRs opened
D.T.Closed = Range of Closed Dates
D.Submit.Date = Range of Submit Dates
Array formula. <Shift-Ctrl-Return>
021209.01SSQ Presentation © 2010 – Holl Consulting
The Result
Product Defect Backlog
0
200
400
600
800
1000
12001/
7/20
06
2/7/
2006
3/7/
2006
4/7/
2006
5/7/
2006
6/7/
2006
7/7/
2006
8/7/
2006
9/7/
2006
10/7
/200
6
11/7
/200
6
12/7
/200
6
1/7/
2007
2/7/
2007
3/7/
2007
4/7/
2007
5/7/
2007
6/7/
2007
7/7/
2007
8/7/
2007
9/7/
2007
10/7
/200
7
11/7
/200
7
12/7
/200
7
1/7/
2008
2/7/
2008
3/7/
2008
4/7/
2008
# O
pen
Def
ects
021209.01SSQ Presentation © 2010 – Holl Consulting
Agenda
Summary of Previous Talk
Defect Management Metrics
• Metrics for Project Management
• Metrics for Process Improvement
• Advanced Metric Models
021209.01SSQ Presentation © 2010 – Holl Consulting
Some Project Management Defect Metrics
• Reviews & Inspections
• Defect Discovery & Closure
• Defect Density
021209.01SSQ Presentation © 2010 – Holl Consulting
A Project Axiom
• The quality of product in one phase is an indicator of the quality in later phases
• Why?
–There are two answers
021209.01SSQ Presentation © 2010 – Holl Consulting
Review Metrics
• Reviews & inspections have proven to be the most cost-effective way to detect and remove defects
• Inspect your key project artifacts (requirements, designs, code, tests, etc.)
021209.01SSQ Presentation © 2010 – Holl Consulting
Review/Inspection Data
• There are many data collected during a review or inspection
• The ones that tell us about quality are:
– Number of Operational defects discovered
– Number of Minor defects found
per changed line of inspected doc/code
• Number of reviews per doc/module
021209.01SSQ Presentation © 2010 – Holl Consulting
Project Defect Density
• Consistency in categorizing inspection defects is important
– Have clear guidelines for Operational and Minor defects
• Defect Density Project =
Operational Defects found by inspection
Delta SLOC (or LOT)
021209.01SSQ Presentation © 2010 – Holl Consulting
Projecting Defect Leakage• Inspected total of 5340 SLOC
• Found 47 operational defects
• Density = 47/5340 = 8.8 defects/KSLOC
• Project changed 18,463 SLOC
• Estimated defects = 18.463 * 8.8 = 162
• Remaining = 162 - 47 = 115
What should you do? Inspect more? Or Test for them?
021209.01SSQ Presentation © 2010 – Holl Consulting
Verification
• Use Defect Density to estimate expected yield
– Similar to estimate described with reviews
• Defect discovery & closure
• Use extrapolation to predict when defect exit criteria will be met
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Discovery
Time
Cum
ulat
ive
Def
ects
Fou
nd
Total Defects Opened
Ideal Curve!
Weekends
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Discovery Curve Assumptions
• Time is a proxy for “exposure”
– Remove weekends to smooth the curve
• Number of test cases run would be better, assuming
– Test coverage is good
– Not continuing to add new test case
021209.01SSQ Presentation © 2010 – Holl Consulting
Well-Behaved Curve
Time
Cum
ulat
ive
Def
ects
Fou
nd
Total Defects Opened
Release Target Date
021209.01SSQ Presentation © 2010 – Holl Consulting
Add: Defect Closure
Time
Cum
ulat
ive
Def
ects
Total Defects Opened & Closed
Opened
Closed
Remaining Open
021209.01SSQ Presentation © 2010 – Holl Consulting
Add: To QA
Time
Cum
ulat
ive
Def
ects
Total Defects Opened, Closed, & To QA
Opened
To QA
Closed
Opened, waiting for development fix
Turned over from development to QA, awaiting verification
021209.01SSQ Presentation © 2010 – Holl Consulting
Finding The Balance
Time
Cum
ulat
ive
Def
ects
Total Defects Opened, Closed, & To QA
Opened
To QA
Closed
What’s happening
here?
QA is a bottleneck
Dev is a bottleneck
021209.01SSQ Presentation © 2010 – Holl Consulting
Add: Legacy & Target Release
Time
Cum
ulat
ive
Def
ects
Total Defects Opened & Closed
Total
Targeted
To QA
Closed
Inherited from previous history, targeted for or found in the project/release
021209.01SSQ Presentation © 2010 – Holl Consulting
Extrapolation to Release Date
Time
Cum
ulat
ive
Def
ects
Project Defects
Total
Targeted
To QA
Closed
Release Target Date
Should I be concerned?
021209.01SSQ Presentation © 2010 – Holl Consulting
Extrapolation to Release Date
Time
Cum
ulat
ive
Def
ects
Project Defects
Total
Targeted
To QA
Closed
Release Target Date
When will the project be done?
021209.01SSQ Presentation © 2010 – Holl Consulting
Computing in Excel
Query defects into Excel
Convenient: Use named ranges for data
021209.01SSQ Presentation © 2010 – Holl Consulting
Computing in Excel• In a second sheet, create a date column and count
the defects submitted and closed for each date.
Array formula. <Shift-Ctrl-Return>
021209.01SSQ Presentation © 2010 – Holl Consulting
Real Life ExampleAardvark 4.7 Defect Discovery & Closure
(With extrapolations)
0
50
100
150
200
250
300
350
400
450
5006/
17/0
8
6/24
/08
7/1/
08
7/8/
08
7/15
/08
7/22
/08
7/29
/08
8/5/
08
8/12
/08
8/19
/08
8/26
/08
To
tal C
Rs
0
10
20
30
40
50
60
70
80
90
100
Fo
un
d in
4.7
Σ Open
Σ Closed
Σ Found in 4.7
6/16 = Start of testing
Target Release
Date
2nd order polynomial fit
021209.01SSQ Presentation © 2010 – Holl Consulting
Release Criteria: Defects
• Goals:
– Provide good-enough quality to customers
• Don’t release any “bad” defects
• Don’t release “too many” defects
– Minimize support costs
• Customers will need support proportional to product quality
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Discovery
Time
Cum
ulat
ive
Def
ects
Fou
nd
Total Defects Opened
Release Target Date
Should I release?
021209.01SSQ Presentation © 2010 – Holl Consulting
Bad Defects• Bad is a function of
– Severity = the consequences of the failure
– Probability that it will occur
– The number of customers likely to experience it
– May also include “important” customers or functions
• These are often complex to calculate, so most companies just use Severity, tempered by other factors
021209.01SSQ Presentation © 2010 – Holl Consulting
Quality Threshold• No more than the following number of open
defects:
– Where New = defects discovered in the current project, and
– Old = defects discovered in previous releases
Severity* New Old
Critical 0 0
Important 5 10
Moderate 30 100
Trivial 50 200
Release team (stakeholders) need to determine values based on history. The goals are:• Not to release any Critical defects• Not to introduce “too many” new defects• Manage the backlog
021209.01SSQ Presentation © 2010 – Holl Consulting
What’s Good Enough?
The two market leaders – Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block
It also offers better end-of-return reviews …
TurboTax offers better videos than TaxCut …
021209.01SSQ Presentation © 2010 – Holl Consulting
What’s Good Enough?TaxCut offers easier tagging of items you’d like to deal with later.
It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTax’s screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected.
021209.01SSQ Presentation © 2010 – Holl Consulting
What’s Good Enough?
… if you’re choosing which program to go with for the first time this year, I’d go with TaxCut – solely because of the text errors in TurboTax.
I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didn’t affect the bottom line, they did affect my confidence in the program’s makers.
It’s unfortunate, because I felt TurboTax offered better features.
021209.01SSQ Presentation © 2010 – Holl Consulting
Detroit Free Press
> 1.7M Readers+ web site+ wire serviceThe article came from the Monterey Peninsula Herald
021209.01SSQ Presentation © 2010 – Holl Consulting
Predicting Defects• There are several models for predicting
defects
• The predictions of interest are:
– How many defects are going to be created?
– When will I find them?
– How long will it take to fix them?
– When will I be done?
– How good will the result be?
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Defect Density• Phase Defect Density
DDPhase =
• At first this may not seem right, but it’s a reasonable predictor
• Which is more reliable?
– 67 specification defects per 200 pages of documentation (0.34 defects/page)
– 67 specification defects per 4300 SLOC (15.6 defects/KSLOC)
Defects InjectedPhase
Total Project Affected SLOC
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Defect Density• Project/Process Defect Density
– Number of defects peraffected SLOC
– Affected SLOC can be difficult to get without a good tool.
– The best I’ve found is CodeReports (aka SourceROI)
– http://www.polariguana.com/sourceroi.php
Release A Release B
Defects 452 563Total SLOC 8,600 43,500Defect Density 52.6 12.9
Project Changed SLOC
5,400 4,350
Project Defect Density
83.7 129.4
021209.01SSQ Presentation © 2010 – Holl Consulting
Calculating DD per Phase
# Defects Injected Per Phase Project KSLOC
Number of Defects Injection Phase
Discovery Phase Req's Design Code Test Production
Requirements 46 0 0 0 0
Design 8 26 0 0 0
Code 83 19 83 0 0
Testing 33 22 37 2 0
Production 3 2 16 0 1
Total Injected 173 69 136 2 1
Defect Density 4.94 1.97 3.89 0.06 0.03
Project SLOC = 35,000Note: if this is only test defect data, you’re missing an opportunity to get better information.
DDPhase =
021209.01SSQ Presentation © 2010 – Holl Consulting
Next Project• Estimated to be 20,000 SLOC
Historical Defect
Density
Likely to
Inject
Phase Containment
Likely to Leak
Effort to Resolve*
Total Rework
Requirements 4.94 99 26.6% 26 85.3 2242.2
Design 1.97 39 17.3% 7 38.5 263.3
Code 3.89 78 62.1% 48 18.3 882.9
Testing 0.06 1 81.7% 1 2.2 2.1
Production 0.03 1 100.0% 1 1 0.6
Total 218 Grand Total 3391.0
* Worst case cost (defects discovered in test)
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Discovery & Closure
• Predictions can bemade in Excel by extrapolating andseeing where the discovery and close lines meet
Cumulative Defects Submitted
0
50000
100000
150000
200000
250000
300000
10/1
4/200
6
10/2
8/200
6
11/1
1/200
6
11/2
5/200
6
12/9
/200
6
12/2
3/200
6
1/6/
2007
1/20
/200
7
2/3/
2007
2/17
/200
7
3/3/
2007
3/17
/200
7
3/31
/200
7
4/14
/200
7
4/28
/200
7
5/12
/200
7
5/26
/200
7
6/9/
2007
6/23
/200
7
7/7/
2007
7/21
/200
7
8/4/
2007
021209.01SSQ Presentation © 2010 – Holl Consulting
Extrapolation
• This method is quick and in some cases surprisingly good
• But not in all cases
021209.01SSQ Presentation © 2010 – Holl Consulting
Other Excel Extrapolations
• Adding a trend line uses all the data
• You may only want to use some of the data
• Create a parallel column of the data you want to extrapolate and extrapolate only that curve
021209.01SSQ Presentation © 2010 – Holl Consulting
Example
021209.01SSQ Presentation © 2010 – Holl Consulting
Simple Linear Extrapolation
• To create a linear extrapolation on the last few data points, start by extending the date column:
021209.01SSQ Presentation © 2010 – Holl Consulting
Simple Linear Extrapolation
• Select the cells that look linear
• Copy and PasteSpecial - Values
021209.01SSQ Presentation © 2010 – Holl Consulting
Simple Linear Extrapolation
• Then drag the cells down to the last extrapolated date
• Excel willdo a linearextrapolationof the data
021209.01SSQ Presentation © 2010 – Holl Consulting
Extrapolation• In order for extrapolations to be reliable,
the discovery rate must have peakedWeekly Defects Submitted
0
10
20
30
40
50
60
1/1
/20
08
1/1
5/2
00
8
1/2
9/2
00
8
2/1
2/2
00
8
2/2
6/2
00
8
3/1
1/2
00
8
3/2
5/2
00
8
4/8
/20
08
4/2
2/2
00
8
5/6
/20
08
5/2
0/2
00
8
6/3
/20
08
6/1
7/2
00
8
7/1
/20
08
7/1
5/2
00
8
7/2
9/2
00
8
8/1
2/2
00
8
8/2
6/2
00
8
9/9
/20
08
9/2
3/2
00
8
10
/7/2
00
8
021209.01SSQ Presentation © 2010 – Holl Consulting
Done?
• What does it mean to be done (from a defect perspective)?
– No “bad” defects remain
– Predictable/manageable defects remain
• Two categories of defects remain
– Those you know
– Those you don’t know
021209.01SSQ Presentation © 2010 – Holl Consulting
Latent Defects
• Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF
• What needs to happen before the predictions become reliable?
– The defect discovery rate must flatten
d f(t) dt
= 0
021209.01SSQ Presentation © 2010 – Holl Consulting
Discovery RateCumulative Defect Submitted
0
100
200
300
400
500
600
1/1/
2008
1/15
/200
8
1/29
/200
8
2/12
/200
8
2/26
/200
8
3/11
/200
8
3/25
/200
8
4/8/
2008
4/22
/200
8
5/6/
2008
5/20
/200
8
6/3/
2008
6/17
/200
8
7/1/
2008
7/15
/200
8
7/29
/200
8
8/12
/200
8
8/26
/200
8
9/9/
2008
9/23
/200
8
10/7
/200
8
Weekly Defects Submitted
0
10
20
30
40
50
60
1/1/
2008
1/15
/200
8
1/29
/200
8
2/12
/200
8
2/26
/200
8
3/11
/200
8
3/25
/200
8
4/8/
2008
4/22
/200
8
5/6/
2008
5/20
/200
8
6/3/
2008
6/17
/200
8
7/1/
2008
7/15
/200
8
7/29
/200
8
8/12
/200
8
8/26
/200
8
9/9/
2008
9/23
/200
8
10/7
/200
8
021209.01SSQ Presentation © 2010 – Holl Consulting
Fitting Data to a Model• Fitting data
to a curve helps us tell the rate is decreasing
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Excel
• As before, create a linear extrapolation of the last <criteria> days:
• “Flattening” criteria example:
– The discover rate must less than 5 defects per day in the past two weeks of testing
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Excel
• Add linear trend line
• Select “Display equation on chart”
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Excel• This gives the slope of the line m,
where y=mx + b
– In this case, m = 2.2 defects/day
– Not good enough
Cumulative Defect Submitted
y = 2.163x - 85395
0
100
200
300
400
500
600
1/1/
2008
1/15
/200
8
1/29
/200
8
2/12
/200
8
2/26
/200
8
3/11
/200
8
3/25
/200
8
4/8/
2008
4/22
/200
8
5/6/
2008
5/20
/200
8
6/3/
2008
6/17
/200
8
7/1/
2008
7/15
/200
8
7/29
/200
8
8/12
/200
8
8/26
/200
8
9/9/
2008
9/23
/200
8
10/7
/200
8
021209.01SSQ Presentation © 2010 – Holl Consulting
Latent or Known Defect Risk• Probability of occurrence in production
– Risk = Severity * Probability
• Need to have a decent measure/estimate of probability
– Need a usage profile
• Also, frequency does not always equal value
– I may not use the “Restore” function very often, but it better work
021209.01SSQ Presentation © 2010 – Holl Consulting
And Finally…
• Track latent/released defects carefully
• They are your best indicator of Quality
021209.01SSQ Presentation © 2010 – Holl Consulting
Release Defect DensityFrom:
Software Quality Measurement:A Framework for Counting Problems and Defects
CMU/SEI-92-TR-022
http://www.sei.cmu.edu/pub/documents/92.reports/pdf/tr22.92.pdf
021209.01SSQ Presentation © 2010 – Holl Consulting
Agenda
Summary of Previous Talk
Defect Management Metrics
Metrics for Project Management
• Metrics for Process Improvement
• Advanced Metric Models
021209.01SSQ Presentation © 2010 – Holl Consulting
A True Story
• A while back, I looked in the mirror and decided I needed to loose some weight.
• What is the first thing I’m going to do?
021209.01SSQ Presentation © 2010 – Holl Consulting
My First Step
021209.01SSQ Presentation © 2010 – Holl Consulting
My First Step – On A Scale
• If I don’t measure myself today, I won’t know
– If the changes I’m planning to make are taking me the right direction
– Which changes have the best affect
– How far I’ve come
– When I’m done
• It’s important to establish a baseline
021209.01SSQ Presentation © 2010 – Holl Consulting
What Are The Goals?
`Would you tell me, please, which way I ought to go from here?'
`That depends a good deal on where you want to get to,' said the Cat.
`I don't much care where--' said Alice.
`Then it doesn't matter which way you go,' said the Cat.
021209.01SSQ Presentation © 2010 – Holl Consulting
Closure Data Collection
Submit reason
• Test case failed• Pass criteria unclear• Tested wrong version• Setup/config wrong• Data-related error• Other
No
• Correction made• Duplicate• Change not justified
Injection Phase
Root Cause
Defect?
Resolution
Yes
Classification, Mode
Corrected Component(s)
This is where we record what we’ve learned
Total Resolution Effort
Resolved
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Classification Models
• Industry guidelines to standardize information collection
• The most prevalent are from:
– IEEE
– IBM
– Hewlett Packard
021209.01SSQ Presentation © 2010 – Holl Consulting
HP Defect Origins, Types, and Modes
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Leakage Reasons1. Environmental Dependency
2. Test Not Run
3. Test Case Error
4. Data Dependency
5. Inconsistent Symptoms
6. Test Case Success Criteria Unclear
7. Test Execution Error
8. Test Case Results Not Verified
9. Code Changed After Test Passed
10. No Test Case/Coverage
11. Configuration Management Problem
12. Requirements Missing/Changed
021209.01SSQ Presentation © 2010 – Holl Consulting
Using Leakage Data• For each defect found in production, log a leakage
reason
• Use the frequency (volume) to improve the related process
• Resist temptation to weight the frequency by something important
– This assumes cause & effect, and there is none
– This is different from root cause, where the cost to repair is related to the type of defect
021209.01SSQ Presentation © 2010 – Holl Consulting
Specific Process Improvement Metrics
• Two types of process metrics:
1. Metrics that monitor a process (tell you the effect of changes)
• Basic SPC-type charts
2. Metrics that tell you what to do
• Phase Containment
• Root Cause Analysis
• Test Coverage
• Defect Leakage Reasons
• Static Code Analysis - Complexity
021209.01SSQ Presentation © 2010 – Holl Consulting
Should all projects be measured?• Pros:
– More data points
– Evaluate everything we’re doing
• Cons:– Projects may use different processes
– Variation may be too great
• Suggestions:– Don’t measure projects – measure processes
– Only measure your best projects
– Tailoring is okay, but separate out the data
021209.01SSQ Presentation © 2010 – Holl Consulting
Hold Periodic Process Reviews
• Monthly Quality Reviews focused on
– Process Improvement initiatives
– Measures & Analysis/Interpretation
– Lessons Learned
• Rules of metrics etiquette:
– Preview data with stakeholders = no surprises
021209.01SSQ Presentation © 2010 – Holl Consulting
Phase Containment
• Identifies the lifecycle activities that leak defects
• Tells you where to improve defect detection efforts
Phase Containment
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Requirements Design Code Build Testing
% D
efe
cts
Co
nta
ine
d
021209.01SSQ Presentation © 2010 – Holl Consulting
Root Cause Analysis
• Pareto the types of errors being created, weighted by cost (effort)
Root Cause Total Hours
0
500
1000
1500
2000
2500
3000
Req
uire
men
ts A
mbi
guou
s/va
gue
Req
uire
men
ts M
issi
ng
Logi
c flo
w
Mis
sing
logi
c
Req
uire
men
ts C
hang
ed
Wro
ng v
aria
ble
used
Oth
er
Cas
e st
atem
ent
Typo
Spe
cific
atio
n M
issi
ng
Gar
bage
col
lect
ion
Exc
eptio
n ha
ndlin
g
Par
amet
er m
ism
atch
Req
uire
men
ts W
rong
Uni
nitia
lized
var
iabl
e
Bad
val
idat
ion
chec
k
Mis
sing
end
sta
tem
ent
Env
not
mat
ch t
est
Spe
cific
atio
n-re
quire
men
t m
ism
atch
Cod
ing
stan
dard
not
fol
low
ed
Spe
cific
atio
n A
mbi
guou
s
Env
cap
acity
Spe
cific
atio
n W
rong
Wro
ng m
etho
d ca
lled
Des
ign
logi
c w
rong
Des
ign-
spec
mis
mat
ch
Req
uire
men
ts I
ncon
sist
ent
Env
har
dwar
e fa
ilure
Doc
typ
o
Doc
vag
ue
Env
Ope
ratio
nal E
rror
Inst
all m
issi
ng f
iles
Req
uire
men
ts O
ther
Doc
info
mis
sing
Inst
all w
rong
con
figur
atio
n
Inst
all i
ncom
plet
e
Doc
wro
ng
Inst
all w
rong
ver
sion
Root Cause x Rework Hours
021209.01SSQ Presentation © 2010 – Holl Consulting
Agenda
Summary of Previous Talk
Defect Management Metrics
Metrics for Project Management
Metrics for Process Improvement
• Advanced Metric Models
021209.01SSQ Presentation © 2010 – Holl Consulting
Defect Probabilities• Weibull distributions are useful engineering
models, where the tail approaches zero asymptotically
• Weibull Probability Density Function (PDF) is:
Where m determines the shape of the curve, andc is the scale parameter
tc
mt
m
etc
m
-
f(t) =tc
mc
m-1
etc
m
-
=
(Wikipedia)
021209.01SSQ Presentation © 2010 – Holl Consulting
Different Weibull Distributions
Weibull Curves
0
1
2
3
4
0 1 2 3
m=0.5
m=1 m=2 m=4
m=10
Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions)
021209.01SSQ Presentation © 2010 – Holl Consulting
Weibull Probability Distribution
Weibull Curvesm = 0.1 to 5.0 in .1 increments
0.0
0.5
1.0
1.5
2.0
0.0 0.5 1.0 1.5 2.0
Rayleigh (m=2)
021209.01SSQ Presentation © 2010 – Holl Consulting
Weibull Curves
Weibull Curvesm = 0.1 to 5.0 in .1 increments
0.0
0.5
1.0
1.5
2.0
0.0 0.5 1.0 1.5 2.0
Rayleigh (m=2)
(almost) Normal (m=3.4)
021209.01SSQ Presentation © 2010 – Holl Consulting
Rayleigh Probability Distribution Function
• Weibull distribution with shape (m) = 2
• Let x = 2t
tc
2t
2
etc
2
-
f(t) =2tc2
etc
2
-
=
xc2
ex2c
2
-
f(x) =xc2
ex2
4c2-
=
021209.01SSQ Presentation © 2010 – Holl Consulting
Rayleigh Probability Distribution Function
xc2
ex2
4c2-
f(x) =Rayleigh Curves
c = 0.1 to 0.5
0
1
2
3
4
5
6
7
8
9
0.0 0.5 1.0 1.5
c = 0.1
c = 0.2
c = 0.3
021209.01SSQ Presentation © 2010 – Holl Consulting
Excel Formula
021209.01SSQ Presentation © 2010 – Holl Consulting
Applying Rayleigh PDF• Let tm = time at which the curve reaches its peak
• To find the peak, we set = 0
• Solving gives tm =
• To scale the area under the curve, multiply by K
• Substitute c = tm 2
d f(t) dt
c
2
021209.01SSQ Presentation © 2010 – Holl Consulting
Fitting The Rayleigh PDF
2tc2
etc
2
-
f(t) = K 2t
(tm 2)2e
t(tm 2)
2
-
= K
2t2t2
me
- t2
2t2m= K
1tm
te- t2
2t2mf(t) = K
2
If we can estimate how many defects we’ll find (K) and when we’ve hit the peak (tm), we can create a PDF for our project!
021209.01SSQ Presentation © 2010 – Holl Consulting
Building Your Estimate
021209.01SSQ Presentation © 2010 – Holl Consulting
Predicting Latent Defects• At peak, the area under the Rayleigh curve is 39.35%
• This means 60.65% of the defects are yet to find
• Example:– At peak you found a cumulative total of 58 defects.
– Total = (58/39.5)*100=147.4
– 147 - 58 = 89 remaining.
Latent Defects = K dt 1tm
te- t2
2t2m
2
Release Date
1 year?
021209.01SSQ Presentation © 2010 – Holl Consulting
Easier that Calculus
• In Excel, run a column for cumulative total and a column for remaining defects
• From your release date you can easily estimate latent defects
021209.01SSQ Presentation © 2010 – Holl Consulting
Cumulative Rayleigh Model
tc
2
-
CDF = 1 - e
021209.01SSQ Presentation © 2010 – Holl Consulting
Rayleigh Assumptions
• The defect rate observed during the development process is positively correlated with the defect rate in the field.
• Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages.
021209.01SSQ Presentation © 2010 – Holl Consulting
Exponential Models
• Exponential Model (Weibull, m=1)
1ce
tc
-
f(t) = ExponentialWeibull, m = 1
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0.0 0.5 1.0 1.5 2.0
• Error detection rate• Instantaneous failure rate• Hazard rate
• In general, this is a reliability growth model
021209.01SSQ Presentation © 2010 – Holl Consulting
Fitting A Model to Data• EasyFit from www.mathwave.com
021209.01SSQ Presentation © 2010 – Holl Consulting
Model Summary
• Rayleigh is Weibull distribution with m=2
• Other values of m produce other probability distributions
• There are many models; try several that fit
• “All models are wrong. Some models are useful.”
– George E. Box
021209.01SSQ Presentation © 2010 – Holl Consulting
Summary
• A defect management program includes
– A balanced solution
– Training & explaining
– Metrics that are used