the testing improvement story: turning defects into dollars
TRANSCRIPT
![Page 1: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/1.jpg)
The Testing Improvement Story: Turning Defects into Dollars
Rebecca Staton-Reinstein, Ph.D., CSQAAdvantage Leadership, Inc.
![Page 2: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/2.jpg)
You must demonstrate the value of your quality efforts to Management
Demonstrate ROI for every effortTrack impact of every effortUse objective and subjective measuresSpeak the language of management:
SavingsCost AvoidanceRevenue/Time to MarketReturn on Investment
![Page 3: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/3.jpg)
The Beta Company Case Study Situational Analysis -- 2006
Beta is a big player in its industryBeen around a long time, bureaucratic, slow to change – stuffyNew, vigorous competition,markets shifting and changingMany long-term employees who are dedicated to improvement & qualityWeak IT organization, diffuse reporting structure at beginning of study
![Page 4: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/4.jpg)
Gap Analysis DIY or Consultant
QA and Testing SurveyMeasurement Readiness SurveyInterviewsDocument ReviewDefect AnalysisObservation
![Page 5: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/5.jpg)
Use Surveys to Demonstrate Need and Results of Efforts
Keep it simple, short, and focusedUse a consistent format: positive statement, 5 choices (1 = Strongly Disagree; 5 = Strongly Agree)Ask for specific comments – adds depthSupplement with interviews, doc reviewBe as concerned with range of responses as calculating the average
![Page 6: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/6.jpg)
Report Content
Context for report and recommendations = Cost of Quality ModelDefinitions of Quality, Quality Assurance, Quality Control, Quality ImprovementInternational standards: CMM(I), TMMStrategic approach to improving results of testing and development
![Page 7: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/7.jpg)
PLAN
DOCHECK
ACT
Product, O/PFIT FOR USE
-- JOSEPH JURAN
Continuous Improvement of ALL Processes-- W. EDWARDS DEMING
QU
ALI
TY IS
…
![Page 8: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/8.jpg)
Quality Assurance QADefine SDLC Processes
DevelopmentMaintenancePurchaseOutsource
Define StandardsDefine Review ProcessesDefine Measurement
Analyze dataDefine Test Processes, ToolsDefine:
QA, QC, QI, IT processesTrain
ISO Six Sigma/LeanMBNQA IEEECMM(I) TMM
TickIT SPICEPMI ITIL Other
Quality Control QCReviews:
Walkthroughs, Peer Reviews, Inspections
Audits: product, processTesting: All types, UAT Controls: Change, Configuration
Defect DATALessons Learned
Quality Improvement QIImprove productImprove processReduce variationReduce defectsRe-engineer process
Relationship among all quality elements
QA = PREVENT DEFECTS
QC = FIND DEFECTS
QI = REDUCE DEFECTSPL
AN
DO CHECK
ACT
![Page 9: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/9.jpg)
Cost of QualityThe Good, the Bad and the Ugly
FAILURE
PREVENTION
APPRAISAL
PRODUCTION
Cost of Quality
Joseph Juran
Phil Crosby
![Page 10: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/10.jpg)
ROI $FAILURE FAILURE
PREVENTION PREVENTION
APPRAISAL APPRAISAL
PRODUCTION PRODUCTION
Cost of QualityThe Good, the Bad and the Ugly
C
O
Q
![Page 11: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/11.jpg)
Strategies to Reduce COQIncrease the Cost of PreventionShift Appraisal emphasis to beginning of Life Cycle, more static testingUse test tools more effectively, efficiently
Why should BETA do this?Appraisal: Find defects early, save time, $Prevention: Reduce defects, reduce costs;Use defect data to improve processesImprove time to market
![Page 12: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/12.jpg)
BETA had Quality Policy but ignored it
QT used existing policy as starting point
Surveyed all IT employees, managers
Questions based on TMM, CMM, specific conditions at BETA
![Page 13: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/13.jpg)
BCBS MN Testing & QA
3.01
3.19
2.64
3.08
3.07
2.38
2.50
2.71
0
1
2
3
4
51. Continuous Improvement
2. Team Skills & Responsibilities
3. Configuration Management & Standards
4. Testing Process Definition
5. Testing Execution
6. Training & Support
7. Risk & Defect Management
8. Measurement Analysis & Reporting
Kiviat Diagram shows results dramatically
50.4% Response Rate
![Page 14: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/14.jpg)
QA & Testing Survey: Quality Process Conclusions
Islands of Good SW Engineering Practice“We have pockets of order”
“’It’s too late…ship it’ is the rule”“We need a quality group that can’t be circumvented”Lack of fully deployed Quality PolicyVariable processes are defect generators
![Page 15: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/15.jpg)
Conclusions compared BETA practices to
CMM KPAs to demonstrate maturity levelBETA believed it was CMM Level 2 – it was Level 1
Highlighted good practices, teams
Place conclusions in relation to GSEP –
Good Software Engineering Practice
Made recommendations specific for company
![Page 16: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/16.jpg)
Testing Maturity Model (TMMSM)
Initial
Phase Definition
Integration
Management,Measurement
Optimization defect prevention,
quality control
Builds infrastructure to assureQuality of Fact matches
Quality of Perception
![Page 17: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/17.jpg)
BCBS MN Testing MATURITY
3.01
3.08
2.64
2.96
2.96
2.65
2.40
2.59
0
1
2
3
4
51. Continuous Improvement
2. Team Skills & Responsibilities
3. Configuration Management & Standards
4. Testing Process Definition
5. Testing Execution
6. Training & Support
7. Risk & Defect Management
8. Measurement Analysis & Reporting
Testing Maturity
![Page 18: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/18.jpg)
Conclusions compared BETA practices to TMM Goals to demonstrate maturity level
BETA was at TMM Level 1
Highlighted good practices, teams
Place conclusions in relation to GSEP –
Good Software Engineering Practice
Made recommendations specific for company
![Page 19: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/19.jpg)
Testing Center of Excellence(recommendations)
Define, deploy testing methodologyTrain testers on tool useMatrix manage all testersManage technical testersAudit use of testing methodologySelect, train new testersConsult to projects on testing, UAT
![Page 20: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/20.jpg)
Implementation Considerations
Establishing Mission Critical Processes for TestingCost/Benefit Analysis: 10:1 ROI – couldavoid $5 million first year of implementationFunded through defect elimination and prevention
![Page 21: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/21.jpg)
The Defect StudyAnalyze defect data for one year from test tool defect trackerAll defects reported and closed prior to production and in production cycleAnalyze defect data by phase found; ignore ‘cause’dataAnalyze data using industry studies
![Page 22: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/22.jpg)
Software Engineering EconomicsBarry BoehmPrentice Hall,
1981
![Page 23: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/23.jpg)
Where Defects Caused/Found
56%
27%
7%
(10%)
James Martin
![Page 24: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/24.jpg)
Defects by Stage FOUND
2616
506
93 89
18
7
14
Testing Production ConstructionRequirements Warranty Analysis & DesignConfiguration & Change Mgmt
TESTING
PRODUCTION
![Page 25: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/25.jpg)
Calculating Defect Cost
Life Cycle StageRelative
Cost/DefectExample:
Actual Cost/Defect
Requirements $2 $100Design $5 $250Code $10 $500Testing $20 $1000Configuration Mgt $50 $2500Warranty $100 $5000Production $200 $10,000Cost to Repair = (Defects) X (Relative Cost) X (Hourly Rate) X (Time)(Based on Boehm) X
Example: $50/hour
![Page 26: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/26.jpg)
What defects cost BETA in 2007
Life Cycle Phase
Number of Defects Found
Percent of Defects Found
Cost to Find/Fix Defects
Requirements 46 1% $ 4,600 Design 27 0.6% $ 6,750Code/Unit Test 89 2% $ 44,500 Test 3954 86% $ 3,954,000Configuration 14 0.3% $ 35,000 Warranty 37 0.8% $ 185,000 Production 437 9.5% $ 4,370,000
TOTAL 4604 100% $ 8,599,850Based on hourly rate of $50 – note actual rate for BETA was higher
![Page 27: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/27.jpg)
BETA Costs to Find 85% of Defects Where They Are Caused in 2007
Life Cycle Stage
Actual Cost of Found Defects
% of Defects Found
Expected % of
Defects
Expected Number
of Defects
85% of Defects in Stage
Cost to find/fix 85%
of Defects
Requirements $ 4,600 1% 56% 2578 2191 $ 219,100
Design $ 6,750 0.60% 27% 1243 1057 $ 264,000
Code/Unit Test $ 44,500 2% 7% 322 274 $ 137,000
Test $ 3,954,000 86% 10% 461 *931 $ 931,000
Configuration $ 35,000 0.30% 0 0 0 ---
Warranty $ 185,000 0.80% 0 0 0 $ ---
Production $ 4,370,000 9.50% 0 0 **151 $ 1,510,000
TOTAL $8,599,850 100% 100% 4604 4604 $ 3,061,000*Test estimated to find 86% of remaining 1082 defects **Remaining defects = 3% of total
![Page 28: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/28.jpg)
Conclusions:•The current estimated cost of defects for 100% of projects is$14,000,000 out of a budget of $30,000,000 annually.•Defect costs will increase as X projects emphasized.•Favorable cost comparison to competitors unmaintainable. •Defect source identification wrong, doesn’t point to root causes•Impossible to fix flawed processes with incorrect information.•QT is not charged with providing vital data to management, •Management cannot make effective decisions on Quality.•If ALL defects found in stage where they were created, cost to find and fix defects would be reduced to $1,400,000!
Defect Study Conclusions
![Page 29: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/29.jpg)
BETA established IT Plan Quality Framework
Quality Management Definitions : Defined PrinciplesIT Steering Committee; COO, VPsIT Policies: SDLC, Quality, Testing, RequirementsOverall goal: reduce number of defects being introduced, identifying/resolving defects closer to their point of origin
![Page 30: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/30.jpg)
Initiative to Improve Testing2006 - 2008
•Testing Resource Center •Automation and Performance Testing•Stage Gate Assessments•Business Acceptance Testing•Testing and Quality Policy
•Use of XYZ Test Tools•Definitions of Testing Roles, Responsibilities•Improved Templates for Testing•Early Involvement Project Control Boards•IT Plan Deployment
IT Quality Plan Framework embraced by management
![Page 31: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/31.jpg)
Increased use of ZYZ Test Toolto track Defects
100%
90%
80%
70%
2005-6 2006-7 2007-8
60%
81%
84%
![Page 32: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/32.jpg)
Improved defect detection from improved testing processes
100%
90%
80%
70%
2005-6 2006-7 2007-8
78.3%
86.0%90.5%
![Page 33: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/33.jpg)
Fewer defects enter production due to improve testing processes20%
15%
10%
5%
2005-6 2006-7 2007-8
15.1%
9.5%
6.2%
![Page 34: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/34.jpg)
BCBS MN Testing & QA
0
1
2
3
4
5Continuous Improvement
Team Skills & Responsibilities
Configuration Management & Standards
Testing Process Definition
Testing Execution
Training & Support
Risk & Defect Management
Measurement Analysis & Reporting
Survey Dimension 2008 Response 2006 Response
Baseline = Blue vs. Year 3 Change = Purple
80.5% Response Rate
![Page 35: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/35.jpg)
Defect Costs Comparison
$-
$2,000,000
$4,000,000
$6,000,000
$8,000,000
$10,000,000
$12,000,000
Require
mentsAna
lysis/
Design
Constru
ction Tes
tConfi
guratio
n
Warranty
Produc
tion
Y1 CostY2 CostY3 CostInd Cost
![Page 36: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/36.jpg)
BETA began implementing Inspections in 2008Trained Inspectors and ModeratorsEstablished Defect Analysis and ReportingCompared similar projects with and without Inspections
![Page 37: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/37.jpg)
# Defects % of Defects Cost of Defects
TOTAL 234 100 $45,846 $243,250
Cost Avoid $518,404 -0-
Stage Found
Project A Project B Project A Project B Project A Project B
REQ 220 -0- 94.00% -0- $22,000 -0-
DES 2 1 0.85% 1.00% $500 $250
CODE -0- -0- -0- -0- -0- -0-
TEST 12 83 5.13% 83% $12,000 $83,000
Config -0- -0- -0- -0- -0- -0-
Warranty -0- -0- -0- -0- -0- -0-
PROD -0- 16 -0- 16% -0- $160,000
Inspection/ Train Cost
$11,346
Project A With Inspections vs. Project B Without Inspections
![Page 38: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/38.jpg)
Other ResultsFirst 5 pilot Inspections resulted in $5 million in cost avoidanceQT preserved after major reorganization; value recognized by new executivesInspections established as part of required methodologyContinued defect studies and comparisons of projects with and without inspections published; dramatic results
![Page 39: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/39.jpg)
What can you do?Conduct gap analysis with/without outside helpCompare to CMM, TMMExtract defects from test tool defect tracker, analyze, demonstrate unnecessary costsUse GSEP - Good Software Engineering Practice
Educate management, testers, PMs, others –show relevance Conduct follow-up analyses; show progressSHOW ROI BE A LEADER
![Page 40: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/40.jpg)
Evolution of Profound Knowledge
![Page 41: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/41.jpg)
Good Judgment comes from Experience
Experience comes from Bad Judgment
-- Anon
![Page 42: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/42.jpg)
CRISIS
Danger
Opportunity
![Page 43: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/43.jpg)
Knowledgemust come
throughAction
-- Sophocles
![Page 44: The Testing Improvement Story: Turning Defects into Dollars](https://reader030.vdocuments.us/reader030/viewer/2022032610/6238db63b2a732527259e143/html5/thumbnails/44.jpg)
Advantage Leadership, Inc.
Rebecca Staton-Reinstein, Ph.D., CSQAPresident
633 NE 167th Street, Suite 1015North Miami Beach, FL 33162, USA
Phone: 305-652-3466 Fax: [email protected]
www.AdvantageLeadership.com