experiences in root cause analysis and defect prevention ... · – an advisory is an informational...

Copyright 2003 -- Raytheon Company,

Experiences inRoot Cause Analysis and

Defect Prevention MethodsKelly L. Lanier

RaytheonNetwork Centric Systems

[email protected]


Case for Action

Output of a R6SProject

Characterization

Deliver Quality and Reduce Costly Rework!

Six Sigma Specialist Project thatfocused in reducing escaping defects.Design defects that escape toIntegration are 10 times moreexpensive than if they were caughtin the Design stage.Another thing to note is that one of thebiggest problems we have found with cost andschedule is in integration.


Set the Foundation

� Define and agree to the the following:– Defect

�A flaw or imperfection that results in incorrect software. Adefect may or may not be detected during software use.

�A deficiency which has the potential of producing incorrectresponse orundesired effect.

– Comment Type�Assignment, checking, performance, etc.

– Comment Priority�1 through 5


When to Count a Defect

� Count a defect if:– The requirements documentation could lead to

incorrect source code�By being incorrect�Or by leading to incorrect design�Etc.

– The design documentation could lead to incorrectsource code�By being incorrect itself�Or by being easy to misunderstand

– The source code is incorrect


Defect Analysis and Prevention

� Defect Analysis is the process of analyzing adefect to determine its root cause.

� Defect Prevention is the process of addressingroot causes of defects to prevent their futureoccurrence.


Process Focus

� Defect Containmentfocus– Finding defects in the

stage they wereintroduced and asearly in the lifecycle aspossible

– Eliminating escapingdefects

� Defect Preventionfocus– Preventing the

occurrence of anindividual defect orgroup of defects

Requirements Design Code & TestSW

Integration Qual TestSYS

IntegrationPost-Release Totals

Requirements 3 3

Design 1 15 16

Code & Test 2 10 45 57SW

Integration 1 2 12 20 35

Qual Test 0 0 5 0 5 10SYS

Integration 2 3 2 2 0 2 11

Post-Release 0 1 2 0 0 0 5 8

Totals 9 31 66 22 5 2 5 140

Stag

e D

etec

ted

Stage Originated

Defect Prevention

Defect Containment

Focus on

AND


Get DetailedDefect Datafrom DefectLogger and

Synergy

PerformCausal

Analysis

Identify RootCauses andSolutions

Next Priority

Defect Analysis andPrevention

Select Defects forFurther Analysis

ContinuousImprovement

DevelopAction Plan

andMeasurementfor Success

ImplementAction Plan

SubmitImprovement

toOrganization

Identify andPrioritize

ImprovementOpportunities

Select Defects

CCBs, Peer Reviews,Defect Containment

MetricPareto Charts

MonitorAgainstPlanned

Measurement

Re-

eval

uate

Defect Analysis and PreventionProcess



� The program metrics analysis team regularlyreviews defect data to determine if defectanalysis is necessary– Defect metric has exceeded threshold– Defect data shows a trend– Individual defect is flagged for analysis


Select Defects


MetricPareto Charts


Select Defects


MetricPareto Charts


Analysis Tools

FishboneFishboneDefect Containment MatrixDefect Containment Matrix

Five WhysFive Whys

ParetoPareto

Requirements Design Code & TestSW

Integration Qual TestSYS

IntegrationPost-Release Totals

Requirements 3 3

Design 1 15 16

Code & Test 2 10 45 57SW

Integration 1 2 12 20 35

Qual Test 0 0 5 0 5 10SYS

Integration 2 3 2 2 0 2 11

Post-Release 0 1 2 0 0 0 5 8

Totals 9 31 66 22 5 2 5 140

Stag

e D

etec

ted

Stage Originated

? ? ? ? ?

http://homext.ray.com/sixsigmahttp://homext.ray.com/sixsigma

Get DetailedDefect Datafrom DefectLogger and

Synergy

PerformCausal

Analysis



http://homext.ray.com/sixsigma


ContinuousImprovement (1 of 3)

� Identify and prioritize improvementopportunities based on the potentialsolutions to defect root cause

� Submit improvement opportunities to theorganization via the OrganizationalImprovement Website

Next PriorityContinuousImprovement

DevelopAction Plan



SubmitImprovement

toOrganization




Measurement


� Organizational Improvement Website– A proposal is a request to have the SWEC

organization evaluate, select, and adopt arecommended improvement and to provide a fundingand evaluation path for the proposal.

– An advisory is an informational message to the SWECorganization advising that a program is providing afunding and evaluation path for an improved process,method, tool, technology, etc to support its ownbusiness/project goals. However, this improvementmay have relevance to the organization at large andhave strategic importance to the enterprise



DevelopAction Plan



SubmitImprovement

toOrganization




Measurement


� Work as a Six Sigma Project– Create an action plan for the improvement

�Include how to measure success of theimprovement

– Implement the action plan– Monitor the progress of the action plan

�Track progress for measurementof success

– Communicate the results to theprogram & organization



DevelopAction Plan



SubmitImprovement

toOrganization




Measurement


Defect Prevention

� Examples– New checklists– Modified processes– Updated plans– Scheduled training– Tool support


DevelopAction Plan



SubmitImprovement

toOrganization




Measurement


Defect PreventionNext PriorityContinuousImprovement

DevelopAction Plan



SubmitImprovement

toOrganization




Measurement

Check to mark a defect

Check to analyze


Organization Defect Analysis andPrevention Process

Level 5 Behaviors

Get DetailedProgram

Defect Data

PerformCausal

Analysis


Next Priority



ContinuousImprovement

ExperimentInstitutionalize



SubmitImprovement

Proposal

Select Defects

Out of Threshold OrgDefect Metric, Quarterly

Process CapabilityAnalysis

Monitor

Re-

eval

uate

Program Defect Analysis and Prevention


Results

SWEC Defect Containment Trend

81%84%

50%

60%

70%

80%

90%

100%

Feb

01M

ar01

Apr 0

1M

ay01

Jul 0

1Se

p01

Dec

01

1Q02

2Q02

3Q02

4Q02

1Q03

2Q03

3Q03

4Q03

Upper DROV Nominal value Lower DROV Mean Performance


Defect Density

Improved Defect Density by 44 percentage points, and reduced variation by 31%

SWEC Defect Density TrendFe

b01

Mar

01Ap

r 01

May

01Ju

l 01

Sep

01D

ec01

1Q02

2Q02

3Q02

4Q02

1Q03

2Q03

3Q03

4Q03

Upper DROV Nominal value Lower DROV Mean Performance

CMM Level 4June 2001

CMMI Level 5September 2003


Summary

� Out of phase defects cause expensive rework� Use common definitions and counting approach� Focus on defect detection and prevention� Analyze metrics at least monthly using R6σ tools� Prioritize and implement defect prevention

activities� Promote lessons learned to the organization

experiences in root cause analysis and defect prevention ... · – an advisory is an informational...

Documents