Download - Defect Prevention Training
Protection notice / Copyright notice
Defect Prevention Training Induction – Sep 2007
Version 2.0
Page 2 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Introduction
Defect Prevention is a process of improving quality and productivity by preventing the injection of defects into a software work product.
Definition: “…an activity of continuous institutionalized learning during which common causes of errors in work products are systematically identified and process changes eliminating those causes are made.” [Eickelmann]
SEI has identified ‘Causal Analysis and Resolution’ as Level 5 PA of CMMI
Page 3 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Objectives
Course ObjectiveTo enable participants understand and apply defect prevention concepts
Defect Prevention Objectives
Identify and analyze the causes of defects& Reduction in number of defect categories
Reduction in the extent of defect escape between phases
Reduction in frequency of common defects
Improvement in PCB values
Page 4 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Contents
Defects and Bugs (Examples)Origin of DefectsClassification of DefectsDefect ManagementDefect DetectionDefect Prevention Cycle
Page 5 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Objectives of Defect Prevention
• Establish practice of Root Cause Analysis within projects for Analysis
of Identified Defects
• Identify critical processes as part of root cause analysis
• Set goals for improving critical process (shift mean and narrow
variation)
• Reduce most frequent type of defects such as “ not following coding
guidelines”
• Analyze opportunities for improvement by conducting escape analysis.
• Use defect distribution data to drive process improvement activities
• Spread lessons learnt - Team Meetings, SEPG, Process Database
Page 6 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Responsibility
Project team is responsible for the DP activities pertaining to
the project life cycle activities &
Project Manager (at project level)
Project Quality Manager (at project level)
P&Q (at Org level)
SEPG (at Org level)
Page 7 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defects and BugsDefects and Bugs
Page 8 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defects and Bugs - Example
The Atlas-Agena spacecraft, destined for Venus, had to be blown up during launch because it became unstable about 90 miles up. (Malfunctioning rockets have to be destroyed to avoid crashes in populated areas). A missing hyphen in the flight plan resulted in the loss of the $18.5 million US spacecraft.
Page 9 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defects and Bugs - Example
On January 15, 1990, 114 switching computers in the AT&T telephone network crashed because of a software flaw. 65 million subscribers were unable to use their phones. The problem arose when a switching computer in New York crashed, sending out a digital “out of service” message to nearby computers. Normally, other switches would route traffic around the disabled computer. However, a misplaced “break” in a C statement caused the nearby computers to go down as well. For the next 9 hours, the switches went down, rebooted themselves, and came back up, only to go back down immediately.
Page 10 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Effect of Software Error
•unreasonable added cost•lost time and effort•inconvenience and annoyance•death
Page 11 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Could these incidences of software errors been prevented?
YES!
Page 12 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Product and Process Defects
PRODUCT PROCESS
Definition
Types of Defects
Strategy for Handling Defect
Artifacts created during the life cycle of the project.
Complete set of activities needed to transform user requirements to a product.
Process Defects are related to tasks/activities: Non-adherence to standards Poor Documentation Schedule overrunTraining related
Product Defect are related to requirements : Functional and Non-Functional
Defect PreventionDefect Removal/Elimination
Product defect is always a result of Process Defect
Process defect is like a potential carrier of disease
Page 13 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Stages of a Software Cycle
ReviewsReviewsWhite BoxBlack Box
Stress/Load
White BoxBlack Box
Stress/Load
ProblemReports/CRs
ProblemReports/CRs
Defect Prevention - Feedback and Process adjustmentsDefect Analysis and Process Improvement
Defect Prevention - Feedback and Process adjustmentsDefect Analysis and Process Improvement
RequirementsRequirements DesignDesign CodingCoding TestingTesting MaintenanceMaintenanceProposalProposal
Page 14 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Origin of DefectsOrigin of Defects
Page 15 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Human Errors
Types of Errors
Omission
Ignorance
Commission
Typography
Knowledge
Information
External
More than 80% of software errors are human
Page 16 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Translation Errors
A requirement is often stated in terms of a solution
Focus on solution may hide the real requirement
The mismatch between the solution desired and the real requirement leads to translation errors
NEEDED TOLD URS DESIGN
TRANSLATION ERRORS
BUILD
Detected Bugs
Hidden Bugs
Page 17 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Design Errors
Errors that affect data integrityErrors that alter correctly stored dataIncorrect algorithm used to compute a value
Types of Errors
Some examples:
•Does each module in the system design exist in detailed design?•Are all assumptions explicitly stated? Are they acceptable?•Have the exceptional conditions been handled?•Are all data formats consistent with the system design?•Are the loop termination conditions properly specified?
Checklist
Mitigation of design
errors
Page 18 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Coding Errors
Exception handling
Incorrect Algorithm
Missing Functionality
Language pitfalls
Memory release
Omitted program sections
A programming error alters a program’s ability, in a negative sense, to completely and effectively meet the user’s requirement.
Page 19 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Testing Errors
Failure to notice a problem
Misreading the screen
Failure to execute a planned test
Failure to use the most ‘promising’ test cases.
Ignoring programmers suggestions
Corrupt data file used
Incorrect test cases
Concentration on trivial
Failure to report
Page 20 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
0
20
40
60
80
100
120
140
RS Revi
ew
FS Rev
iew
Desig
n Rev
iew
Code Rev
Testin
g
Proble
m R
eport
Ideal
Acceptable
Costly
Disaster
Defect Detection as early as possible
High proportion defect reported by customer - Unacceptable
More defect detected in testing – Quality at High cost
Maximize defect detection during reviews- Quality at right
cost
Objectives of Defect Prevention
Page 21 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
So where should the focus be?
On Proactive Defect Prevention
On Early Defect Detection
On Usage of Past Experience
Page 22 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defect PreventionDefect Prevention
Page 23 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defect Management Rules
•Fixing downstream is costly…
•Rework eats away resources...
•Pareto Rule - About 80% of the available rework comes from 20% of the defects
•Another Pareto Rule - About 80% of the defects come from 20% of the modules (and about half the modules are defect free)
•Peer Reviews catch 60% of the defects
•Perspective-based reviews catch 35% more defects than non-directed reviews (use of checklists)
•Disciplined personal practices can reduce defect introduction rates by up to 75%
•About 40-50% of user programs enter with trivial defects
Barry BoehmVictor Basili
Page 24 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Activities Performed during DP – Organization Level
• Defect Prevention Plan is prepared at SBU-level (by TC/Business Partner) – contains lessons learnt and improvement actions
• SPI (Software Process Improvement) Plan is an overall Plan for the Organization and bears reference to DP Plan as well
• DP Plan is shared with SEPG members and PQMs
• Lesson’s Learnt from SBU is disseminated through SEPG
• Improvement Action are tracked and DP Plan is updated accordingly
Page 25 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Activities Performed during DP – Project Level
•Kick-off/Start PES meeting – •Goal setting, • Identification of critical processes• Incorporation of Lessons Learnt from previous projects as preventive measures• Phase-wise Defect Distribution goal setting
• Defect reporting – Reviews & Testing
• Root Cause Analysis
• Action Implementation
• Information Dissemination
Page 27 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Causal Analysis Cycle
Test LogsLODs
Classify Defects (Type, Injected &
Detected Phase)
Identify Top 80% Defects for RCA
Prj RCAReport
And select all high impact defects
Perform Fishbone AnalysisUsing Potential Causes
Arrive at Root CauseAnd Action List
Application
Reviews/Testing
Preventive Feedback
Page 28 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Pareto Chart
What is a Pareto ChartBar chart arranged in descending orderBars on the left are more important than those on the right Separates the “vital few” from the “trivial many”
Uses of Pareto ChartBreaks a big problem into smaller piecesIdentifies most significant factors (80-20 rule)Shows where to focus effortsAllows better use of limited resources
Page 29 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Pareto Chart - Exercise
Participants to discuss possible Code Review Defects
Classify the defects under different categories/types and assign a number of defects against each
Prepare a Pareto Chart using Excel to focus on the most significant defects (80-20 rule)
Page 30 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Pareto Chart - Code Review
0
1
2
3
4
5
6
7
8
Logi
cal e
rror
Red
unda
nt c
ode
Rem
ove
debu
ggin
g co
mm
ents
Com
men
ts n
ot e
xhau
stiv
e
Impl
emen
tatio
n er
ror
Fun
ctio
nalit
y m
issi
ng
Cod
e re
adab
ility
Hea
der
inco
mpl
ete
Wro
ng p
aram
eter
s pa
ssed
Nam
ing
conv
entio
n no
t fol
low
ed
Nu
mb
er o
f D
efec
ts
0
20
40
60
80
100
120
% o
f D
efec
ts
No. of defects
Percentage
Pareto Chart (contd.)
Page 31 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Cause & Effect Diagram - Fishbone
What is a Cause & Effect Diagram?A graphic tool that helps identify, sort and display possible causes of a problem or quality characteristic
Benefits of CEDDetermination of root causesEncourages group participationIndicates possible causes of variationUses a orderly, easy-to-read formatIdentifies areas for collecting data
Page 32 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Cause & Effect Diagram
• Decide the “Effect” to examine
• Identify the main categories
• Identify as many causes or factors as possible and attach them as sub branches of the major branches
• Identify increasingly more detailed levels of causes by asking a series of why questions
• Look for causes that appear repeatedly. These may be root causes
• Identify and circle the causes that we can take action on
Page 33 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Driven toofast
PoorMileage
Method
MaterialsMan
Machinery
Wronggears used
Carburetor needsadjustment
Under inflated tires
PoorDriving habits
Wrong Octanegas used
ImproperLubrication
Poor Maintenance
Cause & Effect Diagram
Page 34 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Exercise on RCA
Make Groups
Assign a PM to each group
Brainstorm and prepare a cause and effect/fishbone analysis
Present the result
(20 mins)
Page 35 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Defect Estimation
Phases Defect DistributionRequirement 7%Design 14%Coding & UT 49%IT/ST 27%AT 3%
Proposed Goal9%
16%44%29%2%
Design14%
Coding & UT49%
IT/ST27%
AT3%
Requirement7% - Use Historical Data
- Focus on Business Objectives and Process Improvement - Set more Challenging Goals
Page 36 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Applicability of RCA
•Defects•Customer Feedback•Non-conformance (NC)•Process Capability Baselines•Major Issues (that impact cost, quality, schedule)
Page 37 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Example
Coding & UT
People Measurement
Hardware/Software/ Tools
Support/ Guidance
Process/Standards
Guidelines not followed
Guidelines notupdated
Not aware
DEFECTS AT END OF CODING & UT
Assets not available
Not CommunicatedLong OverdueNot Available
Not Trained/Inadequate resources
Not adequateTool not inspected
Data not adequate
Page 38 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Common Defect Types
Defect Types Examples
Function/Class/Object error is one that affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture, or global data structure(s) and should require a formal design change.
Database design/modeling error, functionality not working, etc
Assignment error indicates a few lines of code, such as the initialization of control blocks or data structure.
Oversight during coding, initialization of parameters/variables, incorrect setting of variables, java script validation, etc
Interface/Messages corresponds to errors in interacting with other components, modules or device drivers via macros, call statements, control blocks or parameter lists.Checking addresses program logic, which has failed to properly validate data and values before they are used.
Incorrect validation, missing validation, error handling, return value not checked
Timing/serialization errors are those, which are corrected by improved management of shared and real-time resources. Build/package/merge describe errors that occur due to mistakes in library systems, management of changes, or version control. Incorrect packaging, Setup problem, etc
Page 39 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Common Defect Types (contd.)
Defect Types Examples
Documentation errors can affect both publications and maintenance notes. Unclear specifications, standards not followed,
redundant code, GUI errors, incorrect description, ambiguous description, etc
Algorithm errors include efficiency or correctness problems that affect the task and can be fixed by (re)implementing an algorithm or local data-structure without the need for requesting a design change.
Hard coded values used, data type mismatch, etc
External Environment errors that occur due to factors that are outside the application scope.
Test data, test drivers, other tool defects, support system, concurrent work, inherited from previous release, third party software dependency, etc.
Performance errors affect the performance of the system.
Memory not released, Web session timeout not handled properly, browser cache related problems, etc
Database errors are related to errors in database or scripts.
Integrity constraint violated, SQL statements not tuned, Error in SQL statement, etc
Trivial/MinorTypo/minor errors in documentation, rephrasing, extra information in document, etc
Page 40 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Common Root Causes
Major Defect Categories (from Fishbone) Root Cause
Support/Guidance (e.g. Management Support, Training, etc)Handover (Change Coordination)Inadequate training (QMS, Defect Prevention, technical)
Process/Standards Guidelines/Standards/Procedures not updatedInadequate Process for Handling Requirements/DesignChange in Requirements/Design
People Breakdown of communicationsLack of knowledge (domain/system/tool)Oversight
Hardware/Software/Tools Configuration related problemInadequate tools
Measurements Incorrect analysis of data
Page 41 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Escape Analysis
(Requirements defects getting slipped to next phases of Life cycle)
Phase Detected RequirementsDesignCode Review UT IT ST AT Total
Phase InjectedRequirements 10 9 1 1 2 3 26Design 16 3 1 1 2 23Code review 24 20 16 6 66UT 0IT 2 2ST 3 3AT 0Total 10 25 28 22 21 14 0 120
Slippage (from Requirements to Design = 9/19) 47%DRE of Requirements Phase (10/19) 53%Slippage (from Coding Phase to subsequent phases = 52/80) 65%DRE of Code Review = 28/80) 35%
Defects Found during Code Review 28Defects slipped from Coding (not considering those injected in IT & ST) 52Defects slipped into Coding Phase 4TOTAL (Coding + subsequent Phases) 80
Page 42 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Root Cause & Action Planning
Root Causes of Defects for <Month_1>, <Phase_1> <Month_n>, <Phase_n>
Special Causes:(Root causes of high impact defects)Common Causes:(Root causes of high occurrence defects)
Action PlanImplementation Technique
Monitoring technique
PriorityAssociated Risk
Impact on PCB *
Expected Date of Closure
Person Responsible
StatusActual Date of Closure
Team to be given training on Domain Knowledge
Workshop every Friday by each team member in turn
PM to ensure that training is held
HighDefects in software
Schedule Slippage
20th June 2004
PM Open
Page 43 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Some Lessons Learnt
1Integration Testing should be scheduled so that the core modules are initially tested.
2 Client should be given overview for SISL's P&Q processes
3Client responsibilities should be clearly communicated in the beginning of the project.
4Design documents should contain all the necessary validations to avoid validation error
5 Checklist should be used to avoid GUI errors6 Rigorous unit testing to be done to avoid logical errors.
7Basic level review and testing should be done at developer's level before handing over the code to the testers and reviewers.
8 Test cases should be formed with test data.
Page 44 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Recap…
Defects and Bugs (Examples)Origin of DefectsClassification of DefectsDefect ManagementDefect DetectionDefect Prevention Cycle
Page 45 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Any Questions?
Page 46 Sep-07Protection notice / Copyright notice
For Internal Use OnlyP&Q
Thank You