what do we know about defect detection methods p. runeson et al.; "what do we know about defect...

What Do We Know about What Do We Know about Defect Detection MethodsDefect Detection Methods

P. Runeson et al.; "P. Runeson et al.; "What Do We Know What Do We Know about Defect Detection Methods?about Defect Detection Methods?", IEEE ", IEEE Software, May/June. 2006 Page(s): 82-90. Software, May/June. 2006 Page(s): 82-90.

SummarySummary

A survey of defect detection studies comparing A survey of defect detection studies comparing inspection and testing techniquesinspection and testing techniques

Yields practical recommendations:Yields practical recommendations: Use inspections for requirements and design Use inspections for requirements and design

defectsdefects Use testing for codeUse testing for code

Article Vocab ReviewArticle Vocab Review

Evidence-based software engineeringEvidence-based software engineering Determines which methods to use and for what Determines which methods to use and for what

purposepurpose EBSE involves relevant questions, surveying and EBSE involves relevant questions, surveying and

appraising available evidenceappraising available evidence Integrates and evaluates new practices in a target Integrates and evaluates new practices in a target

environmentenvironment Used as basis of article to help determine most Used as basis of article to help determine most

efficient and effective methods for defect detectionefficient and effective methods for defect detection

Article Vocab Review (con’t)Article Vocab Review (con’t)

DefectDefect Relates to one or more underlying faults in an artifact such Relates to one or more underlying faults in an artifact such

as codeas code Maps to single faults (faults and defect are used Maps to single faults (faults and defect are used

interchangeably)interchangeably) ArtifactArtifact

RequirementsRequirements DesignDesign CodeCode Using industrial artifacts improves the studies Using industrial artifacts improves the studies

generalizabiltygeneralizabilty

Types of DefectsTypes of Defects

Grammatical defects are much different from missing Grammatical defects are much different from missing requirements in a requirements specificationrequirements in a requirements specification

Testing and inspection methods may deem better or Testing and inspection methods may deem better or worse depending on the type of defectworse depending on the type of defect

Omission (missing) vs. Commission (incorrect)Omission (missing) vs. Commission (incorrect) Unimportant, unimportant, crucialUnimportant, unimportant, crucial Classify defects on origins :Classify defects on origins :

RequirementsRequirements DesignDesign CodeCode

Actors & TechniqueActors & Technique

Who is the reviewer or testerWho is the reviewer or tester Freshman student in an experimentFreshman student in an experiment Experienced software engineerExperienced software engineer

TechniqueTechnique Relative to inspection and testingRelative to inspection and testing Inspection – families of verification techniquesInspection – families of verification techniques Testing – structural (white box) & functional Testing – structural (white box) & functional

(black box)(black box)

PurposePurpose

What's the purpose of the inspection and What's the purpose of the inspection and testing activity?testing activity? Validation – assuring the correct system is Validation – assuring the correct system is

developeddeveloped Verification – assuring the system meets its Verification – assuring the system meets its

specificationsspecifications Both?Both?

Detection (for later isolation) vs. Isolation Detection (for later isolation) vs. Isolation (isolate the underlying fault)(isolate the underlying fault)

Defect detection activitiesDefect detection activities

Consider defect origins:Consider defect origins: Primary defect detection (1)Primary defect detection (1) Secondary defect detection (2)Secondary defect detection (2)

Evaluation criteriaEvaluation criteria

Efficiency - # of defects found per time unit Efficiency - # of defects found per time unit spent on verificationspent on verification

Effectiveness – the share of the existing Effectiveness – the share of the existing defects founddefects found

The Survey (finally)!!!!!The Survey (finally)!!!!!

Data from the studiesData from the studies

The issues covered from Table 2&3The issues covered from Table 2&3

Requirements defectsRequirements defects Design defectsDesign defects Code defectsCode defects Different defect typesDifferent defect types Efficiency versus effectivenessEfficiency versus effectiveness

Requirements defects – Thank god Requirements defects – Thank god for the experts!for the experts!

Different requirements inspection methods; Different requirements inspection methods; same conclusion:same conclusion: Spending effort up front to establish a good set of Spending effort up front to establish a good set of

requirements is more efficient than developing a requirements is more efficient than developing a system on basis of incorrect requirements and then system on basis of incorrect requirements and then reworking itreworking it

Design defectsDesign defects

Is inspecting design documents or testing the Is inspecting design documents or testing the implemented function more efficient?implemented function more efficient? Inspections were significantly more effective and Inspections were significantly more effective and

efficient than testingefficient than testing More than half of the defects (53.5 %) found More than half of the defects (53.5 %) found

during inspectionduring inspection Less found during testing (41.8 %)Less found during testing (41.8 %)

Bottom line : design inspection is the winner!Bottom line : design inspection is the winner!

Code defectsCode defects

Which is better? Code inspection or testing Which is better? Code inspection or testing (functional or structural)?(functional or structural)?

Which is more effective?Which is more effective? Which is more efficient?Which is more efficient? No clear winner No clear winner

We’ll see why next slideWe’ll see why next slide

Average effectiveness for code defect Average effectiveness for code defect detectiondetection

Average efficiency for code defect Average efficiency for code defect detectiondetection

Survey issuesSurvey issues Different defect typesDifferent defect types

Absence or presence of different types of defects might Absence or presence of different types of defects might affect efficiency and effectivenessaffect efficiency and effectiveness

Studies report defects by differing types and cannot be used Studies report defects by differing types and cannot be used to investigate whether the differences depend on fault type, to investigate whether the differences depend on fault type, not only the techniquenot only the technique

Frequency of defects vary widelyFrequency of defects vary widely Only a fraction of defects were found among studies – Only a fraction of defects were found among studies –

affects defect type frequency affects defect type frequency Classification schemes involve subjective judgmentClassification schemes involve subjective judgment

Conclusion: jury is out as to the effect of defect types Conclusion: jury is out as to the effect of defect types on the performance of inspection versus testingon the performance of inspection versus testing

Survey issues (con’t)Survey issues (con’t)

Effectiveness and efficiencyEffectiveness and efficiency Only 25 to 50 percent of defects found during Only 25 to 50 percent of defects found during

inspectioninspection 30 to 60 percent found during testing30 to 60 percent found during testing More than HALF defects remain!!More than HALF defects remain!!

Conclusion: Primary defect detection methods’ Conclusion: Primary defect detection methods’ low effectiveness and efficiency values is that low effectiveness and efficiency values is that secondary detection methods play a larger role secondary detection methods play a larger role that the surveys concludethat the surveys conclude

The results are in!The results are in!

For requirements defects, no empirical evidence exists at all, but the fact that costs for requirements inspections are low compared to implementing incorrect requirements indicates that reviewers should look for requirements defects through inspection.


For design specification defects, the case studies and one experiment indicate that inspections are both more efficient and more effective than functional testing.


For code, functional or structural testing is ranked more effective or efficient than inspection in most studies. Some studies conclude that testing and inspection find different kinds of defects, so they’re complementary. Results differ when studying fault isolation and not just defect detection.


Verification’s effectiveness is low; reviewers find only 25 to 50 percent of an artifact’s defects using inspection, and testers find 30 to 60 percent using testing. This makes secondary defect detection important. The efficiency is in the magnitude of 1 to 2.5 defects per hour spent on inspection or testing.

Results Summary TableResults Summary Table

Questions ???Questions ???

what do we know about defect detection methods p. runeson et al.; "what do we know about defect...

Documents