05 26 2011 metrics for testing
TRANSCRIPT
-
7/31/2019 05 26 2011 Metrics for Testing
1/20
http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
2/20
Metrics for Testing: Managing with Facts
By using metrics, we can manage withfacts and reality
How can we use metrics to manage testing?
What metrics can we use to measure the testprocess?
What metrics can we use to measure ourprogress in testing a project?
What do metrics tell us about the quality ofthe product?
Lets see if we can find some answers
Metrics for Testingwww.rbcs-us.com
Copyright (c) RBCS 2005-2011 Page 2
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
3/20
Why Metrics?
Managing with metricsfor testing oranythingallows us to manage withfacts
Subjective, uninformed opinions are nota sound basis for management
What sounds reasonable can be wrong(e.g., Aristotle)
Metrics for Testing Page 3www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
4/20
Why Testing Metrics?
Testing by itself has no valuebut it producespotentially valuable information
Test information must be generated and
communicated effectively to be valuableCommon communication goalsNotify (We have 24 bugs remaining to close)
Enlighten (See the time lost due to reopens)
Influence (We propose a bug triage meeting)
The metrics should be used as part of regularstatus reports (dashboards) as well as periodicanalyses
Metrics for Testing Page 4www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
5/20
How to Develop Metrics?
Identify objectivesCreate questions about effectiveness,efficiency, and elegance of achievement of
objectivesDevise a measurable metric (or find asurrogate metric) to address the questionSet a goal for each metric
Implement improvements to improve goalLets look at types of metrics andexamples
Metrics for Testing Page 5www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
6/20
Example
Objective: Find defects
Effectiveness question: Have we
finished finding new defects?Metric: Track trend in defect discoveryduring testing
Goal: Watch for convergence (morelater)
Improvement: Find bugs earlier
Metrics for Testing Page 6www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
7/20
Example
Objective: Build confidenceNote that we need a surrogate metric forconfidence; one way is to measure coverageCoverage has multiple dimensions
Effectiveness question: Do any requirements haveidentified failures?Metric: What % of requirements are completelytested without any failures, what % requirementshave failures, what % are untestedGoal: 100% requirements tested without must-fixfailuresImprovement: Reduce % of requirements that failin testing
Metrics for Testing Page 7www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
8/20Metrics for Testing Page 8
Good Metrics
Define a useful, pertinent, and conciseset of quality and test metrics
Avoid too large a set of metricsToo difficult to measure later
Confusing to participants
Ensure uniform, agreed interpretations
of these metrics, to minimize disputesand divergent opinions about measuresof outcomes, analyses, and trends
www.rbcs-us.comCopyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
9/20Page 9
Tracking Metrics
Automated tool support can reduce the time requiredto capture, track, analyze, report, and measuremetrics
Apply objective and subjective analysis for specificmetrics over time, especially when trends emergethat could allow for multiple interpretations ofmeaning
Be aware of and manage the tendency for peoplesinterests to affect the interpretation they place on aparticular metric or measure
Metrics for Testingwww.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
10/20Page 10
Reporting Metrics and Measurements
Reporting of metrics and measures should enlightenmanagement and other stakeholders, not confuse ormisdirect them
Good testing reports based on metrics should be
easily understood, not overly complex or ambiguousGood testing reports based on metrics should helpmanagement improve processes, guide the project tosuccess, and manage product quality
Metrics include snapshots of status at a moment intime, trends emerging over time, and analysis ofcauses and relationships between factors thatinfluence testing and quality outcomes
Metrics for Testingwww.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
11/20
Process Metrics for Testing
UsesTo measure the effectiveness, efficiency, andelegance of the test processTo provide the insights to guide whereprocess improvements should occur
Part of certain test process assessments(e.g., using CTP framework)Not used as frequently as should be thecaseNot typically part of any dashboard (butcan be)
Metrics for Testing Page 11www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
12/20
Developing Good Process Metrics
Determine key objectives for the testprocessTypical objectives
Finding bugs, especially important bugsReducing quality risk
Decide which important effectiveness,efficiency, and elegance questions apply
Create metrics that address thosequestionsSet goals that reflect current or desiredcapability
Metrics for Testing Page 12www.rbcs-us.com
Copyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
13/20
Bug Finding Effectiveness
Objective: Finding bugsQuestion: What percentage of bugs found?Metric: Defect detection percentage
Goal: A typical number is 85%, thoughsome teams do much better
Metrics for Testing Page 13
presentbugs
detectedbugs=DDP
bugsproductionbugstest
bugstest)(
+
=testingforDDP
www.rbcs-us.comCopyright (c) RBCS 2005-2011
http://www.rbcs-us.com/http://www.rbcs-us.com/ -
7/31/2019 05 26 2011 Metrics for Testing
14/20
Bug Finding Focus
Objective: Finding important bugs
Question: Do we find more important
bugs than less important bugs?Metric: Relation of DDPs (see below)
Goal: Relationship shown below is
always true
Metrics for Testing Page 14
)()( bugscriticalDDPbugsallDDP