the good, the bad, and the metrics
DESCRIPTION
Has your organization ever considered replacing a tester that did not write, for example, 15 test cases per day? Is the testing team blamed if defect leakage is greater than 5% into production? What drives decisions like these? The common thread in these examples is “Test Metrics” Test Metrics... Everyone has an opinion about them. Some believe they are the most valuable way to communicate the results of testing. Some think that they are useless, misleading, and damaging to the communication of test results. Some believe that without measurement you are not managing the effort. And some believe that bad metrics are worse than no metrics at all. Where does your organization fit in the metrics and measurement debates? Is your team aligned? Do you agree with the team? Do you use a reporting process for test results? Are you forced to report on metrics you don't believe are valuable? Do you have dozens of metrics that you are reporting periodically that no one looks at, and when they do look at them, there is room for misinterpretation? In this session, Mike Lyles and Jay Philips will challenge the audience to discuss the topic of metrics and measurement, review multiple viewpoints on the topic, and address many of the questions that organizations have today around metrics and measurement. Takeaways: - Top metrics that are misused or misunderstood in most every organization. - Metrics that you should you get rid of ASAP! - Best and Worst metrics - based on opinions of the speakers & audience. - Metrics that everyone should use – and how they compare to your organization’s metrics. - Tools and processes that can help your organization better measure your testing. ** Presentation given at STPCon Spring 2014TRANSCRIPT
Session 205The Good, The Bad,
and The Metrics
Jay Philips & Mike Lyles
1
About Us
Jay Philips - CEO & President of Project Realms, Inc. an IT consulting firm focusing on software quality. Jay is also the CEO & President of TeamQualityPro, which is a real-time integrated dashboard platform used to evaluate the entire ecosystem of application projects and resources. TeamQualityPro is code agnostic so all organizations can implement a real-time executive dashboard system.
Mike Lyles - Sr. QA Manager with 20+ years of IT experience. He has held multiple roles in QA, from functional testing to test environments and data management. He is currently leading Performance testing, Test Automation, and Service Virtualization for all business communities within his organization.
22
What is your definition of metrics?
33
What Is Your Opinion?
• Some believe they are the most valuable way to communicate the results of testing
• Some think that they are useless, misleading, and damaging to the communication of test results
• Some believe that without measurement you are not managing the effort.
• Some believe that bad metrics are worse than no metrics at all.
44
55
How is Your Organization Using Metrics Today?
• Is your team aligned? Do you agree?
• Do you use a reporting process for test results?
• Are you forced to report on metrics you don't believe are valuable?
• Do you have dozens of metrics that you are reporting periodically that no one reads?
66
77
Do you understand your metrics?
88
99
Discussion Topic #1
The Good,
The Misunderstood,
and
The Bad Metrics…
1010
Comparing the Survey Results
Good Metrics Misunderstood Metrics Bad Metrics
Cost to fix defects Cost per detected defect Cost per detected defect
Defect open vs. close trends Defect counts Defect counts
Test Case Pass / Fail / Blocked %
Pass / fail ratio Pass / Fail %
Progress – status of execution, plan vs. actual, project progress
Test Progress (plan vs. executed)
Test cases executedEffort / schedule variance
Resource productivity Defect by developer
Defects entered by testerExecution by tester
Cases executed by testerDefect count by tester
Defect AgeDefect Fix %
Defect resolution timeDefect Removal Efficiency
Defect Leakage Defect LeakageTest Coverage Test Coverage
Developer vs QA Ratio Developer vs QA Ratio
Test case count Test case count
1111
Interesting Survey Quotes
Good“Team intuition as to whether the software is fit for release”
“Any metric that fits the context, provides insight in the state of the software, shows risks to project goals and schedules and allows the management to make good, well-informed decisions”
“I can't think of any metric that every organization should be using. I don't think these exist.”
“Tell stories, don't just dish out numbers!”
Misunderstood“Cost of testing phase wise”“Comparing pass rates between projects”“Pretty much all of them.”
1212
Interesting Survey Quotes
Bad“Ones that do not provide value in making decisions or determining risk”
“Any metric that is not going to be used for improving defect slippage, productivity, or Cost of Dev/testing”
“Everything other than Customer Happiness”
“All of them, except the ones used by a team to gather feedback for themselves”
“Any metric used out of context”
“Anything related to test cases”
1313
Good? Misunderstood? Bad?
1414
Good? Misunderstood? Bad?
1515
Good? Misunderstood? Bad?
1616
Good? Misunderstood? Bad?
1717
Discussion Topic #2
What tools do you use for metrics?
1818
The Metric Tools…
1919
The Metric Tools…
Excel HP QC / ALMJIRA In-House Automated ToolAssure TotalView MS TFS with Cube for AgileRally Microsoft Test ManagerSharepoint GITIBM Rational Clear Quest I-DashJenkins / Junit Mingle – for team reportsSite Scope SmartBear ALMSonar Test DirectorVersion One Zephyr
2020
Discussion Topic #3
How are metrics in your
organization validated / used
as actionable data?
2121
Metrics Should Not Be One-Sided
2222
6 Steps to Creating Metrics that Matter!
2323
Getting Started
Does your current set of metrics allow you to see:•The ability to see resource shortages and surpluses•The ability to see funding shortages and surpluses•The ability to see objectives being met and when
Can your metrics pass the following criteria?•Does it have a direct/indirect impact on the revenue?•Does it have a direct/indirect impact on costs?•What is the action that you will take based on the result?
2424
What to Measure
2525
Step 1: Define the Business Model
•Do you know the overall strategy for your department or organization?
•Determine the facilitators and business results for each item
2626
Step 2: Determine How to Execute
•Most organizations have multiple domains (IT, Marketing, Operations, etc).
•Review each area and determine how that domain can help meet the organization’s strategy.
2727
Step 3: Link to the Organization’s
Strategy / Roadmap
•Determine which items are Customer Focused and which are Cost Related
2828
Step 4: Create Key Performance
Indicators (KPI’s)
•Focus on efficiency (cost & productivity) as well as effectiveness (quality & value)
•Evaluate the metrics in context to determine which areas are doing well
2929
Step 5: Implement A Process That Can Be Reproduced
•Move from a manual process to an automated process
•Determine the velocity (hourly, daily, weekly)
•Team can now focus on objectives
3030
Step 6: Communicate Metrics and KPI’s
•Communicate what is being measured, how, and when data is collected
•Communicate expectations on how the metrics should be used
•Determine action steps
3131
6 Steps - Conclusion
• Verify your metrics bring value
• Verify the metrics can be reproduced
• Verify the metrics are available when you need them
• Verify the business objectives and KPI’s have been communicated
3232
Contact Information
Jay PhilipsCEO, Project Realms, Inc.
CEO, TeamQualityPro, [email protected]
http://www.ProjectRealms.com
http://www.TeamQualityPro.com
Twitter: @jayphilips, @ProjectRealms, @TeamQualityPro
http://www.linkedin.com/in/jayphilips
www.projectrealms.com
www.teamqualitypro.com
3333
Contact Information
Mike LylesSr. QA Manager
Lowe’s Companies, [email protected]
www.mikewlyles.com
http://about.me/mikelyles
Twitter: @mikelyles
http://www.linkedin.com/in/mikewlyles
www.mikewlyles.com
http://about.me/mikelyles
3434
Thanks for Attending
Now for the metrics that are important To US!!
Please fill out an evaluation form and drop it in the collection basket located at the back of the room.
3535