infostretch - business driven quality metrics: beyond counting defects
DESCRIPTION
Inculcate metrics based optimization into your business process without losing sight of business needs and client focus. Defining quality metrics is one of the fundamental and most important tasks for QA managers. However, business goals and customer experience often get neglected while identifying QA metrics because of the silos QA teams work in. Quality assurance is not just about testing and finding bugs, it is in fact, crucial to the success of the business itself. But again, too much focus on alignment with business goals while defining QA metrics can put excess stress on the QA team and adversely affect their productivity. If for example, the prime business goal is customer satisfaction, then finding customer defects is more important than the cost of finding those defects. While ‘customer defect metrics’ is important in this scenario, time taken to find these defects can’t be ignored. So, productivity related metrics still need to be considered. How do we strike the right balance between defining QA metrics that are in line with business goals without overburdening the QA team? In this webinar, Infostretch CEO Rutesh Shah addresses the above question and provide a comprehensive approach along with tools that will help you inculcate metrics based optimization into your business process without losing sight of business needs and client focus. In this webinar, Rutesh focuses on following : Defining business driven metrics – Going beyond SDLC - Industry focused metrics, Accounting for competition; Benchmarking success metrics - Capturing business goals like costs, product leadership, operational efficiency into QA metrics End-user/customer focused metrics - Metrics driven actions Watch this webinar if you want to learn about how your QA team can become a significant contributor to the business baseline and goals.TRANSCRIPT
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.All trademarks are the property of their respective owners.©2004-2010 InfoStretch Corporation. All rights reserved.
Business Driven Quality MetricsBeyond Counting Defects
March, 2011
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
1. Introduction
2. Types QA Metrics and Examples
3. Business Driven Metrics – Going Beyond SDLC
Industry Focused, Competitive - Success Metrics
Cost, Timelines, Effectiveness – Optimization Metrics
End-User / Customer focused Metrics
4. Metrics Driven Actions
5. Q & A
Agenda
INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]©2004-2010 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah
©2004-2011 InfoStretch Corporation. All rights reserved.
InfoStretch OverviewElevating Confidence
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Organization HQ in Silicon Valley Offices in New York, Seattle 105+ Clients and 160+ Projects Strong contributor to QA Community
People Average 6 years experience 300+ Engineers/Consultants Industry Expertise in Finance,
Retail, Mobile and Healthcare Strong Project Mgmt. team
Infrastructure Extensive Platform
Compatibility and Performance Test Labs
2000+ Mobile Handset Test Lab ISO 9000-2001 Certified High Availability and 24 hours
secure offshore centers with ISO 27001 Audited
Overview
INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.
Partners
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
RecognitionUnique Perspective – Leadership Team with Entrepreneurial, Startup, Fortune 500, Top Technology & Big 4 Consulting Firms Experience.
Long Term Relationships – Focus on building long term relationships and competencies around client 100% reference-able. Clients including T-Mobile, AT&T, Amazon, Intel, TNS, Harmony & Wells Fargo
Strong Growth – Strong Year over Year Growth and Profitable Operation Since 2004. Significant Investments in Platform Compatibility and Device Labs
(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]
19th among Fast growing
privately owned
companies in Bay Area
Awarded Inc 500/5000
company of the year 2010
Part of 10 Most Promising
Testing Services Company
Gartner Top 20 QA Service Provider
Exceptional QA Services and
Solutions
GESIA Rising Star Award 2010 – Best
Software Product
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah
©2004-2011 InfoStretch Corporation. All rights reserved.
QA MetricsWe best Manage, What we can Measure
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Metrics – The Map
“Standards of measurement by which efficiency, performance, progress, or quality of a plan,
process, or product can be assessed."
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
If only Metrics ruled the World
Note: Cartoon by Andy Glover
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Quality Metrics – The Guide
“Measuring whether or not a given process will produce
products and services that meet quality standards.”
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Metrics – The Compass
“Measuring the outcome of business objectives with-in the defined constraints –
Predictability of process
Resource utilization
product/service quality
Timeliness
Market Impact
Customer Satisfaction
Financial Sanity
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Lost without a Guide or Quality Metrics
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
What is a successful outcome?
Measurement
Impact Criteria
Happy Customer &
Business Success
Outcome
Quality
Financial &BudgetTimeline
CustomerSatisfaction
Emp Happine
ss
Competition Impact
Sales
< Cust.Support
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Influence Metrics
Influence Metrics from Quantity to Quality
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Why Quality Metrics?Companies that consistently produce high quality products and services will have high customer loyalty and a strong corporate brand.
Is this a good business practice?
Is this guaranteed?
What about Cost, Timelines, Competitive Threat?
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2010 InfoStretch Corporation. All rights reserved.
Business MetricsLet us to do what is important to Business
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Types of Business Metrics – QA Centric
Defect Metrics
Resource Utilization
Metrics
Process Effectiveness
Metrics
Requirements Coverage Metrics
Test Coverage Metrics
Automation ROI Metrics
Quality Trend (Optimization)
Metrics
Compliance Metrics
Performance Metrics
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Quality MetricsDefect Reports
Lookup33%
Updates3%
Cache8%
Playlist30%
Lists3%
Core12%
Submit6%
Discover5%
Defect Density by Com-ponent
Rel1
Rel1.1
Rel2
Rel2.1
Rel3 Rel4 Rel5 Rel6
Rel6.1
Rel6.2
Rel6.3
Rel7
0
20
40
60
80
100
120
0
5
10
15
20
25
30
Defect trends
Total Defects
Total Fixed Defects
Total Open defects
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Quality MetricsDefect Reports
1 (Catastrophic)Severity
9%
2 (Critical)Severity
50%
3 (Moderate)Severity
31%
4 (Minor)Severity
10%
Defect Severity Report
<4 weeks 4-10 weeks >10 weeks
0
10
20
30
40
50
60
70
80
90
100
Open Defects (Aging Report)
Open Defects
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Quality MetricsDefect Reports
Rel1
Rel1.1
Rel2
Rel2.1
Rel3 Rel4 Rel5 Rel6
Rel6.1
Rel6.2
Rel6.3
Rel7
02468
101214161820
Defects By Release
Total Defects
Defects Fixed
Defects Deferred
Release
De
fec
ts
Rel1
Rel1.
1Rel
2
Rel2.
1Rel
3Rel
4Rel
5Rel
6
Rel6.
1
Rel6.
2
Rel6.
3Rel
7
4
8
4
1113
1
6
11
18
4
12
8
1 1 1 0 0 0 0 0 1 0 0 0
Pre and Post Product De-fects
Pre Production Defects Post Production Defects
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Quality MetricsDefect Reports
GUI2%
DataBase1%
Cosmetic12%
Business Logic46%
Error Handling2%
Server Response21%
Performance/ Optimization1%
Logical Errors6% Security Errors
8%GUIDataBaseCosmeticBusiness LogicError HandlingCompatibilityServer ResponsePerformance/ Op-timizationLogical ErrorsSecurity Errors
Rel1
Rel1.
1Rel
2
Rel2.
1Rel
3Rel
4Rel
5Rel
6
Rel6.
1
Rel6.
2
Rel6.
3Rel
70.00
20.00
40.00
60.00
80.00
100.00
120.00
Defect Removal Efficiency
Series1
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Examples & Importance
•These metrics give the status of quality of products based on defects and its impact on quality
Defect Based Metrics
• These metrics show the test as well as business requirement coverage as part of the testing effort
Coverage Based Metrics
•These metrics give the details of resource utilization and efficiency details which helps to optimize the QA/Testing operation. These metrics give the details surrounding effectiveness of testing process to meet quality objectives
Process Effectiveness/
Resource Efficiency / Optimization
Metrics
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Examples & Importance
• These metrics give the overall quality picture for product over the life span which enables to understand what is working and what is not working
Quality Trend Metrics
•These are special purpose metrics created specifically for business outcomes
Solution/Client Specific
Metrics
•It also includes the impact of Automation related ROI in testing organization
Automation ROI Metrics
• These metrics are usually produced as part of performance testing engagements.
Performance Metrics
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Quality MetricsEfficiency and Optimization
4.3.
84.
3.9
4.3.
104.
4.0
4.4.
24.
3.0
4.3.
24.
3.9
050
100150200250300350400
Testcase Maintenance For Release
Deleted TestcasesModified TestcasesNew Testcases
John Klien
Jeff Warner
Rakesh Kamani
Leslie Debra Nick Mencha
0
50
100
150
200
250
Resource Utilization and De-liverables
Testcases DevelopedDefects LoggedTestcases Automated
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Quality MetricsEfficiency and Optimization
4.3.
6.1
4.3.
6.2
4.3.
6.3
4.3.
7.1
4.3.
7.2
4.3.
7.3
4.3.
7.4
4.3.
7.5
4.3.
7.6
4.3.
8.1
4.3.
8.2
4.3.
8.3
4.3.
8.4
4.3.
8.5
4.3.
9.1
4.3.
9.2
4.3.
9.3
4.3.
9.4
4.3.
10.1
4.3.
10.3
4.3.
10.4
4.3.
11.1
4.3.
11.2
0
200
400
600
800
1000
1200
1400
0
20
40
60
80
100
120
140
160
180
Product Test Optimization per Release
Time for Testing Total Testcases
Tota
l Tes
tcas
es
Time For Testing
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Quality MetricsValue Driven - Automation and ROI
1 2 3 4 5 6 7 80
200
400
600
800
1000
1200
1400
Automation ROI
Manual Testcases Automated TestcasesManual Execution Time Automated Execution Time
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Quality MetricsEnd-Result Driven – User Satisfaction and Trend
Rel1 Rel2 Rel3 Rel402468
1012141618
Usability Measure
New Features Features Used in 1st monthDefects logged
Rel1 Rel2 Rel3 Rel40
200
400
600
800
1000
1200
1400
1600
Regression Efficiency
Regresion TestcasesRegression Time
Axis Title
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Business Driven Quality MetricsOverall Indicator
4.10
4.20
4.30
4.40
4.50
4.60
4.70
4.80
4.90
5.00
5.10
Stability IndexStability IndexStability Index
Stability IndexStability Index
Stability IndexStability IndexStability Index
Stability Index
Stability IndexStability Index
Stability IndexStability Index
Stability IndexStability Index
Stability IndexStability IndexStability Index
Stability Index
Stability Index
Stability Index
Stability Index
Stability Index
Product - Stability Index
Stability Index
Index Description
1Poor Quality2Becoming Stable3Needs Improvement4Desired Stability5Highly Stable
Quality indicators of the testing effort
using complex formula
reasonably indicates product quality
overall quality variation from
release to release
Stability Index
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2010 InfoStretch Corporation. All rights reserved.
Metrics Driven ActionMake the change and measure to analyze the impact
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Actions and Impacts
Metrics Description Frequency Actions
Effort & Schedule Variance
Effort spent versus Effort planned & Actual End Date Versus Planned End Date
Monthly • Adjust Resources and/or Project Timeline
• Incorporate learning for next phase planning
Time Variance
Number of releases that were successfully delivered to QA within the timelines
Monthly • Calculate the cost/quality impact due to delay
• Development schedule check
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Actions and Impacts
Metrics Description Frequency Actions
Test Containment
Efficiency
# Defects detected in Testing / (# Defects detected in Testing + Escaped defects detected by Client or subsequent testing)
Monthly • Modify testcases and coverage as necessary
• Retire the testcases that are not aiding value or repetative
Defect Sanity
Number of Valid Defects / Total Number of Invalid + Valid defects)
Monthly • Review testcases and refersh tester knowledge about the product
• Ensure completeness of defect description
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Actions and Impacts
Metric Description Frequency Actions
Test Case Efficiency
Number of Test Cases which are finding defects Vs Total Number
of Test Cases
End of testing round
• Review and Enhance/Retire testcases
• Link testcases to requirement, Usecases and/or defects
Cost of Testing
Ratio of Testing effort versus Total Project effort. Cost
effectiveness for Onsite vs offshore testing cost.
Monthly • Adjust tester focus based on the module/build/release importance
• Cross-train testers for different types of testing
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
QA Metrics – Actions and Impacts
Metric Description Frequency Actions
ROI on Automation
Calculate the Return on Investment on Automation (ROI
Calculator)
Quarterly • Measure reduction in regression time and compare with faster time-to-market
• Framework/Keyword driven automation for reduced maintenance
Usability and Performance
Measure the Usability of application with customer’s usage of features intuitively
(How many tech support calls?)
Release • Try new features with different UI
• Measure the usability and performance with customer data volume
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.
Best Theory
Mobile App DesignIt’s an art, not a science
Presenter: Morgan Russell, InfoStretch Corporation
All trademarks are the property of their respective owners.©2004-2011 InfoStretch Corporation. All rights reserved.
InfoStretch Next Webinar
Mark your CalendarApril 21, 2011
Mark your CalendarApril 21, 2011
Santa ClaraHeadquarters2880 Lakeside Drive, #200Santa Clara, CA 95054(408) 727-1100
Partners
New York17 Tanglewood Rd,Scarsdale, NY 10583(703) 395-9791
Ahmedabad5th Floor, Regency Plaza, Anand Nagar, Ahmedabad-15, Gujarat, India91-79-40266266
MumbaiVashi InfoTech Park, B103, Tower # 1, Vashi Rail Station, New Mumbai, Maharashtra, India91-22-6947900
Website: www.infostretch.com
phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]
©2004-2011 InfoStretch Corporation. All rights reserved.