statistical 'encounter' with testing - stepinforum.org · organizations and 9 years of...
TRANSCRIPT
STeP-IN SUMMIT 2014
11th
International Conference on Software Testing
June 2014 at Bangalore, Hyderabad, Pune - INDIA
Statistical 'Encounter' with TESTING
by
Prasad Gundu, Senior Manager – Process
Cigniti Technologies
And
Siva Kumar Alahari, Manager – Process
Cigniti Technologies
Copyright: STeP-IN Forum
Published with permission for restricted use in ‘STeP-IN SUMMIT 2014’ in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this
publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without
the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during ‘STeP-IN SUMMIT 2014’ as per the applicable
conditions.
Practices Experience Knowledge Automation
Produced By Hosted By
www.stepinforum.org www.qsitglobal.com
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
1
Produced by Hosted by
SOFTWARE QUALITY | ASSURED
Statistical Encounter with Testing Siva Kumar Alahari, Manager – Process, Cigniti Technologies
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 2
Abstract & Speaker Profile
Statistics are a powerful tool to derive inferences on current performance, future predictions and making management decisions. Statistical techniques provide an understanding of business performance baselines, insights for process improvements, communication of value, results of processes, and visible involvement. SPC provides real time analysis to establish controllable process baselines; to learn, set, and dynamically improve process capabilities; and focus business on areas needing improvement. SPC moves away from opinion-based decision making.
Organizations have been using the 7 QC tools for various metric analysis. Control charts are used for all measurements and baseline metrics which are released periodically. These control charts help teams to derive process performance models, PPMs are used to predict or forecast the process failures well in advance to control them by using appropriate techniques.
This in-depth tutorial will focus on how to leverage statistical techniques for testing and predict vulnerabilities well in advance with usage of PPMs while educating participants on creating correlation regression equations.
Siva Kumar Alahari is a graduate in Mechanical Engineering from Nagarjuna University with over 19 years of career reflecting strong leadership experience in Quality and Process Excellence. 10 years in Software Products & Services organizations and 9 years of consulting & manufacturing. Hands on designing, developing and implementing Quality Strategy to ensure it meet business goals and customer requirements. Deft in handling QMS definition, training, process consulting, implementation, auditing and process improvements through ISO 9001, ISO 27001, CMMI and Six Sigma. Competent in establishing, maintaining organizations processes and people capabilities to meet and exceed customers’ satisfaction
Abstract
Speaker Profile
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
2
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 3
Agenda
» Latest Technology to treat Cancer
» Lets talk about process
» Statistical Process Control (SPC)
» Pilot can, why not we?
» SPC in Testing
» Control charts & Out of control condition
» Predicting the future
» Process Performance Model (PPM)
» Types of Modelling techniques
» PPM – Regression Analysis
» PPM Process
» A case study
» Q& A
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 4
Statistical Encounter with Testing
Testing
Measure
Analyze
Test Optimize
Repeat
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
3
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 5
Lets watch a clip first …
» Clip here
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 6
Let us talk about Process …
What is a Process ? What is Process Capability ? What is Process Performance ?
Cp Sigma Ppm
2 6 3.4
1.67 5 233
1.33 4 6210
1 3 65800
0.67 2 3,08,540
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
4
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 7
Statistical Process Control
» Statistics ?
» Statistical Control ?
» Statistical Process Control ( SPC ) ?
SPC
METHODS
TOOLS
DATA
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 8
Pilot can, why not we?
Pilot lands
Aircraft within
50% of
Runway area
LSL USL
50% of Tolerance Band
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
5
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 9
SPC in Testing
Xbar R/S
Production Defect
leakage %
Test Case Execution
Productivity
Test Case Effectiveness
Test Case Authoring
Productivity
Rejected Defect %
Test Case Rejection %
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 10
Control Charts
Types Of Data
Variable Data
Sub group size Is 1
I and MR Charts
Sub group size
2 to 8
X bar and R Charts
Sub group size more than 8
X bar and S Charts
Attribute Data
Defects
Sub group of same size
C-Charts
Sub group of different size
U-Charts
Defectives
Sub group of same size
np-Charts
Sub group of different size
p Charts
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
6
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 11
Control Chart out-of-control conditions
Zone A
Zone B
Zone C
Zone C
Zone B
Zone A
Rule 1 Point beyond control limit
Rule 2 2 out of 3 in Zone A or beyond
Rule 3 4 out of 5 Zone B or beyond
Rule 5 6 or more in a row increasing or decreasing
Rule 4 8 or more on one side of centerline without crossing
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 12
Predicting the future ..
Process is Managed
Process is Stable
Process is Capable
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
7
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 13
Process Performance Model (PPM)
What is a Process Performance Model ?
» Relationships between attributes of process and its work products
» Developed from historical process performance data
» Calibrated using collected process and product measures
» Used to predict results to be achieved by a process.
Why do we need it ?
» Estimating, analyzing, and predicting the process performance of their defined processes.
» Assessing (potential) ROI for process improvement activities.
» Work groups use them for selecting processes or sub-processes for use.
» Estimating progress toward achieving the quality and process performance objectives.
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 14
Process Performance Models in CMMI..
CMMI
Level 4 - Organizational
Process Performance
SG 1 : Establish Performance Baselines and
Models
SP 1.5 – Establish Process
Performance Models `
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
8
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 15
PPM in a nutshell
Prediction Model
Business Understanding
Data Understanding
Data Preparation
Modeling
Evaluation
Deployment
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 16
Types of Modeling techniques
Simple Techniques
Regression Equations
Monte Carlo
Simulation
Probabilistic Modelling
including BBN
Discrete Event Process
Simulation
There are other Advanced techniques also available
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
9
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 17
PPM – Regression Analysis
ANOVA and Dummy Variable Regression
Chi – Square, Logit and Logistic Regression
Correlation and Linear Regression
Logistic Regression
Regression Analysis
Continuous (Y) Discrete
Co
nti
nu
ou
s
(X
)
Dis
cret
e
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 18
Process Performance Models (PPM)
Stable Process Normal: Histogram is bell shaped
No Assignable/ special causes: Remove CC outliers
Capable Process
Cp minimum “1”
Regression
Linear “Y = F(Xi)”
Clients’ Delight
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
10
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 19
Process Performance Models
» PPM Case Study
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 20
Q & A
STeP-IN SUMMIT 2014 Pre-Conference Tutorials Statistical 'Encounter' with TESTING
11
Produced by Hosted by
UK | US | INDIA | CANADA | AUSTRALIA | www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2014, Cigniti Technologies 21
References
» http://www.tutorialspoint.com/cmmi/cmmi-glossary.htm
» CMMI Services 1.3
» http://paris.utdallas.edu/IEEE-RS-ATR/document/2007/2007-14.pdf