structured exploratory testing – method in madness
DESCRIPTION
Structured Exploratory Testing – Method in Madness. Vivek Devarajan. Senior Consultant, Wipro Technologies. 1. Scripted versus Non-Scripted Testing. Some Exploratory Testing Ideas. Session Based Test Management. Benefits of Exploratory Testing. Case Studies. 2. 3. 4. 5. - PowerPoint PPT PresentationTRANSCRIPT
© 2012 WIPRO LTD | WWW.WIPRO.COM1
Structured Exploratory Testing – Method in Madness
Vivek DevarajanSenior Consultant, Wipro Technologies
© 2012 WIPRO LTD | WWW.WIPRO.COM2
Scripted versus Non-Scripted Testing
Some Exploratory Testing Ideas2
1
Session Based Test Management3
Benefits of Exploratory Testing4
Case Studies5
© 2012 WIPRO LTD | WWW.WIPRO.COM3
Scripted versus Non-Scripted Testing
© 2012 WIPRO LTD | WWW.WIPRO.COM4
Scripted versus Non-Scripted Testing
Freestyle
Testing
Scripted Testing
Structured Exploratory Testing / Session Based Test Management
•Elaborate Documentation•Static Test Suite•Higher Testing Costs•Limited Defect Detection
•Concise Documentation•Dynamic Test Optimization•Lower Testing Costs•Better Defect Detection
•No Accountability•No Predictability•High Possibility of Defect Slippage
Leveraging the benefits of both Testing StylesMethodical Madness
© 2012 WIPRO LTD | WWW.WIPRO.COM5
Some Exploratory Testing Ideas
© 2012 WIPRO LTD | WWW.WIPRO.COM6
Exploratory Testing using Question PatternsWhile testing a system, how often have you felt “I have tested something similar before?”
The Domain FactorOn the other hand, how often have you been bogged down by an application, just because it was from a different domain that you are not so familiar with?
The Reuse FactorIs there a way to develop testing ideas once and reuse them across domains / applications?
The Knowledge FactorIs there a way an experienced tester or domain expert can impart his/her experience to others?
The Exploratory Testing Context Is there a way to explore the application systematically and find bugs?
© 2012 WIPRO LTD | WWW.WIPRO.COM7
Question Patterns and Exploratory TestingQuestion Patterns: A Set of interrelated Questions grouped together, focused on an aspect of the system under test that•Enable someone to understand the application better•Can be used as a technique for requirements capture
Patterns help the tester explore the application systematically and find bugs
Example: Parent and Child Entity PatternScenario A).Consider an online shopping application where the user can add / modify / delete multiple items in his shopping cart order. In this case, the shopping cart is a business entity which has multiple child entities (items) within itself (a one-to-many relationship between the cart and the items) where the child entity (item) can be added/updated or deleted.Scenario B). Consider an application where the user can apply for an insurance policy online, to which he can add / modify / delete multiple beneficiaries. In this case, the policy is a business entity which has multiple child entities (beneficiaries) within itself (a one-to-many relationship between the policy and the beneficiaries) where the child entity (beneficiary) can be added/updated or deleted.
© 2012 WIPRO LTD | WWW.WIPRO.COM8
Question Patterns and Exploratory TestingQuestions that can be asked: Parent and Child Entity Pattern•What are the business entities involved?
•What is the relationship between them? (One to Many, One-to-One, etc)
•What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search)
•What are the field level and business validations for each entity?
Answers to such questions enable testers to find gaps in requirements (or even capture requirements).
Answers to such questions also serve as a guideline and a checklist during exploratory testing.
© 2012 WIPRO LTD | WWW.WIPRO.COM9
Question Patterns and Exploratory TestingQuestions that can be asked: Parent and Child Entity Pattern•What are the business entities involved?
•What is the relationship between them? (One to Many, One-to-One, etc)
•What are the operations that can be performed in either of the entities (Add/Modify/delete/view/search)
•What are the field level and business validations for each entity?
Answers to such questions enable testers to find gaps in requirements (or even capture requirements).
Answers to such questions also serve as a guideline and a checklist during exploratory testing.
© 2012 WIPRO LTD | WWW.WIPRO.COM10
Exploratory Testing by Looking for Specific Problems
•Purpose of Testing – Looking for Bugs/Problems that could impact the stakeholders
•Defect Seeking v/s Defect Finding
•Make a list of possible issues
•Design tests to find ways to stimulate the possible issues identified
© 2012 WIPRO LTD | WWW.WIPRO.COM11
Exploratory Testing by Analyzing Data Flow
•Most Applications process / manage data
•Identify the input and output data
•Verify the output data for various combinations of input data (including green path values, invalid values, boundary conditions, large values, etc)
•Try modifying the data at various stages in the flow and see the impact
•Identify the right combinations / representatives of data
© 2012 WIPRO LTD | WWW.WIPRO.COM12
Exploratory Testing by Analyzing Flows
•Identify end-to-end flows and branching
•Depict the branching graphically, if needed
•Try parallel flows
•Try varying the sequencing / timing of the flows
•Analyze the impact of input data values on the flows
•Check the main flows as well as alternate/exceptional flows
•Check for state transition flows
© 2012 WIPRO LTD | WWW.WIPRO.COM13
Session Based Test Management – Accountable and Manageable Exploratory
Testing
© 2012 WIPRO LTD | WWW.WIPRO.COM14
Session Based Test Management - Approach
Test Execution is divided into time-boxed Test Sessions, where the tester does (uninterrupted) exploratory testing with a specific charter
Tester takes brief notes about the testing done and the issues found
Tester uses a checklist of testing ideas as a guide for testing
Further tests are dynamically optimized depending on the results of the current test
The Test Lead conducts a ‘Debrief’ session with the testers on the test findings and plans further executions based on the findings
© 2012 WIPRO LTD | WWW.WIPRO.COM15
Session Based Test Management – Session Contents
The contents of a typical test session are as follows:
•Charter: Contains the mission of the given session. Typically it mentions the application areas to be covered / explored, at a high level.•Tester name, Session Time Start and End•Task Breakdown: Time taken for Test execution v/s Bug Reporting, Charter (Original Session Charter) v/s Opportunity (Diversions taken)•Test Notes: Notes of what the tester executed, in brief.•Bugs/Issues found
© 2012 WIPRO LTD | WWW.WIPRO.COM16
Session Based Test Management – Accountability
•Tester’s Activities: Tracked via Test Session reports
•Coverage and Testing Progress Tracking: • High Level Coverage: Number of functional areas and quality
criteria covered• Granular Coverage: Number of checklists/items/test ideas
covered for each functional area•Traceability: Each test idea and item in the checklist can be mapped to requirements
© 2012 WIPRO LTD | WWW.WIPRO.COM17
Benefits of Exploratory Testing
© 2012 WIPRO LTD | WWW.WIPRO.COM18
Exploratory Testing - Benefits
•Lesser Preparation and lesser effort• •Quicker Bug Detection
•Leverages on Tester’s Abilities, experience and creativity
•Works even if requirements are not fully frozen
© 2012 WIPRO LTD | WWW.WIPRO.COM19
Skilled Testing v/s Deskilling – The Classic Debate
•Deskilling via Detailed Test Cases leads to higher test design and maintenance costs and limited bug detection capabilities
•Testers with lesser experience can be trained in skilled exploratory testing
•Skilled Testing is more cost effective and results in better product quality
© 2012 WIPRO LTD | WWW.WIPRO.COM20
Case Studies
© 2012 WIPRO LTD | WWW.WIPRO.COM21
Case Study 1 – The Problem
The Context:•Previous customer concerns over quality in beta releases
•Large volume of functional tests to be executed in a short time span
- Conventional Testing Estimations: 3000 test cases to be designed and executed in 2 months (including KAP phase), requiring 14 testers
•Inexperienced Testers with no domain knowledge
•Customer more focused on bugs/issues than in testing processes
A Police IT Application for a Government Customer
© 2012 WIPRO LTD | WWW.WIPRO.COM22
Case Study 1 – Testing Approach and Results
•Question Patterns used to arrive at checklist of testing ideas- Entity Relationship pattern- State Transition Pattern- Master Data – Transaction Data Pattern
•Checklists consisting of Business Conditions and Testing Ideas
•Initial Test Sessions focused on exploration of the business operations of entities
•Deep Dive Test Sessions focused on finding specific problems
•Test Sessions for End to End Testing focused on Application Flow and Data flow
Results: 65% reduction in testing efforts and 60% better defect detection
A Police IT Application for a Government Customer
© 2012 WIPRO LTD | WWW.WIPRO.COM23
Case Study 2 – The Problem
The Context:
•The application is intended to enable healthcare providers and pharmacies to place orders quickly
•Bugs not being found early enough
•High cost of testing
•Frequent patch releases with incomplete functionality
•Mix of experienced and inexperienced testers, with fair domain knowledge
•Parallel run of conventional testing and Session Based Test Management
An Order Management Application for a Healthcare customer
© 2012 WIPRO LTD | WWW.WIPRO.COM24
Case Study 2 – Testing Approach and Results
•Question Patterns used to arrive at checklist of testing ideas for regression testing
•Checklists consisting of Business Conditions and functional changes to be checked for
•Initial Test Sessions focused on the exploration of new functionalities
•Subsequent Test Sessions focused on user experience
Results: 50% reduction in test design efforts and quicker defect detection
An Order Management Application for a Healthcare customer
© 2012 WIPRO LTD | WWW.WIPRO.COM25
Thank youVivek DevarajanSenior Consultant, Wipro Technologies
© 2012 WIPRO LTD | WWW.WIPRO.COM26
Appendix
© 2012 WIPRO LTD | WWW.WIPRO.COM27
Artifacts Created
Checklists Created for Session Based Test Management:
Sample Test Session:
Microsoft Office Excel Worksheet
Microsoft Office Excel 97-2003 Worksheet
Microsoft Office Excel 97-2003 Worksheet
Microsoft Office Excel 97-2003 Worksheet
Microsoft Office Excel Worksheet
© 2012 WIPRO LTD | WWW.WIPRO.COM28
Session Based Test Management
Freestyle
Testing
Scripted Testing
Session Based Test Management
Exploratory Testing
SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles
S. No.
Factor Scripted Testing Session Based Test Management
Freestyle Testing
1 Cost Very High Medium to Low Very Low2 Defect
Detection• Limited Effectiveness• Sometimes late in the cycle
•Highly Effective•Early in the cycle
•Highly Effective•Very unpredictable
3 Accountability Very High Very High Very Low
© 2012 WIPRO LTD | WWW.WIPRO.COM29
SBTM - Leveraging the benefits of Scripted and Freestyle Testing Styles
S. No.
Factor Scripted Testing Session Based Test Management
Freestyle Testing
4 Predictability Very High Medium Very Low5 Possibility of
Defect Slippage
Medium Low High
6 Documentation Level
Very High Medium Very Low
7 Requirements Coverage
Very High Very High Very Low
8 Requirements Traceability
Very High High Very Low
Session Based Test Management
© 2012 WIPRO LTD | WWW.WIPRO.COM30
Comparison of Structured v/s Freestyle Testing
Freestyle Testing
Scripted Testing
Benefits: •High Predictability•Deskilling (Test Execution)•High AccountabilityDrawbacks:•Very High Costs (Resources + Tools) for Test Creation as well as Maintenance•Limited Defect Detection Capabilities (Lack of Proactive/Creative Approaches)•Needs well documented requirements
Benefits: •Very Low Costs for Test Creation/Maintenance•Highly effective in quick Defect Seeking, especially effective for tricky bugs•Does not need well documented requirementsDrawbacks:•Low Accountability•Relies on the creativity of the Tester
© 2012 WIPRO LTD | WWW.WIPRO.COM31
Session Based Test Management
Freestyle
Testing
Scripted Testing
Automated Testing
Session Based Test Management Exploratory
Testing
•Elaborate Documentation•Static Test Suite•Higher Testing Costs•Limited Defect Detection
•Concise Documentation•Dynamic Test Optimization•Lower Testing Costs•Better Defect Detection
•No Accountability•No Predictability•High Possibility of Defect Slippage
SBTM - Leveraging the benefits of both Testing Styles
© 2012 WIPRO LTD | WWW.WIPRO.COM32
Evolution of Session Based Test Management
Adhoc / Freestyle Testing
Exploratory Testing
Session Based Test
Management
Evol
utio
n
© 2012 WIPRO LTD | WWW.WIPRO.COM33
Salient Features of Session Based Test Management• Application Learning, Test Design and Test Execution happen in
Parallel
• A Structured, measurable and accountable method of Exploratory Testing
• Crisp Checklists to guide the tester on what needs to be tested
• Execution happens in Test Sessions (Uninterrupted Test Execution Time) with specific charters, after which tester takes notes and makes a report
• Tester takes a different execution route, if required, depending on issues found in a Test Session
• Future test sessions are planned based on results from current session
© 2012 WIPRO LTD | WWW.WIPRO.COM34
Metrics for Session Based Test Management
Microsoft Office Excel Worksheet
Test Session Metrics
Cumulative Defect Trend
Defect TrendExecution Metrics
E:\Projects\Cardinal\Test Execution Prod E:\Projects\Cardinal\
Defect Trend.bmp
E:\Projects\Cardinal\Cumulative Defects