![Page 1: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/1.jpg)
Presented by Stephen Allott
CaseMaker InternationalA Diaz & Hilterscheid group company
Effective Testing
Using industry standard techniques
SoftTest Ireland 16th February 2006
![Page 2: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/2.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 2
Agenda
• Why do we test software?
• How do we test effectively?
• Risk based testing• Black Box techniques• Pairwise testing
Photo: Courtesy of NASA
![Page 3: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/3.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 3
Why test software?
![Page 4: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/4.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 4
Read all about it!
US site – down for 3 days,
forced to return $30 fee to nearly
260,000 online “fantasy
baseball” players.
US site down for 2
days, $280,000 in
lost sales.
Shopping Site: 6-hour downtime:
$400,000 in direct costs plus additional
in discounts to inconvenienced
customers.
UK - Concerns over bugs
and delays in new air
traffic control software
UK – Payment clearing system failure at month end
delayed wages to many customers
UK - Televisions,
Cameras sold at
very low prices
![Page 5: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/5.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 5
Ineffective Testing
• Creates too many of the wrong tests• Duplicates effort unnecessarily• Costs too much and takes too long• Has an adverse impact on the business as a
whole
![Page 6: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/6.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 6
Effective Testing
• Is aligned with the business need• Uses a risk-based approach and formal
techniques to create fewer tests with increased coverage
• Facilitates communication with the business• Provides pre-production confidence• Saves time and money
![Page 7: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/7.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 7
Risk Based Testing
• Identify business risks• Assess the impact
– Severity, if it were to fail– Probability of failure
• Focus more testing on high risk areas
• Continually measure and monitor the risks
![Page 8: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/8.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 8
Test Case Design Techniques – 1
• Black box and white box techniques have been proven to be effective
• Techniques are described in BS7925• Analysis/design separated from execution• Improved and demonstrable test coverage
![Page 9: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/9.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 9
Test Case Design Techniques – 2
• Analysis of the test conditions• A formal design process gives you
– Objectivity– Repeatability– Consistency– Documentation– Auditability– Traceability– Confidence
![Page 10: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/10.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 10
Test Case Design Techniques – 3
• The design activity alone can find faults• Helps drive the need for good requirements• Use of formal techniques helps with estimates
of overall test effort required• Supports test automation
![Page 11: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/11.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 11
Black Box Techniques
• BS7925 techniques– Equivalence class
partitioning– Boundary value
analysis– Decision tables– State transition
testing– Syntax testing– Random testing
• Other techniques– Pairwise– Domain analysis– Use case testing
![Page 12: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/12.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 12
White Box Techniques
B
S
MCDC
BC
L
All Paths
DU
DPU
DCU
where: B - Branch coverage/Decision coverage BC - Branch Condition coverage BCC - Branch Condition Combination coverage DCU - All C-uses coverage DPU - All P-uses coverage DU - All du-paths coverage L - LCSAJ coverage MCDC - Modified Condition Decision coverage S - Statement coverage
BCC
![Page 13: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/13.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 13
Test Data
• Copy of live data• Realistic data volumes• Range of values and
inputs
![Page 14: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/14.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 14
Equivalence Class Partitioning
• Can you ever see a need to: – try the view from every seat of a cinema or theatre?– test drive every car in the showroom?– withdraw cash from all the ATMs in your country?
• If you answered “No, of course not!”– what would you do instead, and why?
![Page 15: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/15.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 15
Equivalence Class Partitioning
• Equivalence class or partition• Ranges of data• Based on an assumption about the software’s
behaviour• Reduces the number of test cases whilst
maintaining reasonable test coverage
![Page 16: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/16.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 16
Equivalence Class Partitioning
• An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price
Under 17 17 to 24 25 to 49 50 to 64 65 and over
£1,000 £700 £800
![Page 17: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/17.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 17
Boundary Value Analysis
• Builds on Equivalence Class technique• Assumes errors are made on the boundaries• Applicable to value and structure of data
– where there is a continuous range of values– length and character type
• Generates three test cases per boundary– one on the boundary– one either side of the boundary– the smallest distance away
![Page 18: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/18.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 18
Boundary Value Analysis – Example
• An online car insurance quotation system – Under 17 – unable to quote– 17 to 24 (premium price)– 25 to 49 (30% discount)– 50 to 64 (20% discount)– 65 and over (price on application)
Under 17 17 to 24 25 to 49 50 to 64 65 and over
16,17,18 49,50,5124,25,26 64,65,66
![Page 19: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/19.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 19
Boundary Value Analysis – Example
• Online Banking (Account balance)• Lower boundaries
– EUR 0.01 (DR)– EUR 0.00 (CR)– EUR 0.01 (CR)
• Upper boundaries– EUR 999,999.99 (CR)– EUR 1,000,000.00 (CR)– EUR 1,000,000.01 (exceeds max for this account
type)
![Page 20: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/20.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 20
Boundary Value Analysis – Example
• Online Banking (Set up standing order)• Boundaries for from date
– Yesterday (invalid)– Today (should be allowed)– Tomorrow (should be allowed)
• Boundaries for until date– 12 months from today (ok)– 12 months less 1 day (ok)– 12 months and one day from today (invalid)
![Page 21: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/21.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 21
Decision Tables
• Another intuitive black box technique• Express complex business rules in a table• Cause-Effect graphing helps with the analysis • Define the combination of conditions (causes)• Define the resulting actions (effects)• Easy to create test cases
![Page 22: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/22.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 22
Example – Train Tickets
Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Holds a railcard Y Y N N
Travels after 9 am
Y N Y N
ActionsPay £10
Pay £15
Pay £20
Pay £30
![Page 23: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/23.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 23
Example – Train Tickets Test Cases
Inputs
Test 1
Test 2
Test 3
Test 4
Holds a railcard Y Y N N
Travels after 9 am
Y N Y N
Expected results
Pay £10
Pay £15
Pay £20
Pay £30
![Page 24: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/24.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 24
Example – ATM
Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Enter valid card Y Y N N
Enter valid PIN Y N Y N
Actions Auth Reject Reject Reject
![Page 25: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/25.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 25
Example – ATM “don’t care”
Conditions
Rule 1
Rule 2
Rule 3
Enter valid card Y Y N
Enter valid PIN Y N ---
Actions Auth Reject Reject
![Page 26: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/26.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 26
Example – File Backup
ConditionsRule
1Rule
2Rule
3Rule
4
Stick inserted N Y Y Y
Stick formatted --- N Y Y
Sufficient space --- --- N Y
Actions Fails Fails Fails Works
![Page 27: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/27.jpg)
Pairwise Testing
Acknowledgement
This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test
design, published by Artech House, 2004, ISBN 1-58053-791-X
![Page 28: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/28.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 32
Pairwise Testing
• Pairwise is based on testing pairs of combinations of variables rather than all combinations
• It reduces the number of test cases• Many faults are either
– Single mode– Double mode
• There are two recognised methods:– Orthogonal arrays– Allpairs algorithm
![Page 29: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/29.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 33
Pairwise Example
• Consider a simple database enquiry screen:
– It has 4 fields
– Each field has three possible choices
– Giving 3 X 3 X 3 X 3 = 81 possible combinations to test
• With Pairwise Testing
– You can reduce this to 9 tests!
![Page 30: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/30.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 34
Pairwise Example
• Holiday destination– UK, USA, Europe
• Price range (for family of 5)– Budget: Under £2000– Medium: £2000 to £5000– Expensive: Over £5000
• Standard of hotel– Basic, 4 star, luxury
• Availability– Now: Within 7 days from today– Soon: After a week but up to one month from today– Later: Later than one month from today
![Page 31: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/31.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 35
Orthogonal Array L9 (34)
A B C D
Row 1 1 1 1 1
Row 2 1 2 2 2
Row 3 1 3 3 3
Row 4 2 1 2 3
Row 5 2 2 3 1
Row 6 2 3 1 2
Row 7 3 1 3 2
Row 8 3 2 1 3
Row 9 3 3 2 1
This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column
![Page 32: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/32.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 36
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 1 1 1 1
Row 2 1 2 2 2
Row 3 1 3 3 3
Row 4 2 1 2 3
Row 5 2 2 3 1
Row 6 2 3 1 2
Row 7 3 1 3 2
Row 8 3 2 1 3
Row 9 3 3 2 1
Each variable becomes a column in the orthogonal array
![Page 33: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/33.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 37
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 UK 1 1 1
Row 2 UK 2 2 2
Row 3 UK 3 3 3
Row 4 US 1 2 3
Row 5 US 2 3 1
Row 6 US 3 1 2
Row 7 Europe 1 3 2
Row 8 Europe 2 1 3
Row 9 Europe 3 2 1
For each column we fill out the possible choices
![Page 34: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/34.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 38
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 UK Budget Basic Now
Row 2 UK Medium 4 star Soon
Row 3 UK High Luxury Later
Row 4 US Budget 4 star Later
Row 5 US Medium Luxury Now
Row 6 US High Basic Soon
Row 7 Europe Budget Luxury Soon
Row 8 Europe Medium Basic Later
Row 9 Europe High 4 star Now
Now the array is complete
![Page 35: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/35.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 39
Orthogonal Array L9 (34)
Where Price Hotel When
Test 1 UK Budget Basic Now
Test 2 UK Medium 4 star Soon
Test 3 UK High Luxury Later
Test 4 US Budget 4 star Later
Test 5 US Medium Luxury Now
Test 6 US High Basic Soon
Test 7 Europe Budget Luxury Soon
Test 8 Europe Medium Basic Later
Test 9 Europe High 4 star Now
Each row becomes a test case – you can add more if you wish
![Page 36: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/36.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 40
How to use Orthogonal Arrays
• Identify the variables• Determine the number of choices for each• Locate an orthogonal array of the right size• Map the testing problem onto the array• Construct the test cases
![Page 37: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/37.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 41
What size array do we need?
Variable Choices
Browsers IE5, IE6, …. 8 in total
Plug-ins None, Real player, Media Player
Operating systemsW95, W98, W2000, NT, XP, XP-PRO
Server IIS, Apache, WebLogic
Server O/S NT, XP, Linux
How many possible combinations are there?
![Page 38: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/38.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 42
Using Orthogonal Arrays
• We require an array with 5 columns– Column 1 needs 8 levels (browsers)– Column 2 would need 3 levels (plug ins)– Column 3 would need 6 levels (O/S)– Columns 4 & 5 would need 3 levels (Server, Server O/S)
• Ideally we need 81 61 33 but it doesn’t exist!
• The next largest is L64 (82 43) – 2 columns of 1 – 8– 3 columns of 1 – 4
• So rather than 1,296 tests we need only 64 tests!• Remember - one test case per row
![Page 39: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/39.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 43
Allpairs Algorithm
• Devised by James Bach• Generates fewer tests• In our example, it generates just 48 tests
compared to 64 from using orthogonal arrays• Consider 100 variables, 2 choices
– Using orthogonal arrays you need 101 tests– Using all pairs you need only 10 tests
![Page 40: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/40.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 44
Pairwise Testing – Summary
• Test all pairs rather than all combinations– With 4 variables and 3 choices for each you would
require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests
– For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests
• Pairwise guides us to help find– Single mode defects– Double mode defects
• You should always add any known risky combinations
![Page 41: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/41.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 45
Effective Testing – Summary
• Effective Testing uses– A risk-based approach– Formal test case design techniques
• The benefits are– Reduced number of test cases– Repeatable well documented tests– Improved planning, estimation and communication– Facilitates earlier involvement with the business– Find faults earlier– Give you a consistent approach to testing– Save time and money– Enables test automation
![Page 42: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/42.jpg)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 46
After all, what is quality?
![Page 43: Steven K Allott - Effective Testing - SoftTest Ireland](https://reader035.vdocuments.us/reader035/viewer/2022081413/5481062ab4795994578b47d4/html5/thumbnails/43.jpg)
Presented by Stephen Allott
CaseMaker InternationalA Diaz & Hilterscheid group company
Effective Testing
Using industry standard techniques
SoftTest Ireland 16th February 2006