Use Test Trees to get an Overview of your Test
INTERNAL USE
Use Test Trees Test Coverage
Test Design and Implementation
Introduction to a Test Tree
Example
When are we done with the test
As much as we like testing, we cannot go on ‘for ever’ – this is why acceptance criteria are defined
When the acceptance criteria are reached we can stop the testing How much do we have to test with this acceptance criteria: No more than 10 unresolved defects
Classification Internal Use Author AMQH Approved by N/A Version 1.0 3
This is more like it
Acceptance criteria should always include a coverage measure Example: • a coverage of 100 % is achieved for the
requirements on which the test is based
• a detailed coverage of 60 % (high risk areas) or 40 % (low risk areas) is achieved
Classification Internal Use Author AMQH Approved by N/A Version 1.0 4
Test coverage definitions
ISTQB: the degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite.
ISO 29119: degree, expressed as a percentage, to which specified test coverage items have been exercised by a test case or test cases
Classification Internal Use Author AMQH Approved by N/A Version 1.0 5
Specified test coverage items?
So we have to find out what specified test coverage items are. ISO 29119: attribute or combination of attributes that is derived from one or more test conditions by using a test design technique that enables the measurement of the thoroughness of the test execution
Classification Internal Use Author AMQH Approved by N/A Version 1.0 6
Some test case design techniques
Classification Internal Use Author AMQH Approved by N/A Version 1.0 7
Boris Beizer • Control-flow testing • Data-flow testing • Domain testing • Finite-state testing • Loop testing • Syntax testing • Transaction flow t.
ISTQB • Boundary value analysis • Equivalence partitioning • Cause Effect graphing • Checklist • Classification tree • Decision table testing • Orthogonal arrays • Pairwise testing • State Transition testing • Structure based • Use case testing • User story testing • and more..
ISO 29119 • Boundary value analysis • Equivalence partitioning • Cause Effect graphing • Classification tree • Combinatorial e.g.
• pairwise • orthogonal arrays
• Decision table testing • Random testing • State Transition testing • Scenario testing • Structure based • Syntax testing • and more…
Some coverage items
• Boundary value analysis Boundary + 1 or 2 nearest values • Equivalence partitioning Equivalence partition • Cause Effect graphing Decision rule • Checklist Entry • Classification tree Leaf • Decision table testing Decision rule • Orthogonal arrays Pair • Pairwise testing Unique parameter/value set • State Transition testing States and/or Chow’s n-switch • Structure based Depends on which one • Use case testing Unique scenario • User story testing Acceptance criterium • and more..
Classification Internal Use Author AMQH Approved by N/A Version 1.0 8
Calculating detailed coverage
For a complex system where a number of test case design techniques are used, all identified and tested coverage items must be included in the calculation of the detailed coverage:
(∑𝑒=1↑𝑛▒ 𝐶𝑇(𝑒) ) (∑𝑒=1↑𝑛▒ 𝐶𝐼(𝑒) ) where e = number of test case design techniques used CT = number of tested coverage items per technique CC = number of identified coverage items per technq.
Classification Internal Use Author AMQH Approved by N/A Version 1.0 9
Detailed coverage % = x 100
Small example
Equivalence partitioning were used on 2 requirements, where 5 partitions were identified for one requirement and 3 for the other 2 partitions where tested for one requirement and all 3 were tested for the other
State transition were used on 1 requirement and 4 transitions where identified of which all were tested one at the time 2+3+4 5+3+4
Classification Internal Use Author AMQH Approved by N/A Version 1.0 10
Detailed coverage = x100 = (9/12)x100 = 75%
Use Test Trees Test Coverage
Test Design and Implementation Introduction to a Test Tree
Example
Test design and implementation
The test design and
implementation process
is a sub-process to the
entire test process
described by ISTQB and
in ISO 29119
Test design and implementation
items
Definitions
A feature set is a logical part of the system, which you can test in isolation, e.g.: • access to the system, including access right • a user interface, e.g. a form or a pane • a set of rules implemented in a component • general error handling
A test condition is something within a feature set, which you can test in isolation: e.g. • a requirement • a user story • a rule
Classification Internal Use Author AMQH Approved by N/A Version 1.0 14
Use Test Trees Test Coverage
Test Design and Implementation
Introduction to a Test Tree Example
The test tree principle
A test tree is • a presentation of the test item • a presentation of the test basis • an overview of all coverage items for the test item • akin to classification trees – with coverage items
described under independent classes • a way of documenting a selection of coverage
items to be tested in a test case • a mean to calculate detailed coverage • documented in Excel - the most powerful tool :-)
Classification Internal Use Author AMQH Approved by N/A Version 1.0 16
The contents of the tree
Classification Internal Use Author AMQH Approved by N/A Version 1.0 17
FS-3.PersonandRecidenceInformationpaneRequirements:FR_182,FR_191UISpec:Section5.4UserStory:none
TestcasesCiNo.
Classfilter& Ci
filter& Ci
filter& Ci
filter& Ci 1 2
PersonandRecidenceInformation,expandedcontents
1 topline:[-]PersonandRecidenceInformation xMandatorylabels
whichlabel2 Name: x
Gender:which
3 female x4 male
Test tree can be very ‘tall’
Classification Internal Use Author AMQH Approved by N/A Version 1.0 18
FS-3.PersonandRecidenceInformationpaneRequirements:FR_182,FR_191UISpec:Section5.4UserStory:none
TestcasesCiNo.
Classfilter& Ci
filter& Ci
filter& Ci
filter& Ci 1 2
PersonandRecidenceInformation,expandedcontents
1 topline:[-]PersonandRecidenceInformation xMandatorylabels
whichlabel2 Name: x
Gender:which
3 female x4 male5 ID: x6 Addresse: x7 Telephone: x
Citizenship:which
8 UK x9 notUK
DependablecontentswhichlabelortextC/O:
relevant10 yes x11 no
Custodian:infoavailable
12 yes x13 no
[!]CovertNameandAddresspersonsubjecttothis
14 yes15 no x16 buttomline:[+]OtherPeopleatthesameAddress x
actions17 presstopline x18 pressbuttomline
PersonandRecidenceInformation,collapsedcontents
19 topline:[+]PersonandRecidenceInformationactions
20 presstoplineOtherPeopleatthesameAddress,expanded
contents21 Mandatorytext:OtherpeopleattheAdress:<no.>
howmany22 0
morethan0anysubjecttonavne-ogadressebeskyttelse
23 noyesall
24 yes25 no26 Sub-panebuttom:[-]Otherpeopleatthesameaddress
actions27 presstopline28 pressSub-paneltopline
* <no.>lineswith"-"<name><id>ofotherpeople,sorteddescendingbyID** <no.>lineswith"-"<name><id>ofotherpeople,sorteddescendingbyID*** <no.>lineswith"-"<name><id>ofotherpeople,sorteddescendingbyID
This is a relatively small tree with only 3 classes and 28 coverage items Sometimes a test tree can have 50 + coverage items Use ‘divide and conquer’ where possible
Use Test Trees Test Coverage
Test Design and Implementation
Introduction to a Test Tree
Example
Example presentation
The example system is a case management system for case workers, who are responsible for the administration of payments to clients.
Classification Internal Use Author AMQH Approved by N/A Version 1.0 20
Result
Feature sets
The system is broken into the following feature sets (and many more):
• the framework for the overview of case information for the manual handling
• a pane within the framework of the overview
• rules for automatic handling
•
•
Classification Internal Use Author AMQH Approved by N/A Version 1.0 21
Framework for overview of case info.
Classification Internal Use Author AMQH Approved by N/A Version 1.0 22
Case Overview for Case: nn
Family Relations
Person & Residence Info
Result Details
Salary Info
Case History
Requirements for Person and Residence Info
Classification Internal Use Author AMQH Approved by N/A Version 1.0 23
Requirement FR_191: Show person information, family relations, history of the case (and more). When a case worker types a person ID, the system shall present the following information related to the citizen:
• Salary Information • Case History • Person and Residence Information
• (and more)
Requirement FR_182: Read and save person information. Upon reception of person information, the system shall be able to store it with a reference to the person’s ID.
Person and Residence Info
Design and notes for the pane with sub-pane
Classification Internal Use Author AMQH Approved by N/A Version 1.0 24
Note Description
1 This pane will be expanded as default. The following is shown for the chosen person: • Name • Gender • ID • if relevant: C/O • Postal address • Country • Telephone number • Citizenship – if not UK • if relevant: Name of custodian
2 If the persons’s name and address are covert, this is marked with a colored text above the name and address, which are shown. Text = “! Name and address are covert”
3 A click on ”[+] Other people at the same address” will show information about other people living at the same address in a sub-pane.
4 If one or more of the other person(s) at the address has covert name and address this will be shown as a list of people without, if any, sorted by ID, followed by a Text as above and a sorted list of the people with, if any.
5 A click on ”[-] Person and Residence Information” will cause the entire pane to collapse, so that only the top line is visible.
6 A click on "[+] Citizen and Residence Information" will cause the pane to expand, with the sub-pane collapsed.
Coverage items are identified
Test case design techniques are used to find coverage items depending on the nature of the test basis, for example:
Classification Internal Use Author AMQH Approved by N/A Version 1.0 25
Equivalence class partitioning
Classification tree
Pane navigation
The navigation in the pane and sub-pane can be treated like a state machine with 3 states and 5 transitions
Classification xxxx Author xxxx Approved by xxxx Version xx 26
pri = “Person and Residence Information” pane both = pri + “Other People at the same Address” sub-pane
Constructing the tree
Fill in the information about the test item and the test basis Identify the highest level of classes for the test item from the test basis For each class, identify the filters that will partition the class into further sub-classes For each sub-class, identify the coverage items using appropriate test case design techniques For each coverage item, describe expected result
Classification Internal Use Author AMQH Approved by N/A Version 1.0 27
Constructing the tree
Fill in the information about the test item and the test basis Identify the highest level of classes for the test item from the test basis For each class, identify the filters that will partition the class into further sub-classes For each sub-class, identify the coverage items using appropriate test case design techniques For each coverage item, describe expected result
Classification Internal Use Author AMQH Approved by N/A Version 1.0 28
FS-3.PersonandRecidenceInformationpane Coverageitems 28Requirements:FR_182,FR_191 40% 11UISpec:Section5.4UserStory:none
CiNo.
Classfilter& Ci
filter& Ci
filter& Ci
filter& Ci
Expectedresult
PersonandRecidenceInformation,expandedonlyP&RIcontents
1 topline:[-]PersonandRecidenceInformation presentMandatorylabels
whichlabel2 Name: shownwithcorrectname
Gender:which
3 female shownwithtext"Female"4 male shownwithtext"Male"5 ID: shownwithcorrectID6 Addresse: shownwithcorrectaddress7 Telephone: shownwithcorrectno.
Citizenship:which
8 UK labelshown,noinfo.9 notUK labelshownwithcorrectcitizenship
DependablecontentswhichlabelortextC/O:
relevant10 yes C/Oshownwithcorrectinfo.beforeAddress11 no nothingshownshown
Custodian:infoavailable
12 yes labelandinfoshownbelowCitizenship13 no nothingshownshown
[!]CovertNameandAddressClassification Internal Use Author AMQH Approved by N/A Version 1.0 29
[!]CovertNameandAddresspersonsubjecttothis
14 yes textshowninredundertopline15 no nowarningtextshown16 buttomline:[+]OtherPeopleatthesameAddress present
actions17 presstopline([-]) P&RIpanecollapsandtoplinewith[+]18 pressbuttomline([+]) bothpanesexpanded
P&RI(andsub-pane),collapsedcontents
19 topline:[+]PersonandRecidenceInformation presentactions
20 presstopline paneexpandsP&RIandsub-pane,expanded
Sub-panecontents21 Mandatorytext:OtherpeopleattheAdress:<no.> present
howmanywithcovertaddress22 0 textand<no.>=0shown
morethan0anysubjecttonavne-ogadressebeskyttelse
23 no *yesall
24 yes **25 no ***26 Sub-panebuttom:[-]Otherpeopleatthesameaddressshown
actions27 pressP&RItopline([-]) bothpanescollapse28 pressSub-paneltopline sub-panecollapses
* <no.>=actualnumber;lineswith"-"<name><id>ofotherpeople,sorteddescendingbyID** <no.>=actualnumber;warningtext;lineswith"-"<name><id>ofotherpeople,sorteddescendingbyID*** <no.>=actualtotalnumber;linesforwithout;warningtext;linesforwith
The rest of the tree
Classification Internal Use Author AMQH Approved by N/A Version 1.0 30
Select coverage items to test
Classification Internal Use Author AMQH Approved by N/A Version 1.0 31
FS-3.PersonandRecidenceInformationpaneRequirements:FR_182,FR_191UISpec:Section5.4UserStory:none
TestcasesNo. Class filter Ci filter Ci filter Ci 1
PersonandRecidenceInformation,expandedcontents
1 topline:[-]PersonandRecidenceInformation xMandatorylabels
whichlabel2 Name: x
Gender:which
3 female x4 male5 ID: x6 Addresse: x7 Telephone: x
Citizenship:which
8 UK x9 notUK
DependablecontentswhichlabelortextC/O:
relevant10 yes x11 no
Custodian:infoavailable
12 yes x13 no
[!]CovertNameandAddresspersonsubjecttothis
14 yes15 no x16 buttomline:[+]OtherPeopleatthesameAddress x
actions17 presstopline x18 pressbuttomline
FS-3.PersonandRecidenceInformationpane Coverageitems 28Requirements:FR_182,FR_191 40% 11UISpec:Section5.4UserStory:none
CiNo.
Classfilter& Ci
filter& Ci
filter& Ci
filter& Ci
Expectedresult
PersonandRecidenceInformation,expandedonlyP&RIcontents
1 topline:[-]PersonandRecidenceInformation presentMandatorylabels
whichlabel2 Name: shownwithcorrectname
Gender:which
3 female shownwithtext"Female"4 male shownwithtext"Male"5 ID: shownwithcorrectID6 Addresse: shownwithcorrectaddress7 Telephone: shownwithcorrectno.
Citizenship:which
8 UK labelshown,noinfo.9 notUK labelshownwithcorrectcitizenship
DependablecontentswhichlabelortextC/O:
relevant10 yes C/Oshownwithcorrectinfo.beforeAddress11 no nothingshownshown
Custodian:infoavailable
12 yes labelandinfoshownbelowCitizenship13 no nothingshownshown
[!]CovertNameandAddress
Please try this at home
Classification Internal Use Author AMQH Approved by N/A Version 1.0 32
Thank you – and remember
• Test is difficult
• Test requires overview
• Test requires creativity
• Test requires systematic work
• Test requires imagination
• Test requires courage
• Test is fun 06/06/16 33