software testing chapter 12: subsystems nagy akos – nick baetens – kevin buyl – matthias de...
Post on 21-Dec-2015
221 Views
Preview:
TRANSCRIPT
Software TestingChapter 12: Subsystems
Nagy Akos – Nick Baetens – Kevin Buyl – Matthias De Cock - Dieter De Hen
2
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
3
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
4
Subsystems
• What are subsystems Any testable collection of classes, objects, components
and modules• Characteristics
Executable and testable as a whole• Whatever its size, it must be executable to be testable• Candidate for responsibility testing
Has parts that can be tested in isolation Implements a cohesive set of responsibilities Doesn’t provide all the functionality of the application
under test
5
Subsystems
• Types of subsystems Small clusters
• Several tightly coupled classes• Testing the classes in isolation sometimes impractical
(e.g. cyclic dependencies)
Large clusters• Group of related classes or clusters
Pattern participants• Implementation of design pattern tested as a subsystem (e.g.
Model-View-Controller implementation)
6
Subsystems
• Types of subsystems (continued) Build group
• Collection of software components processed by the linkage editor to generate an executable
Task group• Collection of build groups that is linked as an executable
process or that generate processes that do something interesting
Family of Ada 95 packages Developer subclasses
7
Subsystems
• Why test them It is a precondition for testing at system scope It is a practical necessity Support for internal release during incremental,
iterative development Use cases can model requirements for any subsystem
with public interface Some classes and other components have been
bundled for historical reasons and must be tested as a whole
8
Subsystems
• Why test them (continued) Some subsystems must work to receive additional
funding Testing budget or schedule is limited or reduced Dependability of some subsystems is higher than of
others The target environment is unique or different or must
be simulated apart from the rest of the application
9
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
10
Subsystem Test Design Pattern
• Subsystem test plan What features and capabilities should be tested? How should the test plan and test suites be organized?
• Structure and responsibilities determine testing approach
• Key problem: Developing a testable model of aggregate behavior
> Solution: test design patterns
11
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
12
Class Association Test
• Intent Design a test suite to verify the implementation of
required associations among classes• Context
Applicable when the SUT must correctly implement the relationships represented in a class diagram
Reveal bugs in implementation of associations
13
Class Association Test
• Fault Model Kinds of faults
• Incorrect multiplicity Legal combination rejected Illegal combination accepted
• Update anomaly The link that associates several instances of one class with a single instance of
another class necessarily carries some redundant information
• Delete anomaly What happens to owned objects if the owning object is deleted? (depends on
design/requirements)
• Missing link Not all links are created or some links become corrupted
• Wrong link All links created but some links are incorrect or inconsistent
14
Class Association Test
• Fault Model Faults can be reached by exercising the code that
implements the association constraint This code lies on a path that will be executed when
boundary values of the constraints are input (see Section 10.2.4, Domain Testing Model)
15
Class Association Test
• Strategy Test model
• Multiplicity test sets Association A:B with multiplicity (e.g. 0..1, 0..n, 1..1, 1..n, …) has
2 boundary values Binary association A:B and B:A has 4 boundary values Apply 1x1 test strategy (with on points, off points and in points)
and cast results in domain matrix (see Section 10.2.4, Domain Testing Models)
• Testing for update/delete bugs• Wrong and missing links
16
Class Association Test
• Automation Increase testing by interleaving anomaly tests with
max‐value multiplicity tests SUT can be exercised with an API driver interface GUI may provide sufficient control to exercise
associations DBMS: database loader utility to setup test cases
17
Class Association Test
• Entry criteria Each class in the scope of the model has passed an
alpha‐omega cycle Keyed collections are typically used to implement
associations. These classes should be individually tested with Quasi‐modal Class Test
• Exit criteria Each multiplicity on point test passes Each multiplicity off point test passes A deletion anomaly test passes An update anomaly test passes All tests for mutual exclusion pass
18
Class Association Test
• Consequences N associations at most 8n basic multiplicity test cases
(2 for each min and max boundary on A:B association and 2 for each min and max boundary on B:A association)
Typical system: several 100 associations several 1000 test cases
Class association test requires a model of the class associations in the system under test
Difficult for maximal value testing if number of database instances is large
19
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
20
Controlled Exception Test
• Intent Design a test suite that will exercise exception
handlers and provide a systematic way to generate all exceptions
• Fault model Safe exception handling = difficult! Several pitfalls
Improperly passed exceptions Exception is ignored and results in abrupt task end Ignored exceptions resulting in mysterious failures Looping exception structure (retry without limitation) Wrong exception is thrown/wrong catch block activated …
21
Controlled Exception Test
• Test Model To test properly exceptions must be triggered
Can be difficult!
22
Controlled Exception Test
• Test Model To test properly exceptions must be triggered
Can be difficult!
3 ways for exception testing• Exception activation: Trough input and state values
supplied by test cases• Exception inducement: By directly manipulating the
implementation under test• Exception simulation: By using controllable wrappers
for virtual machine services
23
Controlled Exception Test
• Automation Stubs to simulate application specific exceptions Command-line scripts to generate anomalous
conditions which (hopefully) trigger exceptions Wrapper classes
24
25
Controlled Exception Test
• Consequences Design of exception-handling test suite requires
technical understanding of which exceptions can be raised
• Target environment exceptions (e.g. Java Class Libraries’ exceptions)
Well documented• Applications-specific exceptions
Typically undocumented
Testing exceptions in an operational system may be difficult!
26
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
27
Round-trip Scenario Test
• Intent Develops path set with minimal branch & loop
coverage for UML Sequence Diagrams by extracting a control flow model
• Fault Model Lots of different round-trip scenarios
• Scenario: Path trough sequence diagram Lots of scenarios = lots of possible bugs
• Incorrect or missing output• Missing functions or features• Incorrect messages passed• Deadlocks• …
28
29
Round-trip Scenario Test
• Test Model Sequence diagrams are not ideal for testing!Transform sequence diagram to flow graph
• Flow graphs are highly testable• Node: Chunk of unconditional processing• Arrows: Show which node may follow another
30
Test Procedure – Step 1
1. Convert Sequence Diagram into Flow Graph
• Each consecutive group of messages becomes a segment (block)
• Each bracketed condition becomes a decision (hexagon)• Each bracketed loop condition also becomes a decision
(hexagon) – The position of loop predicates is important here (‘while’ & ’for’ loops vs. ‘until’ loops)
Often this conversion will already reveal some ambiguities and omissions!
31
32
Test Procedure – Step 2
2. Identify Paths to Test• No loops and branches?
» Any scenario will reach all segments• Loops or braches?
» Often astronomical number of possible paths
Running over all possible paths is in most cases infeasible!
33
Test Procedure – Step 2
Solution:• Take every branch at least once• Use every loop in at least 3 ways
• Bypass the loop (or do minimum # iterations)• Do 1 iteration• Do the maximum number of iterations
• Advanced tools are not necessary! Use colored markers to draw the paths
34
Test Procedure – Step 2
Procedure:1. Choose the longest or most complex entry-
exit path and color it2. Switch to another color and color another
entry-exit path3. Repeat step 2 until all decision branches are
colored4. Make a table with a row for each path listing
the conditions that must hold for this path
35
36
Test Procedure – Step 2
Procedure - Continued6. Pick any path that leads to the loop condition7. Copy the paths up to the loop entry in the
table and append the predicate conditions to allow one iteration, append the exit path
8. Pick another path to the loop entry and append the predicate conditions to allow the maximum number of iterations, append the exit path
37
38
Test Procedure – Step 2
Path Comment Path Conditions
1 Longest path, all false, Bypass loop !notATMCard, !isStolen, !accountClosed, validPIN, !declineFee
2 Branch 1 true, Bypass loop notATMCard
3 Branch 2 true, Bypass loop !notATMCard, isStolen
4 Branch 3 true, Bypass loop !notATMCard, !isStolen, accountClosed
5 Branch 4 true, Bypass loop !notATMCard, !isStolen, !accoundClosed, validPIN, declineFee
6 Same as path 1 up to the loop, Loop once !notATMCard, !isStolen, !accountClosed, !validPIN && n==1, validPIN && n==2, !declineFee
7 Same as path 1 up to the loop, Loop max times !notATMCard, !isStolen, !accountClosed, !validPIN && n==1, !validPIN && n==2, !validPIN && n==3, !validPIN
39
Test Procedure – Step 3
3. Identify Special CasesSequence diagrams do not include:• Exception handling paths
Covered by Controlled Exception Test
• Bindings to polymorphic servers• The number of paths grows exponentially when a
polymorphic server is several subclasses deep
40
Test Procedure – Step 3
• Reduce the risks for polymorphic servers!• Identify circumstances that must not occur and involve
polymorphic objects without direct interface• Identify constraints that must always be observed and
involve polymorphic objects without direct interface
• Develop test cases for the “must” and “must not” scenarios
41
Test Procedure – Step 4
4. Identify inputs and states• To identify test case values just look at the
path conditions for each path• For each path pick a set of input and state values that
satisfies the particular path conditions • In most cases there will exist many different sets of
values that will satisfy the path conditions!
• Finally determine the expected results for each test case
42
Round-trip Scenario Test
• Consequences Accurate sequence diagram required! Not everything is modeled
Additional tests required Round-trip path heuristic far more effective than
poking around Expansion of paths resulting from polymorphic
bindings not addressed.
43
Overview
• Subsystems What are subsystems Types of subsystems Why test them
• Subsystem Test Design Patterns Class Association Test Controlled Exception Test Round-trip Scenario Test Mode Machine Test
44
Mode Machine Test
• Intent Develop a test suite for subsystem scope behavior
by analyzing component behavior• Modal subsystem
Dominant control object Implements a state pattern Behavior is sequentially constrained
• The same sequence of external events does not always produce the same response
> Example: vending machines
45
Mode Machine Test
• A correctly functioning subsystem must Accept legal events Reject events that are illegal Produce a correct resultant state Produce a correct action for each test event
• Five kinds of control faults Missing transition Incorrect transition Invalid resultant state Production of a corrupt state Sneak Path
46
Mode Machine Test
• Creating a Test Model Identify the dominant control object (DCO) and
develop its state model Identify the external events and actions Map the external events onto the DCO Develop a modal test suite for this state model
47
Mode Machine Test
• N+ Test Suite Conformance tests
• Regular tests
Guard expansion• Boolean outcomes
Iteration expansion• Each of the items gets selected once
Implicit transition expansion• Events accepted in every state
Sneak path analysis• Illegal sequences
48
Mode Machine Test
• Entry Criteria Each component has met the exit criteria for the
applicable test pattern Each cluster has minimal operability requirements
• Exit Criteria There is a test for every root-to-leaf path in the
expanded transition tree and each sneak path pair
top related