chaining test cases for reactive system testing test cases for reactive system testing peter...
TRANSCRIPT
Chaining Test Cases for Reactive System Testing
Peter Schrammel, Tom Melham and Daniel [email protected]
The 25th IFIP International Conference onTesting Software and Systems (ICTSS’13)
Nov 13-15, 2013, Istanbul, Turkey
Test Chains
Context:
Safety critical embedded software
Often modelled as synchronous reactive system
Safety standards: tool support for systematic testing desirable
Problem:
Often lengthy input sequences required to drive the system toa test goal
Reset after each test case: serious problem in on-target testing
Goal:
Find a test case chain: a single test case that covers a set oftest goals and minimises overall test execution time
Model-Based Testing
requirements design model implementation
test suitegenerator
test suite
check check
check
Model-Based Testing
requirements design model implementation
test suitegenerator
test suite
check check
check
Model-Based Testing
textualrequirements
Simulink/Stateflow
formalisedproperties
generatedC code
implementationcheck
check
Model-Based Testing
textualrequirements
Simulink/Stateflow
formalisedproperties
generatedC code
implementation
test suitegenerator
test suite
check
check
Model-Based Testing
textualrequirements
Simulink/Stateflow
formalisedproperties
generatedC code
implementation
test suitegenerator
test suite
check
check
Example: Cruise Control
OFF,0,FALSE
OFF,1,FALSE OFF,0,TRUE
ON,1,TRUE
DIS,2,TRUE DIS,0,TRUE
OFF,2,FALSE OFF,2,TRUE
brake ∨ dec
brake ∨ dec
brake∨ dec
gas ∨ acc
gas ∨ accgas∨ acc
acc ∨ dec
gas ∨ acc
brake ∨ dec button button
button
button
buttonbutton
button
gas ∨ acc
gas ∨ acc
brakegas
∨ accbrake∨ dec
brake∨ dec
brake ∨ dec
Example: Generated C Code from Simulink
void i n i t ( s t a t e t ∗s ) {s−>mode = OFF ;s−>speed = 0 ;s−>enable = FALSE ;
}void compute ( i o t ∗ i , s t a t e t ∗s ) {
mode = s−>mode ;switch (mode) {
case ON: i f ( i−>gas | | i−>brake ) s−>mode=DIS ; break ;case DIS :
i f ( ( s−>speed==2 && ( i−>dec | | i−>brake ) ) | |( s−>speed==0 && ( i−>acc | | i−>gas ) ) )
s−>mode=ON;break ;
case OFF :i f ( s−>speed==0 && s−>enable && ( i−>gas | | i−>acc ) | |
s−>speed==1 && i−>button | |s−>speed==2 && s−>enable && ( i−>brake | | i−>dec ) )
s−>mode=ON;break ;
}i f ( i−>button ) s−>enable = ! s−>enable ;i f ( ( i−>gas | | mode!=ON && i−>acc ) && s−>speed<2) s−>speed ++;i f ( ( i−>brake | | mode!=ON && i−>dec ) && s−>speed>0) s−>speed−−;
}
Example: Generated C Code from Simulink
void i n i t ( s t a t e t ∗s ) {s−>mode = OFF ;s−>speed = 0 ;s−>enable = FALSE ;
}void compute ( i o t ∗ i , s t a t e t ∗s ) {
mode = s−>mode ;switch (mode) {
case ON: i f ( i−>gas | | i−>brake ) s−>mode=DIS ; break ;case DIS :
i f ( ( s−>speed==2 && ( i−>dec | | i−>brake ) ) | |( s−>speed==0 && ( i−>acc | | i−>gas ) ) )
s−>mode=ON;break ;
case OFF :i f ( s−>speed==0 && s−>enable && ( i−>gas | | i−>acc ) | |
s−>speed==1 && i−>button | |s−>speed==2 && s−>enable && ( i−>brake | | i−>dec ) )
s−>mode=ON;break ;
}i f ( i−>button ) s−>enable = ! s−>enable ;i f ( ( i−>gas | | mode!=ON && i−>acc ) && s−>speed<2) s−>speed ++;i f ( ( i−>brake | | mode!=ON && i−>dec ) && s−>speed>0) s−>speed−−;
}
Formalised properties:
p1: G(
mode = ON ∧ speed = 1 ∧ dec ⇒ X(speed = 1))
p2: G(
mode = DIS ∧ speed = 2 ∧ dec ⇒ X(mode = ON))
p3: G(
mode = ON ∧ brake ⇒ X(mode = DIS))
p4: G(
mode = OFF ∧ speed = 2 ∧ ¬enable ∧ button ⇒ X enable)
Example
I = F
OFF,0,FALSE
OFF,1,FALSE OFF,0,TRUE
ON,1,TRUE
DIS,2,TRUE DIS,0,TRUE
OFF,2,FALSE OFF,2,TRUE
brake ∨ dec
brake ∨ dec
brake∨ dec
gas ∨ acc
gas ∨ accgas∨ acc
acc ∨ dec (p1)
gas ∨ acc
brake ∨ dec button button
button
button
button (p4)button
button
button
gas ∨ acc
gas ∨ acc
brake (p3)gas
∨ accgasbrake
∨ decbrake∨ dec(p2)
brake ∨ dec
Example
I = F
OFF,0,FALSE
OFF,1,FALSE OFF,0,TRUE
ON,1,TRUE
DIS,2,TRUE DIS,0,TRUE
OFF,2,FALSE OFF,2,TRUE
brake ∨ dec
brake ∨ dec
brake∨ dec
gas ∨ acc
gas ∨ accgas∨ acc
acc ∨ dec (p1)
gas ∨ acc
brake ∨ dec button button
button
button
button (p4)button
button
button
gas ∨ acc
gas ∨ acc
brake (p3)gas
∨ accgasbrake
∨ decbrake∨ dec(p2)
brake ∨ dec
Preliminaries
Program:
State space Σ, input space Υ
Initial states I ⊆ Σ
Transition relation T ⊆ Σ×Υ× Σ
Preliminaries
Program:
State space Σ, input space Υ
Initial states I ⊆ Σ
Transition relation T ⊆ Σ×Υ× Σ
Bounded Model Checking:
Check the existence of a path 〈s0, s1, . . . , sK 〉 of increasing lengthK from φ to φ′
φ(s0) ∧∧
1≤k≤K
T (sk−1, ik−1, sk) ∧ φ′(sK )
If SAT: satisfying assignment aka counterexample(s0, i0, s1, i1, . . . , sK−1, iK−1, sK )
Preliminaries
Program:
State space Σ, input space Υ
Initial states I ⊆ Σ
Transition relation T ⊆ Σ×Υ× Σ
Bounded Model Checking:
Check the existence of a path 〈s0, s1, . . . , sK 〉 of increasing lengthK from φ to φ′
φ(s0) ∧∧
1≤k≤K
T (sk−1, ik−1, sk) ∧ φ′(sK )
If SAT: satisfying assignment aka counterexample(s0, i0, s1, i1, . . . , sK−1, iK−1, sK )
Test case generation:
φ = I and test goal φ′
Test case: input sequence 〈i0, . . . , iK−1〉, expected outcome
Chaining Test Cases
Temporal logic safety specification:
Set of properties, e.g., of type
G(mode = ON ∧ speed = 1 ∧ dec︸ ︷︷ ︸
assumption ϕ
⇒ X(speed = 1))
Chaining Test Cases
Temporal logic safety specification:
Set of properties, e.g., of type
G(mode = ON ∧ speed = 1 ∧ dec︸ ︷︷ ︸
assumption ϕ
⇒ X(speed = 1))
Test goals: set of assumptions ϕ (finite paths)
Chaining Test Cases
Temporal logic safety specification:
Set of properties, e.g., of type
G(mode = ON ∧ speed = 1 ∧ dec︸ ︷︷ ︸
assumption ϕ
⇒ X(speed = 1))
Test goals: set of assumptions ϕ (finite paths)
Test chain: from initial states I via all ϕs to final states F
Chaining Test Cases
Temporal logic safety specification:
Set of properties, e.g., of type
G(mode = ON ∧ speed = 1 ∧ dec︸ ︷︷ ︸
assumption ϕ
⇒ X(speed = 1))
Test goals: set of assumptions ϕ (finite paths)
Test chain: from initial states I via all ϕs to final states F
Approach
1 Abstraction: property reachability graph
2 Optimisation: shortest path
3 Concretisation: compute concrete test case
Abstraction: Property Reachability Graph
Weighted, directed graph:
Nodes: test goals ϕ
Edges:
from I to all ϕsfrom all ϕs to F
pairwise links between ϕs
Edge weights: number of execution steps
Incrementally build graph by reachability queries:
ϕ1 ϕ3
I F
ϕ2 ϕ4
Abstraction: Property Reachability Graph
Weighted, directed graph:
Nodes: test goals ϕ
Edges:
from I to all ϕsfrom all ϕs to F
pairwise links between ϕs
Edge weights: number of execution steps
Incrementally build graph by reachability queries: K = 1
ϕ1 ϕ3
I F
ϕ2 ϕ4
1
1
1
Abstraction: Property Reachability Graph
Weighted, directed graph:
Nodes: test goals ϕ
Edges:
from I to all ϕsfrom all ϕs to F
pairwise links between ϕs
Edge weights: number of execution steps
Incrementally build graph by reachability queries: K = 2
ϕ1 ϕ3
I F
ϕ2 ϕ4
221
1
2
2
2
2 2
2
1
Existence of a Covering Path
Covering path: path that visits all nodes at least once.
There is a covering path from I to F
iff
(1) all nodes are reachable from I ,
(2) F is reachable from all nodes,and
(3) for all pairs of nodes (v1, v2),
(a) v2 is reachable from v1 or(b) v1 is reachable from v2.
ϕ1
I F
ϕ2
Reachability can be decided in constant time on the transitiveclosure of the graph.
Existence of a Covering Path
Covering path: path that visits all nodes at least once.
There is a covering path from I to F
iff
(1) all nodes are reachable from I ,
(2) F is reachable from all nodes,and
(3) for all pairs of nodes (v1, v2),
(a) v2 is reachable from v1 or(b) v1 is reachable from v2.
ϕ1
I F
ϕ2
Reachability can be decided in constant time on the transitiveclosure of the graph.
Existence of a Covering Path
Covering path: path that visits all nodes at least once.
There is a covering path from I to F
iff
(1) all nodes are reachable from I ,
(2) F is reachable from all nodes,and
(3) for all pairs of nodes (v1, v2),
(a) v2 is reachable from v1 or(b) v1 is reachable from v2.
ϕ1
I F
ϕ2
Reachability can be decided in constant time on the transitiveclosure of the graph.
Optimisation: Shortest Path Computation
Find a covering path from I to F :
Reduce to asymmetric travelling salesman problem (ATSP):
Tour that visits all nodes of a weighted directed graph exactlyonce
Transitive closure
ϕ1 ϕ3
I F
ϕ2 ϕ4
221
1
2
2
2
2 2
21
Optimisation: Shortest Path Computation
Find a covering path from I to F :
Reduce to asymmetric travelling salesman problem (ATSP):
Tour that visits all nodes of a weighted directed graph exactlyonce
Transitive closure
ϕ1 ϕ3
I F
ϕ2 ϕ4
221
1
2
2
2
2 2
21
4 4
3
4
6
4 4
Optimisation: Shortest Path Computation
Find a covering path from I to F :
Reduce to asymmetric travelling salesman problem (ATSP):
Tour that visits all nodes of a weighted directed graph exactlyonce
Transitive closure
ϕ1 ϕ3
I F
ϕ2 ϕ4
221
1
2
2
2
2 2
21
4 4
3
4
6
4 4
0
Optimisation: Shortest Path Computation
Find a covering path from I to F :
Reduce to asymmetric travelling salesman problem (ATSP):
Tour that visits all nodes of a weighted directed graph exactlyonce
Transitive closure
ϕ1 ϕ3
I F
ϕ2 ϕ4
221
1
2
2
2
2 2
21
4 4
3
4
6
4 4
0
ATSP result: 〈ϕ2, ϕ3,F , I , ϕ4, ϕ1〉Shortest path: 〈I , ϕ4, ϕ1, ϕ2, ϕ3,F 〉
Concretisation: Computing the Test Chain
I2
−→ ϕ42
−→ ϕ12
−→ ϕ21
−→ ϕ32
−→ F
I (s0)∧T (s0, i0, s1) ∧ T (s1, i1, s2)∧ϕ4(s2, i2)∧T (s2, i2, s3) ∧ T (s3, i3, s4)∧ϕ1(s4, i4)∧T (s4, i4, s5) ∧ T (s5, i5, s6)∧ϕ2(s6, i6)
∧T (s6, i6, s7)∧ϕ3(s7, i7)∧T (s7, i7, s8) ∧ T (s8, i8, s9)∧ F (s9)
〈i0, . . . , i8〉 = 〈gas, acc, button, dec, dec, gas, dec, brake, button〉
Concretisation: Computing the Test Chain
I = F
OFF,0,FALSE
OFF,1,FALSE OFF,0,TRUE
ON,1,TRUE
DIS,2,TRUE DIS,0,TRUE
OFF,2,FALSE OFF,2,TRUE
brake ∨ dec
brake ∨ dec
brake∨ dec
gas ∨ acc
gas ∨ accgas∨ acc
acc ∨ dec (p1)
gas ∨ acc
brake ∨ dec button button
button
button
button (p4)button
button
button
gas ∨ acc
gas ∨ acc
brake (p3)gas
∨ accgasbrake
∨ decbrake∨ dec(p2)
brake ∨ dec
Optimality
The test case chain is minimal if
(1) the program and the properties admit a test chain,
(2) all test goals are singleton sets, and
(3) the test chain visits each property once in the K -reachabilitygraph.
Optimality
The test case chain is minimal if
(1) the program and the properties admit a test chain,
(2) all test goals are singleton sets, and
(3) the test chain visits each property once in the K -reachabilitygraph.
ϕ1 ϕ3
ϕ2w1
w2
Optimality
The test case chain is minimal if
(1) the program and the properties admit a test chain,
(2) all test goals are singleton sets, and
(3) the test chain visits each property once in the K -reachabilitygraph.
ϕ1 ϕ3
ϕ2w1
w2
w < w1 + w2
Optimality
The test case chain is minimal if
(1) the program and the properties admit a test chain,
(2) all test goals are singleton sets, and
(3) the test chain visits each property once in the K -reachabilitygraph.
ϕ1 ϕ3
ϕ2w1
w2
w < w1 + w2
Reachability diameter d = length of maximum, shortest pathbetween any two states
There is a K ≤ d such that, under the preconditions (1) and (2),the test chain is minimal.
In practice, fix a bound K and obtain minimised chain.
Optimality
The test case chain is minimal if
(1) the program and the properties admit a test chain,
(2) all test goals are singleton sets, and
(3) the test chain visits each property once in the K -reachabilitygraph.
ϕ1 ϕ3
ϕ2w1
w2
w < w1 + w2
Reachability diameter d = length of maximum, shortest pathbetween any two states
There is a K ≤ d such that, under the preconditions (1) and (2),the test chain is minimal.
In practice, fix a bound K and obtain minimised chain.
Multi-State Test Goals
I ϕ1 ϕ2 F0 1
2
2
2
2
p1 : G(mode = OFF ∧ ¬enable ∧ button ⇒ X enable
)
p2 : G(mode = ON ∧ brake ⇒ X(mode = DIS)
)
Multi-State Test Goals
I ϕ1 ϕ2 F
011
2
2
2
2
Broken chain
Path 〈I , ϕ1, ϕ2〉 not feasible in a single step, but requires twosteps.
Multi-State Test Goals
I ϕ1 ϕ2 F1 1
2
2
2
2
Broken chain
Path 〈I , ϕ1, ϕ2〉 not feasible in a single step, but requires twosteps.
Chain repair
Systematically increase edge weights of failed subpath
Minimality lost
Multi-State Test Goals
I ϕ1 ϕ2 F1 1
2
2
2
2
Broken chain
Path 〈I , ϕ1, ϕ2〉 not feasible in a single step, but requires twosteps.
Chain repair
Systematically increase edge weights of failed subpath
Minimality lost
Completeness
Succeeds if path admits chain in concrete program
If for each test goal the states are strongly connected
In practice: many systems are (almost) strongly connected.
Abstraction Refinement
I ϕ1 ϕ2 F0 1
2
2
2
2
Abstraction refinement: Find any path
I ϕ1 ϕ2 F0 1
22
22
2
2
Abstraction Refinement
I ϕ1 ϕ2 F0 1
2
2
2
2
Abstraction refinement: Optimise with TSP solver
I ϕ1 ϕ2 F1
2
22
2
2
Abstraction Refinement
I ϕ1 ϕ2 F0 1
2
2
2
2
Abstraction refinement: Optimise with TSP solver
I ϕ1 ϕ2 F1
2
22
2
2
General Case
Completeness
Not strongly connected systems:
Abstraction refinementMore general solver than TSP solver, e.g. ASP solver
General Case
Completeness
Not strongly connected systems:
Abstraction refinementMore general solver than TSP solver, e.g. ASP solver
Multiple chains :
Partitioning by graph colouring
General Case
Completeness
Not strongly connected systems:
Abstraction refinementMore general solver than TSP solver, e.g. ASP solver
Multiple chains :
Partitioning by graph colouring
Optimality
Would require to optimise over concrete system
In practice, minimised rather than minimal solutions relevant
Implementation
Properties specified as C functions:
void p 1 ( i o t ∗ i , s t a t e t ∗ s ) {CPROVER assume ( s−>mode==ON && s−>speed==1 && i−>dec ) ;
compute ( i , s ) ;a s s e r t ( s−>speed ==1) ;
}
Woven into program during test case generation.
BMC engine of Cbmc
Property reachability graph construction
Exploits incremental SAT solving
Chain repair by concrete chaining
Lkh travelling salesman problem solverClingo answer set programming solver
Benchmarks and Comparison
Benchmarks
Cruise control model
Window controller
Car alarm system
Elevator model
Robot arm model
Comparison with
FShell: a BMC-based test generator with test suiteminimisation
Random case generator with test suite minimisation
Klee: a test case generator based on symbolic execution
Results: Test Case Length
ut ut ut ut ut utut
utut
utut
ut
ut
* * * * * * * **
**
**
ld ld ld ldld
ldld
ld
ld
ld
ld
rs rsrs
rsrs
rs
rs
0
50
100
150
200
250
300
350
400
450
500
550
600
650
0 1 2 3 4 5 6 7 8 9 10 11 12
Number of benchmarks
Accu
mu
late
dte
stca
se
len
gth
s
� KLEE
♦ RANDOMTEST
∗ FSHELL
△ CHAINCOVER
Results: Test Case Generator Runtime
0
1000
2000
3000
4000
5000
0 1 2 3 4 5 6 7 8 9 10 11 12
ut ut ut ut ut ut ut ut ut utut
ut
* * * * **
**
*
*
*
*
ld ld ld ld ld ld ld ld ld ldrs rs rs rs rsrs
Accu
mu
late
dru
ntim
es
Number of benchmarks
� KLEE
♦ RANDOMTEST
∗ FSHELL
△ CHAINCOVER
Summary and Current Work
Summary
Test chain for reactive systems
Test goals from requirements, specification model, codecoverage criteria
Minimal test chain for single-state test goals, otherwiseheuristics
Experimental evaluation
Application: on-target testing, acceptance testing
Current work
Integrate acceleration to handle deep loops
Test chains for code coverage criteria, e.g. MC/DC
Further questions
Incremental test chain generation
In the case of model modificationsWhen test execution gets stuck due to a failed test goal