testing techniques
DESCRIPTION
Testing Techniques. Testing with Finite State Machines. Ed Brinksma course 2004. This Lecture : Overview. Testing with formal methods: Generic framework Testing based on Labelled Transition Systems - ioco Testing based on Finite State Machines (FSM) Now: FSM State based testing - PowerPoint PPT PresentationTRANSCRIPT
Testing Techniques
Testing withFinite State Machines
Ed Brinksma
course 2004
TT 2004, FSM 2© Ed Brinksma/Jan Tretmans
This Lecture : Overview
Testing with formal methods: Generic framework Testing based on Labelled Transition Systems - ioco Testing based on Finite State Machines (FSM)
Now: FSM State based testing H. Ural, Formal methods for test sequence generation,
Computer Communications, 15(5), 1992. Other literature:
D. Lee and M. Yannakakis,Principles and methods of testing finite state machines - A survey.The Proceedings of the IEEE 84, August 1996.
TT 2004, FSM 3© Ed Brinksma/Jan Tretmans
State Machines
Many systems can be specified / modelled as state machines
State machines as the basis for testing : FSM : Finite State Machine
black box
specification based
reactive systems : communication protocols
control systems
embedded systems
TT 2004, FSM 4© Ed Brinksma/Jan Tretmans
State Machines
States Transitions Inputs ( “triggers”) Outputs
outputinput
state
new state
TT 2004, FSM 5© Ed Brinksma/Jan Tretmans
Coffee Machine : State Graph
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 6© Ed Brinksma/Jan Tretmans
Coffee Machine : State Table
conventions (from Z):
plain name: state
name?: input
name!: output
Table gives new state and output as function of state and
input
State 0 5 10
Input
stui? 5 / - 10 / - 10 / stui!
dub? 10 /- 10 / stui! 10 / dub!
koffie? 0 / - 5 / - 0 / koffie!
TT 2004, FSM 7© Ed Brinksma/Jan Tretmans
State Machine : FSM Model
FSM - Finite State Machine - or Mealy Machine is 5-tuple
M = ( S, I, O, , )
S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function
usually we also indicate an initial state
Natural extension to sequences : : S x I* S : S x I* O*
TT 2004, FSM 8© Ed Brinksma/Jan Tretmans
State Machine : FSM Model
FSM restrictions:
deterministic:
: S x I S and : S x I O are functions
completely specified:
: S x I S and : S x I O are complete functions
( empty output is allowed; sometimes implicit completeness )
strongly connected:
from any state any other state can be reached,
or any state can be reached from the initial state
reduced:
there are no equivalent states
TT 2004, FSM 9© Ed Brinksma/Jan Tretmans
Coffee Machine FSM Model
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 10© Ed Brinksma/Jan Tretmans
Testing with FSM
Given: a specification FSM MS a ( black box ) implementation FSM MI
determine whether MI conforms to MS i.e., MI behaves in accordance with MS i.e., whether outputs of MI are the same as of MS i.e., whether the reduced MI is equivalent to MS
Possible errors: extra or missing states output fault transition fault
to other state to new state
TT 2004, FSM 11© Ed Brinksma/Jan Tretmans
FSM Testing
Test with paths of the (specification) FSM
Path = sequence of inputs with expected outputs
( cf. path testing as white-box technique)
Infinitely many paths : how to select ?
Different strategies :
test every state : state coverage (of specification !)
test every transition : transition coverage
test output of every transition
test output + resulting state of every transition
TT 2004, FSM 12© Ed Brinksma/Jan Tretmans
FSM State Testing Make State Tour that covers every state
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
Test sequence : stui? dub? koffie?
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 13© Ed Brinksma/Jan Tretmans
FSM Transition Tour Make Transition Tour that covers every transition
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
Test sequence :
koffie? stui? koffie? stui? stui? dub? koffie? dub? koffie? stui? dub? koffie?
TT 2004, FSM 14© Ed Brinksma/Jan Tretmans
FSM Transition Testing One big tour as test case not always desirable
( too long, too complex, difficult to analyse, not specific ) Make test case for every transition separately:
S1 S2a? / x!
Test transition :
Go to state S1
Apply input a?
Check output x!
Verify state S2 ( optionally )
Test purpose: “Test whether the system, when in state S1,
produces output x! on input a? and goes to state S2”
TT 2004, FSM 15© Ed Brinksma/Jan Tretmans
FSM Transition Testing
Go to state S1 :
synchronizing sequence brings machine to particular state,
say S0, from any state ( but synchronizing sequence may not
exist )
or: use reset transition if available
go from S0 to S1
( always possible because of determinism and completeness )
TT 2004, FSM 16© Ed Brinksma/Jan Tretmans
FSM Transition Testing
synchronizing sequence :
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
dub? koffie?
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -
dub? / dub!
koffie? / -koffie? / - 0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -
dub? / dub!
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 17© Ed Brinksma/Jan Tretmans
FSM Transition Testingsynchronizing sequence :
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
dub? koffie?
koffie? / - 0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -
dub? / dub!
To test dub? / stui! : go to state 5 by : dub? koffie? stui?
TT 2004, FSM 18© Ed Brinksma/Jan Tretmans
FSM Transition Testing
koffie? / - 0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -
dub? / dub!
•To test dub? / stui! : •go to state 5 by : dub? koffie? stui?•give input dub?•check output stui!•verify that machine is in state 10
TT 2004, FSM 19© Ed Brinksma/Jan Tretmans
FSM Transition Testing
State identification and verification : Apply sequence of inputs in the current state of the FSM
such that from the outputs we can identify that state where we started; or verify that we were in a particular start state
Different kinds of sequences UIO sequences ( Unique Input Output sequence, SIOS) Distinguishing sequence ( DS ) W - set ( characterizing set of sequences ) UIOv SUIO Single UIO MUIO Multiple UIO Overlapping UIO
TT 2004, FSM 20© Ed Brinksma/Jan Tretmans
FSM Transition Testing
State verification : UIO sequences
sequence x that distinguishes state s from all other states :for all t s : ( s, x ) ( t, x )
each state has its own UIO sequence UIO sequences may not exist
Distinguishing sequence sequence x that produces different output for each state :
for all pairs t, s with t s : ( s, x ) ( t, x ) a distinguishing sequence may not exist
W - set of sequences set of sequences W which can distinguish any pair of states :
for all pairs t s there is x W : ( s, x ) ( t, x ) W - set always exists for reduced FSM
TT 2004, FSM 21© Ed Brinksma/Jan Tretmans
FSM Transition TestingUIO sequences
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
state 0 : stui? / - koffie? / - state 5 : dub? / stui!state 10 : koffie? / koffie!
TT 2004, FSM 22© Ed Brinksma/Jan Tretmans
FSM Transition TestingDS sequence
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
DS sequence : dub? output state 0 : - output state 5 : stui!output state 10 : dub!
TT 2004, FSM 23© Ed Brinksma/Jan Tretmans
FSM Transition Testing
koffie? / - 0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -
dub? / dub!
•To test dub? / stui! :
Test case : dub? / * koffie? / * stui? / - dub? / stui! koffie? / koffie!
go to state 5 : dub? koffie? stui?give input dub? check output stui!Apply UIO of state 10 : koffie? / koffie!
TT 2004, FSM 24© Ed Brinksma/Jan Tretmans
FSM Transition Testing
- 9 transitions / test cases for coffee machine
- if end-state of one corresponds with start-state of next then concatenate
- different ways to optimize and remove overlapping / redundant parts
- there are (academic) tools to support this
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 25© Ed Brinksma/Jan Tretmans
FSM Transition Testing
Test transition : Go to state S1 Apply input a? Check output x! Verify state S2
Checks every output fault and transfer fault (to existing state) If we assume that
the number of states of the implementation machine MI is less than or equal tothe number of states of the specification machine to MS.
then testing all transitions in this wayleads to equivalence of reduced machines,i.e., complete conformance
TT 2004, FSM 26© Ed Brinksma/Jan Tretmans
FSM: variations on this theme
there exists many variations on this theme: Moore machines:
output determined by state instead of transition Infinite state machines:
infinite number of states (e.g. state contains variable) Non-deterministic FSM:
transition relation instead of transition function Labelled Transition Systems - ioco
...
TT 2004, FSM 27© Ed Brinksma/Jan Tretmans
FSM: Moore machines
Mealy Machine is 5-tuple: M = ( S, I, O, , ) S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function
Moore Machine is 5-tuple: M = ( S, I, O, , ) S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S O output function, not dependent of
input usually we add an initial state
TT 2004, FSM 28© Ed Brinksma/Jan Tretmans
Coffee Machine: Mealy model
0
10
5
stui? / -
stui? / -
dub? / stui!koffie? / koffie!
stui? / stui!
dub? / -
koffie? / -koffie? / -
dub? / dub!
TT 2004, FSM 29© Ed Brinksma/Jan Tretmans
Coffee Machine: Moore model
0/-
100/-
5/-stui?
stui?
dub
?
koffie?
stui?
dub?
koffie?
koffie?
dub?
K/koffie! 1010/10!
105/5!dub?
dub? dub?
koffie?
koffi
e?
stui?
koffie?
stui? stu
i?
TT 2004, FSM 30© Ed Brinksma/Jan Tretmans
Moore Coffee Machine : State Table
State 0 5 100 105 1010 K
output - - - 5! 10! koffie!
Input
stui? 5 100 105 105 105 5
dub? 100 105 1010 1010 1010 100
koffie? 0 1010 K K K 0
Compared to Mealy machine:
more states
simpler output function (not dependent on input)
changes hardly anything for testing
TT 2004, FSM 31© Ed Brinksma/Jan Tretmans
Infinite - State Machines
States : data structures / data bases Inputs ( “triggers”) : operations on data bases Transitions : new data base states Outputs : results of queries
outputinput
state
new state
TT 2004, FSM 32© Ed Brinksma/Jan Tretmans
State Machine : FSM Model
FSM - Finite State Machine - or Mealy Machine is 5-tuple
M = ( S, I, O, , )
S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function
Natural extension to sequences : : S x I* S : S x I* O*
TT 2004, FSM 33© Ed Brinksma/Jan Tretmans
Infinite State Machine Testing
Not all transitions can be tested ….. But principle remains the same : test transitions
Go to start state of transition Apply input Check output Verify result state
Use selection techniques to select transitions / start states equivalence partitioning boundary value analysis ……...
state
input output
new state
TT 2004, FSM 34© Ed Brinksma/Jan Tretmans
FSM Testing vs. InfSM Testing
Restrictions on FSM: deterministic
completeness
FSM has always alternation between input and output
Difficult to specify interleaving in FSM
FSM is not compositional
FSM has “more intuitive” theory
FSM test suite is complete-- but only w.r.t. assumption on number of states
FSM test theory has been around for a number of years