testing techniques

34
Testing Techniques Testing with Finite State Machines Ed Brinksma course 2004

Upload: angus

Post on 01-Feb-2016

40 views

Category:

Documents


0 download

DESCRIPTION

Testing Techniques. Testing with Finite State Machines. Ed Brinksma course 2004. This Lecture : Overview. Testing with formal methods: Generic framework Testing based on Labelled Transition Systems - ioco Testing based on Finite State Machines (FSM) Now: FSM State based testing - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Testing Techniques

Testing Techniques

Testing withFinite State Machines

Ed Brinksma

course 2004

Page 2: Testing Techniques

TT 2004, FSM 2© Ed Brinksma/Jan Tretmans

This Lecture : Overview

Testing with formal methods: Generic framework Testing based on Labelled Transition Systems - ioco Testing based on Finite State Machines (FSM)

Now: FSM State based testing H. Ural, Formal methods for test sequence generation,

Computer Communications, 15(5), 1992. Other literature:

D. Lee and M. Yannakakis,Principles and methods of testing finite state machines - A survey.The Proceedings of the IEEE 84, August 1996.

Page 3: Testing Techniques

TT 2004, FSM 3© Ed Brinksma/Jan Tretmans

State Machines

Many systems can be specified / modelled as state machines

State machines as the basis for testing : FSM : Finite State Machine

black box

specification based

reactive systems : communication protocols

control systems

embedded systems

Page 4: Testing Techniques

TT 2004, FSM 4© Ed Brinksma/Jan Tretmans

State Machines

States Transitions Inputs ( “triggers”) Outputs

outputinput

state

new state

Page 5: Testing Techniques

TT 2004, FSM 5© Ed Brinksma/Jan Tretmans

Coffee Machine : State Graph

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 6: Testing Techniques

TT 2004, FSM 6© Ed Brinksma/Jan Tretmans

Coffee Machine : State Table

conventions (from Z):

plain name: state

name?: input

name!: output

Table gives new state and output as function of state and

input

State 0 5 10

Input

stui? 5 / - 10 / - 10 / stui!

dub? 10 /- 10 / stui! 10 / dub!

koffie? 0 / - 5 / - 0 / koffie!

Page 7: Testing Techniques

TT 2004, FSM 7© Ed Brinksma/Jan Tretmans

State Machine : FSM Model

FSM - Finite State Machine - or Mealy Machine is 5-tuple

M = ( S, I, O, , )

S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function

usually we also indicate an initial state

Natural extension to sequences : : S x I* S : S x I* O*

Page 8: Testing Techniques

TT 2004, FSM 8© Ed Brinksma/Jan Tretmans

State Machine : FSM Model

FSM restrictions:

deterministic:

: S x I S and : S x I O are functions

completely specified:

: S x I S and : S x I O are complete functions

( empty output is allowed; sometimes implicit completeness )

strongly connected:

from any state any other state can be reached,

or any state can be reached from the initial state

reduced:

there are no equivalent states

Page 9: Testing Techniques

TT 2004, FSM 9© Ed Brinksma/Jan Tretmans

Coffee Machine FSM Model

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 10: Testing Techniques

TT 2004, FSM 10© Ed Brinksma/Jan Tretmans

Testing with FSM

Given: a specification FSM MS a ( black box ) implementation FSM MI

determine whether MI conforms to MS i.e., MI behaves in accordance with MS i.e., whether outputs of MI are the same as of MS i.e., whether the reduced MI is equivalent to MS

Possible errors: extra or missing states output fault transition fault

to other state to new state

Page 11: Testing Techniques

TT 2004, FSM 11© Ed Brinksma/Jan Tretmans

FSM Testing

Test with paths of the (specification) FSM

Path = sequence of inputs with expected outputs

( cf. path testing as white-box technique)

Infinitely many paths : how to select ?

Different strategies :

test every state : state coverage (of specification !)

test every transition : transition coverage

test output of every transition

test output + resulting state of every transition

Page 12: Testing Techniques

TT 2004, FSM 12© Ed Brinksma/Jan Tretmans

FSM State Testing Make State Tour that covers every state

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Test sequence : stui? dub? koffie?

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 13: Testing Techniques

TT 2004, FSM 13© Ed Brinksma/Jan Tretmans

FSM Transition Tour Make Transition Tour that covers every transition

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Test sequence :

koffie? stui? koffie? stui? stui? dub? koffie? dub? koffie? stui? dub? koffie?

Page 14: Testing Techniques

TT 2004, FSM 14© Ed Brinksma/Jan Tretmans

FSM Transition Testing One big tour as test case not always desirable

( too long, too complex, difficult to analyse, not specific ) Make test case for every transition separately:

S1 S2a? / x!

Test transition :

Go to state S1

Apply input a?

Check output x!

Verify state S2 ( optionally )

Test purpose: “Test whether the system, when in state S1,

produces output x! on input a? and goes to state S2”

Page 15: Testing Techniques

TT 2004, FSM 15© Ed Brinksma/Jan Tretmans

FSM Transition Testing

Go to state S1 :

synchronizing sequence brings machine to particular state,

say S0, from any state ( but synchronizing sequence may not

exist )

or: use reset transition if available

go from S0 to S1

( always possible because of determinism and completeness )

Page 16: Testing Techniques

TT 2004, FSM 16© Ed Brinksma/Jan Tretmans

FSM Transition Testing

synchronizing sequence :

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

dub? koffie?

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -

dub? / dub!

koffie? / -koffie? / - 0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -

dub? / dub!

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 17: Testing Techniques

TT 2004, FSM 17© Ed Brinksma/Jan Tretmans

FSM Transition Testingsynchronizing sequence :

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

dub? koffie?

koffie? / - 0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -

dub? / dub!

To test dub? / stui! : go to state 5 by : dub? koffie? stui?

Page 18: Testing Techniques

TT 2004, FSM 18© Ed Brinksma/Jan Tretmans

FSM Transition Testing

koffie? / - 0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -

dub? / dub!

•To test dub? / stui! : •go to state 5 by : dub? koffie? stui?•give input dub?•check output stui!•verify that machine is in state 10

Page 19: Testing Techniques

TT 2004, FSM 19© Ed Brinksma/Jan Tretmans

FSM Transition Testing

State identification and verification : Apply sequence of inputs in the current state of the FSM

such that from the outputs we can identify that state where we started; or verify that we were in a particular start state

Different kinds of sequences UIO sequences ( Unique Input Output sequence, SIOS) Distinguishing sequence ( DS ) W - set ( characterizing set of sequences ) UIOv SUIO Single UIO MUIO Multiple UIO Overlapping UIO

Page 20: Testing Techniques

TT 2004, FSM 20© Ed Brinksma/Jan Tretmans

FSM Transition Testing

State verification : UIO sequences

sequence x that distinguishes state s from all other states :for all t s : ( s, x ) ( t, x )

each state has its own UIO sequence UIO sequences may not exist

Distinguishing sequence sequence x that produces different output for each state :

for all pairs t, s with t s : ( s, x ) ( t, x ) a distinguishing sequence may not exist

W - set of sequences set of sequences W which can distinguish any pair of states :

for all pairs t s there is x W : ( s, x ) ( t, x ) W - set always exists for reduced FSM

Page 21: Testing Techniques

TT 2004, FSM 21© Ed Brinksma/Jan Tretmans

FSM Transition TestingUIO sequences

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

state 0 : stui? / - koffie? / - state 5 : dub? / stui!state 10 : koffie? / koffie!

Page 22: Testing Techniques

TT 2004, FSM 22© Ed Brinksma/Jan Tretmans

FSM Transition TestingDS sequence

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

DS sequence : dub? output state 0 : - output state 5 : stui!output state 10 : dub!

Page 23: Testing Techniques

TT 2004, FSM 23© Ed Brinksma/Jan Tretmans

FSM Transition Testing

koffie? / - 0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -

dub? / dub!

•To test dub? / stui! :

Test case : dub? / * koffie? / * stui? / - dub? / stui! koffie? / koffie!

go to state 5 : dub? koffie? stui?give input dub? check output stui!Apply UIO of state 10 : koffie? / koffie!

Page 24: Testing Techniques

TT 2004, FSM 24© Ed Brinksma/Jan Tretmans

FSM Transition Testing

- 9 transitions / test cases for coffee machine

- if end-state of one corresponds with start-state of next then concatenate

- different ways to optimize and remove overlapping / redundant parts

- there are (academic) tools to support this

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 25: Testing Techniques

TT 2004, FSM 25© Ed Brinksma/Jan Tretmans

FSM Transition Testing

Test transition : Go to state S1 Apply input a? Check output x! Verify state S2

Checks every output fault and transfer fault (to existing state) If we assume that

the number of states of the implementation machine MI is less than or equal tothe number of states of the specification machine to MS.

then testing all transitions in this wayleads to equivalence of reduced machines,i.e., complete conformance

Page 26: Testing Techniques

TT 2004, FSM 26© Ed Brinksma/Jan Tretmans

FSM: variations on this theme

there exists many variations on this theme: Moore machines:

output determined by state instead of transition Infinite state machines:

infinite number of states (e.g. state contains variable) Non-deterministic FSM:

transition relation instead of transition function Labelled Transition Systems - ioco

...

Page 27: Testing Techniques

TT 2004, FSM 27© Ed Brinksma/Jan Tretmans

FSM: Moore machines

Mealy Machine is 5-tuple: M = ( S, I, O, , ) S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function

Moore Machine is 5-tuple: M = ( S, I, O, , ) S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S O output function, not dependent of

input usually we add an initial state

Page 28: Testing Techniques

TT 2004, FSM 28© Ed Brinksma/Jan Tretmans

Coffee Machine: Mealy model

0

10

5

stui? / -

stui? / -

dub? / stui!koffie? / koffie!

stui? / stui!

dub? / -

koffie? / -koffie? / -

dub? / dub!

Page 29: Testing Techniques

TT 2004, FSM 29© Ed Brinksma/Jan Tretmans

Coffee Machine: Moore model

0/-

100/-

5/-stui?

stui?

dub

?

koffie?

stui?

dub?

koffie?

koffie?

dub?

K/koffie! 1010/10!

105/5!dub?

dub? dub?

koffie?

koffi

e?

stui?

koffie?

stui? stu

i?

Page 30: Testing Techniques

TT 2004, FSM 30© Ed Brinksma/Jan Tretmans

Moore Coffee Machine : State Table

State 0 5 100 105 1010 K

output - - - 5! 10! koffie!

Input

stui? 5 100 105 105 105 5

dub? 100 105 1010 1010 1010 100

koffie? 0 1010 K K K 0

Compared to Mealy machine:

more states

simpler output function (not dependent on input)

changes hardly anything for testing

Page 31: Testing Techniques

TT 2004, FSM 31© Ed Brinksma/Jan Tretmans

Infinite - State Machines

States : data structures / data bases Inputs ( “triggers”) : operations on data bases Transitions : new data base states Outputs : results of queries

outputinput

state

new state

Page 32: Testing Techniques

TT 2004, FSM 32© Ed Brinksma/Jan Tretmans

State Machine : FSM Model

FSM - Finite State Machine - or Mealy Machine is 5-tuple

M = ( S, I, O, , )

S finite set of states I finite set of inputs O finite set of outputs : S x I S transfer function : S x I O output function

Natural extension to sequences : : S x I* S : S x I* O*

Page 33: Testing Techniques

TT 2004, FSM 33© Ed Brinksma/Jan Tretmans

Infinite State Machine Testing

Not all transitions can be tested ….. But principle remains the same : test transitions

Go to start state of transition Apply input Check output Verify result state

Use selection techniques to select transitions / start states equivalence partitioning boundary value analysis ……...

state

input output

new state

Page 34: Testing Techniques

TT 2004, FSM 34© Ed Brinksma/Jan Tretmans

FSM Testing vs. InfSM Testing

Restrictions on FSM: deterministic

completeness

FSM has always alternation between input and output

Difficult to specify interleaving in FSM

FSM is not compositional

FSM has “more intuitive” theory

FSM test suite is complete-- but only w.r.t. assumption on number of states

FSM test theory has been around for a number of years