online testing of nondeterministic systems with the reactive planning tester

Online Testing ofNondeterministic Systems

withthe Reactive Planning

TesterJüri VainDept. of Computer ScienceTallinn University of Technology

J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

1

Reference


2

Vain, Jüri; Kääramees, Marko; Markvardt, Maili Online testing of nondeterministic systems with reactive planning tester. In: Petre, L.; Sere, K.; Troubitsyna, E. (Eds.). Dependability and Computer Engineering : Concepts for Software-Intensive Systems. Hershey, PA: IGI Global (2012), pages 113-150.

Outline Preliminaries

Model-Based Testing Online testing

Reactive Planning Tester (RPT) Synthesis of the RPT Performance issues Test execution environment dTron Demo


3

4

What is testing for? to check the quality (functionality, reliability,

performance, …) of an (software) object

-by performing experiments-in a controlled way

In avg. 10-20 errors per 1000 LOC30-50 % of development time and cost in software


5

What is a Test?

Software under Test

(SUT)

Test Data Output

Test Cases

Correct result?

Oracle



6

Model-Based Testing

(typically)

Specifics of testing embedded systems


7

– non-determinism– partial observability – RT constraints– dependability test coverage issues

How to address them in testing?

Real environment in the loop!

Option 1: Offline vs Online testing

Offline testing Open “control” loop Test is not adaptive to outputs or timing of SUT Test planning – result analysis loop is long

Online testing Flexible, test control is based on SUT feedback One test covers usually many test cases


8

Online testing Group of test generation and execution

algorithms that compute successive stimuli at runtime directed

by the test purpose and the observed outputs of the SUT


9

Tester

SUT

Online testing Advantages:

The state-space explosion problem is reduced because only a limited part of the state-space needs to be kept track of at any point in time.

Drawbacks: Exhaustive planning is diffcult due to the

limitations of the available computational resources at the time of test execution.


10

Option 2: Test scripting vs model-based generation


11

Scripting: does not need SUT modelling + sensitive to human errors - inflexible, needs rewriting even with small

changes of SUT specs - hard to achieve test coverage -

Model-based generation: considerable SUT modelling effort - correctness of tests is verifiable + easy to modify and regenerate + clear characteristics of coverage +

Model-Based Testing Given

Model of the SUT specification System Under Test (SUT), The test goal (in terms of spec. model elements)

Find If the SUT conforms to the specification in terms

expressed in the test goal.


NEEDED! - Sufficiently rich modelling formalism,- Supporting tool set.

12

Model-Based Testing We assume formal specs as:

UML State Charts Extended Finite State Machines MSC OCL

etc.

UPTA -Timed Automata (timing, parallelism, test data)


13

”Relativized Real-Time i/o conformance” Relation

Spec = UppAal Timed Automata Network: Env || IUT

Timed Trace: i1.2½.o1.3.o2.19.i2.5.i3

Expected system reactionInput event ordering

16J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

Online MBT architecture: UppAal-TRON

[www.uppaal.com]

Bottleneck of online MBT: planning

Spectrum of planning methods Random walk (RW): select test stimuli in random

inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved

RW with reinforcement learning (anti-ant) the exploration is guided by some reward function

........ Exploration with exhaustive planning

MC provides possibly an optimal witness trace the size of the model is critical in explicit state MC state explosion in "combination lock" or deep loop models




inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved

RW with reinforcement learning (anti-ant) the exploration is guided by reward function



???


Full space

1 stepahead

Zero planning



inefficient - random exploration of the state space test cases that are unreasonably long may leave the test purpose unachieved

RW with reinforcement learning (anti-ant) the exploration is guided by some reward function



Planning with adaptive horizon!


Full space

1 stepahead

Zero planning

Reactive Planning in a Nutshell (1) Instead of a complete plan, only a set of

decision rules is derived The rules direct the system towards the

test goal. Based on current situation evaluation just

one subsequent input is computed at a time.

Planning horizon is adjusable


20

Reactive Planning: Planning cycle Identify current state of SUT:

Observe the output (or history) of the SUT

Pick the next move: Select one from unsatisfied test (sub-)goals Gi

Compute the best strategy for Gi: Gain function guides the exploration of the model

(choose the transition with the shortest path to the (sub-)goal Gi)


21


Reactive Planning Tester in a Nutshell (2)

State model of the IUT Test goal

Reactive Planning Tester (RPT) model

Reachability Analysis

Reactive Planning Tester (RPT)

implementation

SUT

Offline phase Online phase

Stimuli / responses

RPT is another

state model22

RPT autonomously generates stimuli to

reach the goal

Constructing the RPT model


23

Model of SUT

Test goal

Model of RPT

Test strategy parameters:- planning

horizon- timeouts

Synthesis of RPT

Extraction of the control strcture

Constructing gain guards

(GG)

Constructing gain functions

Reduction of GG

RPT: Key Assumptions Testing is guided by the (EFSM) model of the

tester and the test goal. Decision rules of reactive planning are

encoded in the guards of the transitions of the tester model.

The SUT model is presented as an output observable nondeterministic EFSM in which all paths are feasible1.

1 - A. Y. Duale and M. U. Uyar. A method enabling feasible conformance test sequence generation for EFSM models. IEEE Trans. Comput.,53(5):614–627,2004.


24

Example: Nondeterministic FSM

s2

s3

e6:i6/o6

e7:i7/o7

s1

e0:i0/o0

e1:i0/o1

e2:i2/o2

e3:i3/o3

e4:i3/o4

e5:i5/o5

i0 and i3 are output observable nondeterministic inputsJ.Vain “...Reactive Planning Testing...”Abo,

Feb 3, 201225

Encoding the Test Goal in SUT Model Trap - a boolean variable assignment

attached to the transitions of the IUT model

A trap variable is initially set to false.

The trap update functions are executed (set to true) when the transition is visited.


26

Goal directed testing: The power of traps

Several goals can be expressed all/selected transitions

transition sequences (traps with reference to other traps)

advanced goals using auxiliary variables, consequent transitions, repeated pass, …

traps with 1st order predicates data variables

Properties not expressible by traps Assertions/invariants – always/never properties

The model specifies only the allowed behaviours

Åbo Akademi University, Dec 2011

Add Test Purpose

s2

s3

e6:i6/o6, t6=truee7:

i7/o7, t7=true

s1

e0:i0/o0t0=true

e1:i0/o1, t1=true

e2:i2/o2, t2=true

e3:i3/o3, t3=true

e4:i3/o4, t4=true

e5:i5/o5, t5=true

bool t0 = false;...bool t7 = false;


28


Example: Adding the Test Goal

s2

s3


i7/o7, t7=true

s1s1

e0:i0/o0

t0=true

e1:i0/o1, t1=true

e2:i2/o2, t2=true

e3:i3/o3, t3=true

e4:i3/o4, t4=true

e5:i5/o5, t5=true

Initial values:bool t0 = false;...bool t7 = false;

Trap variables are initially set to

false

Trap update functions are executed (set to

true) when the transition is visited

Test goal is defined by trap variables ti attached

to transitions

29

Model of the tester Generated from the SUT model decorated with test purpose Transition guards encode the rules of online planning 2 types of tester states:

active – tester controls the next move passive – SUT controls the next move

2 types of transitions: Observable – source state is a passive state (guard true), Controllable – source state is an active state (guard pS /\ pT where pS – guard of the SUT transition; pT – gain guard)

The gain guard (defined on trap variables) must ensure that only the outgoing edges with maximum gain are enabled in the given state.


30

Construction of the Tester Sceleton

s2

s3

e6:i6/o6, t6=true

e7:i7/o7, t7=true

s1

e0:i0/o0t0=true

e1:i0/o1, t1=true

e2:i2/o2, t2=true

e3:i3/o3, t3=true

e4:i3/o4, t4=true

e5:i5/o5, t5=true

s1

sa

s2sb

sc

s3

sd se

sf


States: Transitions: - active - observable - passive - controllable

31

Add IO and Gain Guardss1

s4

s2s5

s6

s3

s6 s9

s7

eo0:

o0/t0=true eo

1:o1/t1=true

ec01:

[pc01(T)]

-/i0 eo

2:o2/t2=true

eo7:

o7/t7=true

ec2:

[pc2(T)]

-/i2

ec34:

[pc34(T)]

-/i3eo

3:o3/t3=true

eo6:

o6/t6=true ec6:

[pc6(T)]

-/i6 ec7:

[pc7(T)]

-/i7

ec5:

[pc5(T)]

-/i5

eo5:

o5/t5=true


32

Constructing the gain guards (GG): intuition

GG must guarantee that each transition enabled by GG is a prefix of some locally

optimal (w.r.t. test purpose) path; tester should terminate after the test goal is reached or

all unvisited traps are unreachable from the current state;

to have a quantitative measure of the gain of executing any transition e we define a gain function ge that returns a distance weighted sum of unsatisfied traps that are reachable along e.


33

Constructing Gain Guards: intuition


Observed current state

Alternative choices

...tri trj trk

...

Planning cones to be covered for decision making

34

Constructing Gain Guards: intuition


tri trj trk

Alternative continuation choices

...

ge(tri,T )

...

ge(trj,T ) ge(trj,T )

Gain functions characterize planning options

35

Current state

Constructing the gain guards: properties of gain functions

ge = 0, if it is useless to fire the transition e from the current state with the current variable bindings;

ge > 0, if fireing the transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap;

gei > gej for transitions ei and ej with the same source state, if taking the transition ei leads to unvisited traps with smaller distance than taking the transition ej;

Having gain function ge with given properties define GG:

pT (ge = maxk gek) ge > 0


36

Constructing the Gain Functions: shortest path trees

Reachability problem of trap labelled transitions can be reduced to single-source shortest path problem1.

Arguments of the gain function ge are then the shortest path tree TRe with root node e VT – vector of trap variables

To construct TRe we create a dual graph G = (VD,ED) of the tester control graph MT where the vertices VD of G correspond to the transitions of the MT, the edges ED of G represent the pairs of consequtive

transitions sharing a state in MT (2-switches)

1 - Fredman & Tarjan 1987 O(E + V log V)J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

37

http://en.wikipedia.org/wiki/Shortest_path_problem



Constructing the Gain Guards: shortest path tree

(example)

The dual graph of the tester model The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition ec

01


38

Constructing the gain guards:

gain function (1) Represent the reduced tree TR(ei, G) as a set of elementary sub-trees

each specified by the production i j{1,..n}j

Rewrite the right-hand sides of the productions as arithmetic terms:

(3) ti - trap variable ti lifted to type N, c - constant for rescaling the numerical value of the gain function, d(0, i) the distance between vertices 0 and i, where

l - the number of hyper-edges on the path between 0 and i

w j – weight of j-th hyperedgeJ.Vain “...Reactive Planning Testing...”Abo,

Feb 3, 201239

Constructing the gain guards:

gain function (2) For each symbol i denoting a leaf vertex in

TR(e,G) define a production rule(4)

Apply the production rules (3) and (4) starting from the root symbol 0 of TR(e,G) until all non-terminal symbols i are substituted with the terms that include only terminal symbols ti and d(0, i)


40

Example: Gain Functions


41

Example: Gain Guards


42


Example: Construction of the Tester Structure

s2

s3


i7/o7, t7=true

s1

e0:i0/o0t0=true

e1:i0/o1, t1=true

e2:i2/o2, t2=true

e3:i3/o3, t3=true

e4:i3/o4, t4=true

e5:i5/o5, t5=true

s1

s4

s2s5

s6

s3

s6 s9

s7

eo0:

o0/t0=true

eo1:

o1/t1=true

ec01:

[pc01(T)]

-/i0 eo2:

o2/t2=true

eo7:

o7/t7=true

ec2:

[pc2(T)]

-/i2ec

34:[pc

34(T)]-/i3 eo

3:o3/t3=true

eo6:

o6/t6=true

ec6:

[pc6(T)]

-/i6ec

7:[pc

7(T)]-/i7

ec5:

[pc5(T)]

-/i5

eo5:

o5/t5=true

eo4:

o4/t4=true

The gain guards guarantee that only the outgoing

edges with maximum gain are enabled in the given

state

43

Complexity of constructing and running the tester

The complexity of the synthesis of the reactive planning tester is determined by the complexity of constructing the gain functions.

For each gain function the cost of finding the TRe by breadth-first-search is O(|VD| + |ED|) [Cormen], where

|VD| = |ET| - number of transitions of MT |ED| - number of transition pairs of MT (is bounded by |ES|2)

For all controllable transitions of the MT the upper bound of the complexity of the computations of the gain functions is O(|ES|3).

At runtime each choice by the tester takes O(|ES|2) arithmetic operations to evaluate the gain functions


44

Performance of reactive planning


• Test Goal: All Transitions

• Test Goal: Selected Transition

45


Experiments: Case Study: Model of the IUT

Model of Feeder Box Controller power management (31 states, 73 transitions)

46

How to plan in large models?

Test goal: all transitions

100

1000

10000

100000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20horizon

step

s

Planning with horizon + anti-ant

Planning with horizon + random choice

10

100

1000

10000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20horizon

step

s

Planning with horizon + anti-ant

Planning with horizon + random choice

Test goal: single transition


48

Aver

age

leng

ths o

f the

test

sequ

ence

s • Adjustable planning horizon• Dependancy between horizon and test length

Horizon saturation point

Shaping RPT planning cones (i)


tri trj trk

Decision point

Alternative choices

In online-testing decision time must be strictly bounded!...

ge(tri,T )

...


Gain functions

49

Shaping RPT planning cones (ii)


tri trj trk

Decision point

Alternative choices

Decision time strictly bounded!

Prune the cone!...

ge(tri,T )

...


Gain functions

Horizon h

Different ways for defining h, and ‘visibility’ depending on depth of tree.

Fully

vi

sible

Parti

all

y visib

le

50

Average lengths of test sequences in the experiments


HorizonAll transitions test coverage Single transition test coverage

Planning with horizon Planning with horizonanti-ant random choice anti-ant random choice

0 18345 ± 5311 44595 ± 19550 2199 ± 991 4928 ± 44551 18417 ± 4003 19725 ± 7017 2156 ± 1154 6656 ± 54472 5120 ± 1678 4935 ± 1875 1276 ± 531 2516 ± 22633 4187 ± 978 3610 ± 2538 746 ± 503 1632 ± 17454 2504 ± 815 2077 ± 552 821 ± 421 1617 ± 14425 2261 ± 612 1276 ± 426 319 ± 233 618 ± 5126 2288 ± 491 1172 ± 387 182 ± 116 272 ± 1887 1374 ± 346 762 ± 177 139 ± 74 147 ± 1258 851 ± 304 548 ± 165 112 ± 75 171 ± 1149 701 ± 240 395 ± 86 72 ± 25 119 ± 129

10 406 ± 102 329 ± 57 73 ± 29 146 ± 19411 337 ± 72 311 ± 58 79 ± 30 86 ± 5912 323 ± 61 284 ± 38 41 ± 15 74 ± 5113 326 ± 64 298 ± 44 34 ± 8 48 ± 3114 335 ± 64 295 ± 40 34 ± 9 40 ± 2315 324 ± 59 295 ± 42 25 ± 4 26 ± 516 332 ± 51 291 ± 52 23 ± 2 24 ± 317 324 ± 59 284 ± 32 22 ± 2 21 ± 118 326 ± 66 307 ± 47 21 ± 1 21 ± 119 319 ± 55 287 ± 29 21 ± 1 21 ± 120 319 ± 68 305 ± 43 21 ± 1 21 ± 1

51

Average time spent for online planning of the next step


13

14

15

16

17

18

19

20

21

22

23

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

horizon

mse

c

All transitions

Single transition

52


53

How to generate test data?

Example: RPT synthesis for INRES protocol


54

Model of the SUT local component

Example: RPT synthesis for INRES protocol

Local SUT component model RPT model

Timeout!Off!

LowICONreq?DR?CC?

?

AK?


Online planning constraints for the RPT


56

How to derive data constraints? For all transitions t(si,.) of state si generate reduced

reachability tree RRTi s.t. transition t(si,.) is a root and the trap labeled transitions the

terminal nodes of the RRTi . Compute data constraint for each path j of RRTi

use wp-algorithm (starting from trap node) for pairs of neigbbour traps of j

unfold loops using gfp for termination for constructing the gain function of j record (when traversing j):

traps remaining on the path j and the lengths of inter-trap paths

construct the gain function for full path j using trap-to-trap distances on that path and the vector of trap variables.

Global data constraint for the path is a conjunction of data constraints pairwise traps of j


Summary RPT synthesis technique relies on offline static

analysis of the SUT model and test goals. Efficiency of planning:

Number of rules that have to be evaluated at each step is relatively small (i.e., = the number of outgoing transitions of a current state)

The execution of decision rules is significantly faster than looking through all potential alternatives at runtime.

Scalability supported by RRT pruning technique Provides test sequences that are lengthwise close to

optimal.


58

59

DTRON: Distributed RPT execution (http://dijkstra.cs.ttu.ee/~aivo/dtron/usage.html)

SPREAD


60

||

LTC communicating timed automata (TA)

TA model of local SUT

component (extended with online planning

constraints)

TA modelof the local test goal (property)

MsMg

||

TA model of local SUT

component (extended with online planning

constraints)

TA modelof the local test goal (property)

MsMg

“TCI update” signals

“Test coverage item (TCI) reached” signalTCI synchroLTC LTC


61

Tester components communicate over channels


62

DTRON: distibuted, heterogeneous components

SPREAD

Selenium TestCast

Z3


DTRON: dynamic test reconfiguration

63

SPREAD

Selenium TestCast Selenium

Model (Testerk ) Model (Tester1) Model (Test Plan Supervisor) Tester component updates

64

DTRON: Visualization of test runs

Questions?

Thank You!


65

online testing of nondeterministic systems with the reactive planning tester

Documents

reactive planning testing

behavioural testing

myers testing

process of planning

specific testing types

common testing activities

test control

test cases j