robust test generation and coverage for hybrid systems€¦ · 1 robust test generation and...
TRANSCRIPT
1
Robust Test Generation and Coverage for Hybrid Systems
A. Agung Julius, Georgios E. Fainekos, Madhukar Anand, Insup Lee and George J. Pappas
Deptartments of ESE and CISUniversity of Pennsylvania
HSCC April 2007, Pisa, Italy
HSCC April 2007, Pisa, Italy 2
Hybrid System Design Cycle
Specification/RequirementsSpecification/Requirements
Mathematical Analysis
Mathematical Analysis
Model DesignModel Design
Testing/VerificationTesting/
Verification
SystemImplementation
SystemImplementation
TestingTesting
HSCC April 2007, Pisa, Italy 3
Testing VS Verification: continuous time
TestingTan, Kim, Sokolsky, Lee IRI’04
TorX [Bohnenkamp.et al]Maler, Nickovic FORMATS’04Briones, Brinksma FATES’04
van Osch FATES\RV’06Krichen, Tripakis SPIN’04
UPPAAL-TRON [Mikucionis. et al]
Systematic TestingBadban, Franzle, Peleska, Teige SOQUA’06
Cheng, Kumar WAFR’06Kim, Esposito ACC’05
Krichen, Tripakis FORMATS’04Kapinski, Krogh, Maler, Stursberg HSCC’03
Branicky, Curtiss, Levine, Morgan, Yale Workshop’05Bhatia, Frazzoli, HSCC’04
HSCC April 2007, Pisa, Italy 4
Testing of hybrid systems
UnsafeInit
L1
L2
L3
In this paper, testing for hybrid automata:1) How robust is a test trajectory?2) How can we provide a confidence level for the system correctness?
HSCC April 2007, Pisa, Italy 5
Defining the robustness of a simulation trajectory
HSCC April 2007, Pisa, Italy 6
How robust is a test trajectory?
y(t)
Sg(x(t))y(t)f(x(t))(t)x
==&
pn y I, x(0),x ℜ∈∈ℜ∈
Consider the dynamical system:
Unsafe
y(t)
)),((inf0
Unsafetydt≥
=ε
Fainekos, Girard, Pappas: Temporal Logic Verification Using Simulation, FORMATS 2006
HSCC April 2007, Pisa, Italy 7
Bisimulation functions
x1
x2
X
time
( ) ( ){ }εφεφ ≤∈= 2121 ,:, xxXxxB
A bisimulation function is nonincreasing along any two trajectories of the system.
HSCC April 2007, Pisa, Italy 8
Bisimulation functions
The function φ : X x X ö + is a bisimulation function if the following properties hold
for all x1, x2 ∈ X it is ||g(x1)-g(x2))||2 ≤ φ(x1,x2)for all x1, x2 ∈ X it is
0)(),()(),(2
2
211
1
21 ≤∂
∂+
∂∂ xf
xxxxf
xxx φφ
A. Girard & G.J. Pappas, Approximation Metrics for Discrete and Continuous Systems, IEEE TAC, to appear.
HSCC April 2007, Pisa, Italy 9
Systems with affine dynamics
HSCC April 2007, Pisa, Italy 10
Lyapunov equation
Lyapunov equation ... always has a solution for stable A.
HSCC April 2007, Pisa, Italy 11
Invariance property
x1
Xx2
x1
x1x2
x2
HSCC April 2007, Pisa, Italy 12
How robust is a test trajectory?
y(t)
Sg(x(t))y(t)f(x(t))(t)x
==&
pn y I, x(0),x ℜ∈∈ℜ∈
Consider the dynamical system:
Unsafe
y(t)
)),((infinf0
ztyUnsafezt
φε∈≥
=
When φ is a metric …
HSCC April 2007, Pisa, Italy 13
How robust is a hybrid test trajectory?
A hybrid automaton is a tuple H = (X, L, E, f, g, U, Inv, Init, G, R, Unsafe)
where X is the continuous state spaceL is the set of control locations E Œ L × L is the set of control switchesInv : L Ø P(X) assigns an invariant set to each locationOut : L × Z Ø V is the control input for S’Init Œ X0 × L is the set of initial conditions G : E Ø P(bd(Inv(l))) is the guard condition that enables transition e=(l,l’)œER : E Ø Inv(l’) is the reset map for the transition e=(l,l’)œEUnsafe Œ X0 × L is the unsafe regionf, g
y(t)
S(x(t))gy(t)(x(t))f(t)x
i
i
==&
pn y I, x(0),x ℜ∈∈ℜ∈
HSCC April 2007, Pisa, Italy 14
How robust is a hybrid test trajectory?
UnsafeInit
L1
L2Note: invariance sets can be different in each location.
HSCC April 2007, Pisa, Italy 16
One step of the algorithm
HSCC April 2007, Pisa, Italy 17
What about neighboring trajectories?
UnsafeInit
L1
L2
Bisimulation metric takes care of that …
HSCC April 2007, Pisa, Italy 18
Robustness implies same qualitative behavior
UnsafeInit
L1
L2
L3
HSCC April 2007, Pisa, Italy 19
We have timing guarantees, too.
g1g2
act
x0
(x ,d )min0Bφ
^g2
ξ(τ,x )0
ξ (τ+ε,x )0
ξ (τ−ε,x )0
Unsafeact
dunsafe
dout
HSCC April 2007, Pisa, Italy 20
Main result: Loop Invariance
Thus, a guarantee on the qualitative behavior and timing.
HSCC April 2007, Pisa, Italy 21
A Testing algorithm
for Hybrid Automata
HSCC April 2007, Pisa, Italy 22
Covering of the parameter space
It is impossible to cover an uncountable testing parameter space with points.
Initial Conditions
HSCC April 2007, Pisa, Italy 23
Covering with robust tests
Each test represents a (nonzero measure) neighborhood of testing parameters.
Parameters that lead to tests with the same qualitative properties are grouped together.
Initial Conditions
HSCC April 2007, Pisa, Italy 24
Each test represents a (nonzero measure) neighborhood of testing parameters.
Parameters that lead to tests with the same qualitative properties are grouped together.
Finite covering is possible!
Only if the system is robust.
Covering with robust tests
HSCC April 2007, Pisa, Italy 25
Overview of algorithm
Includesinitial
conditions
Max simulation time, max number of tests,etc
Pick a point in theparameter space
Pick a point in theparameter space
SimulatetrajectorySimulatetrajectory
ComputeRobustnessCompute
Robustness
Update parameterspace
Update parameterspace
Remove computed ellipsoid from initial conditions
Stoppingcriterion?
No
OutputResultsOutputResults
Yes
Safe?
Yes
HybridAutomaton
HybridAutomaton
InputParameters
InputParameters
SystemUnsafe
SystemUnsafe
No
Includes the computation of bisimulation functions
HSCC April 2007, Pisa, Italy 26
Coverage strategies
Randomized strategy: easy to implement, almost impossible to get 100% coverage.
Grid based strategy: easy to implement, suffers from curse of dimensionality.
Minimal dispersal: based on partitioning the parameter space with weighted Voronoipartitions.
HSCC April 2007, Pisa, Italy 27
Computing distances
linear projections(least squares when we consider the
location dynamics)
quadratic programming Unsafe
semidefiniteprogramming
HSCC April 2007, Pisa, Italy 28
Some Examples
HSCC April 2007, Pisa, Italy 29
Navigation benchmark
0 1 2 30
1
2
3
x1
x 2
Unsafe 2 4
2 3 4
2 2 Goal
HSCC April 2007, Pisa, Italy 30
Navigation benchmark
0 1 2 30
1
2
3
x1
x 2Unsafe 2 4
2 3 4
2 2 Goal
HSCC April 2007, Pisa, Italy 31
0 1 2 30
1
2
3
x1
x 2Unsafe 2 4
2 3 4
2 2 Goal
Navigation benchmark
HSCC April 2007, Pisa, Italy 32
Navigation benchmark 1
0 1 2 30
1
2
3
x1
x 2
Unsafe 2 4
2 3 4
2 2 Goal
HSCC April 2007, Pisa, Italy 33
Navigation benchmark 1
With 25 runs, we cover >48% of the initial set.
Notice that there is a clear divide in the initial set, due to different transitions.
0 1 2 30
0.5
1
1.5
2
2.5
3
(a)
x 2
x1
1 1.5 2
0.8
1
1.2
1.4
1.6
1.8
2
(b)
x 2
x1
HSCC April 2007, Pisa, Italy 34
Benchmark problem 2
Verified to be safe with CHARON
0 1 2 30
1
2
3
x1
x 2
Unsafe 2 4
2 3 4
2 2 Goal
HSCC April 2007, Pisa, Italy 35
Benchmark problem 2
Safety verified after 9 tests!(All traces have the same qualitative behavior and the system is robust wrt to the unsafe set. Termination guaranteed similar to Girard & Pappas HSCC’06, Fainekos et al FORMATS 2006)
Numerically, we compute a coverage estimate of 72%.
2 2.2 2.4 2.6 2.8 3
1
1.2
1.4
1.6
1.8
2
(a)
x 2
x1
2 2.2 2.4 2.6 2.8 3
1
1.2
1.4
1.6
1.8
2
(b)
x 2
x1
HSCC April 2007, Pisa, Italy 36
Navigation benchmark 3
0 1 2 30
1
2
3
x1
x 2
2 3 6
3 3 Goal
2 2 Unsafe
HSCC April 2007, Pisa, Italy 37
Navigation benchmark 3
• Test generation using Voronoi with weights
• We verified unsafetywith 10 tests.
-0.5 0 0.5 1 1.5 2 2.5 3 3.5-0.5
0
0.5
1
1.5
2
2.5
3
3.5
x1
x 2
HSCC April 2007, Pisa, Italy 38
Conclusions and Discussion
HSCC April 2007, Pisa, Italy 39
Conclusions & Discussion
We have introduced : a notion of robustness for test trajectories of hybrid systemsAn algorithm that computes confidence levels for hybrid systems
A toolbox that helps the exploration of a hybrid system
Early stages of HS designThe algorithm is automatic for hybrid systems with affine dynamicsThe framework can be effectively parallelized
HSCC April 2007, Pisa, Italy 40
Future Extensions
Temporal logic testing of hybrid systemsFainekos, Girard, Pappas: Temporal Logic Verification Using Simulation, in FORMATS 2006
Probabilistic testingJulius: Approximate abstraction of stochastic hybrid automata, in HSCC 2006
Nonlinear systemsGirard, Pappas: Approximate bisimulations for nonlinear dynamical systems, in CDC 2005
Hybrid Systems with bounded input (noise)Girard, Pappas: Verification using simulation, in HSCC 2006
HSCC April 2007, Pisa, Italy 41
Thank You! Questions?