event-b in a nutshell
DESCRIPTION
13th CREST Open Workshop 12th-13th of Ma y 2011, London. Search - Based Software Engineering for Model - Based Testing. Event-B in a Nutshell. Test Generation Approaches. SBT Challenges. Test Data Generation. Finite Model Learning. Conclusions. - PowerPoint PPT PresentationTRANSCRIPT
Event-B in a Nutshell
Test Data Generation
13th CREST Open Workshop12th-13th of May 2011, London
*) joint work with colleagues from DEPLOY project
Alin Stefanescu - University of Pitesti, Romania
SBT Challenges
Search-Based Software Engineering for Model-Based Testing
Test Generation Approaches
Finite Model Learning Conclusions
Page 2
[Event-B in a Nutshell]
Page 3
Event-B history
Jean-Raymond Abrial (1938- )
Inventor of the Z and B formal methods.
Z – developed in the 70s
B – developed in the 90s, successfully deployed in industry
Event-B – born with the 21st century
Evolution of B for system level specification
Developement supported by French and European projects:
FP6 RODIN and FP7 DEPLOY
Page 4
DEPLOY project (2008-2012) – funded by FP7
DEPLOY :: Industrial deployment of advanced system engineering methods for high productivity and dependability using formal methods
4 industrial partners■ Bosch, Siemens, SAP, SSF
3 industrial service providers■ Systerel, ClearSy, Cetic
7 academic partners■ Newcastle, Aabo, Düsseldorf, ETH Zurich, ■ Southampton, Pitesti, Bucharest
http://www.deploy‐project.eu
Page 5
Rodin platform for Event-B
Extension of Eclipse IDE (Java-based) Theorem proving as core technology Many other Rodin Plug-ins
ProB: animation, consistency and model-checking Animators (AnimB) Decomposition Modularisation Team-work Code generation UML-B etc.
Page 6
Event-B in a nutshell
■ State-transition model (like ASM, B, Z)■ set theory as mathematical language■ refinement as basic modeling approach
■ Contexts■ carrier sets (domains)■ constants■ axioms
■ Machines■ global variables■ invariants ■ events that update the variables
■ Events■ local parameters■ guards■ actions
ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})
items : Powerset(ITEMS)
Event-B model
Page 7
[Test Generation Approaches]
Page 8
Test generation based on Event-B
We investigate search-based testing (SBT) techniques for Event-B.
Model-based testing (MBT) is a newly introduced topic in DEPLOY
priority topic for industrial partners like SAP
challenges due to the sheer size of the state space of real-life scenarios
Model-Based Testing (MBT)
Page 9
Future MBT plugin in RODIN
MBT Plug-in
University of Pitesti and University of Dusseldorf
Extra test
information
{ Model-checking }
Event-B model
{ Search-based }
{ Constraint-based }
Test cases
MBT Users
Tool developers:
Page 10
Test generation from Event-B
Event-B modelGlobal variables: var1, var2, var3, ...
Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ...
ev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)........................ev3(..),ev7(..),...ev5(..)...............ev6(..),ev5(..),...ev8
1. Generate a set of tests(sequence of events with concrete param.)
2. Optimize test suite(according to some criteria) – if still needed
ev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)...ev3(..),ev7(..),...ev5(..)
SBT Opportunity!
Page 11
What is the explicit state space
Event-B modelGlobal variables: var1, var2, var3, ...
Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ...
State Space of the Event-B model
... ......
(3,4,{a,b},...)
ev3(5)
States given by the values of global variables Transitions labeled by events with concrete parameters
Abstract machine
Page 12
Test generation from Event-B
SBT Opportunity!
State Space of the Event-B model
... ......
(3,4,{a,b},...)
ev3(5)
Approach 1:Explore the state space using the ProB model checker
state space explosion mainly due to dataTry: guide the search
Page 13
Test generation from Event-B – part II
SBT Opportunities!
State Space of the Event-B model
... ......
(3,4,{a,b},...)
ev3
Approach 2:Explore state space ignoring the data(i.e. local parameters)
Problem 1: still large state spaceThen: construct approximations of state space up to depth K using finite automata Try 1: using machine learning and static analysisTry 2: using evolutionary algorithms?
Problem 2: infeasible sequencesTry 1: constraint solving for path feasibilityTry 2: test data generation with metaheuristics
Page 14
[Search-Based Testing Challenges]
Page 15
More details in: A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Towards Search-Based Testing for Event-B Models. To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.
Let’s take a look at some of specific challenges for Event-B...
Page 16
No explicit state space
Fact Event-B has no explicit states like the EFSMs no control state (as in EFSMs)
Problem Large (possibly infinite) state space testing coverage criteria must be defined only recent work addressing SBT for EFSMs
Possible ideas: coverage of all events (or a given subset of them) or coverage of all test
paths of length < K many other coverages possible, so industrial guidance is needed consider the class of Event-B models with a special state variable (see
industrial use cases from SAP, SSF, Bosch and UML-B models)
Page 17
Non-numerical types
Fact Event-B is based on set theory set relations, powersets, functions, set comprehensions, products, records, etc. Complex structured data (e.g. business domain)
Problem fitness functions in literature mostly defined for numerical types
Possible solutions design new fitness functions for set-based (non-numerical) types efficient encoding of mixed non-numerical/numerical test data
Page 18
Hierarchical models
Fact Event-B supports different types of hierarchy refinement from abstract to concrete levels model decomposition modularity most industrial models use some sort of hierarchy (due to size)
Problem no much previous work on SBT addressing hierarchical models
Possible ideas: adapt existing work on test selection for hierarchical state machines use the existing ProB model checker that can partially deal with hierarchy
Page 19
Non-determinism
Fact Event-B has different types of non-determinism :| or :∈ operators (e.g. x :∈ {item1, ..., item20}) non-deterministic choice of the event to be executed when several enabled non-deterministic choice of parameters (ANY construct) non-deterministic initialisation of variables satisfying the set of invariants
Problem no much previous work on SBT addressing non-deterministism
Possible ideas: devise fitness functions that improve the chance of choosing a given path in a
non-deterministic model (under certain assumptions) make the non-determinism visible (model instrumentation)
Page 20
[Test Data Generation]
Page 21
Generating test data for a path
Problem Given one path of events, provide the test data (event parameters) that
enables the execution of the path.
Approach genetic algorithms encoding of sets into binary genes mixed choromosomes (numerical and binary genes)
More details in:I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Test Data Generation for Event-B Models using Genetic Algorithms. In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.
Page 22
Test data generation with genetic algorithms
Simulator (ProB)
Fitness evaluation
ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})
items : Powerset(ITEMS)
Encoding of variables
Mutation
Selection
Crossover
Event-B modelitems …0 1 1 0 1 0
Step 1• Purchase
Step 2• ValidateLarge
Step 3• CheckSpecial
0 1 1 0
1 0 1 0 1 0 0 1
0 1 1 0
0 1 1 1
1 0 10
Fitness functions
“Chromosome”
End?
Page 23
Fitness functions for one path
fitness := approach level + normalized branch level
Classical Tracey’s objective for numerical types New objective functions for set types
Page 24
Examples from the benchmark
Page 25
Statistical results
Statistical comparison of
Genetic Algorithms (GA) and
Random Testing (RT)
on 18 paths covering 5 Event-B models
using statistical test like t-test and U-test
And (of course) the winner is:
® GA performs significantly better than RT on most paths
Note: We are currently evaluating constraint-solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).
Page 26
[Finite Model Learning]
Page 27
Generating finite models from Event-B
Problem There is no explicit state space of an Event-B model
Approach Finite automata learning (adapted L* algorithm ) Aproximation through cover automata K-bound on the length of executions Use finite automata for conformance test generation
More details in:F. Ipate, I. Dinca, A. Stefanescu:Model Learning and Test Generation for Event-B using Cover Automata. Submitted to SEFM’11.
Page 28
First experiments
Preliminary approach
Approximation through cover automata
for bound l Incremental -> fits very well with model
refinements Minimal finite automata
Sometimes difficult to find counterexamples (to the approximation)
Scales for medium size models:
... ......
ev3
Bound l
SBT Opportunity?!
Page 29
[Conclusions]
Page 30
Opportunities for Search-Based Techniques
To wrap-up opportunities:
■ Test suite minimisation with multi-objective optimisation
■ Test data generation for one path with search-based algorithms
■ Construct finite models with evolutionary algorithms
■ Combine ProB model-checker with meta-heuristics
■ Combine ProB constraint-solver with meta-heuristics
■ Experiment with different search algorithms (PSO, ACO, SA,...)
To be answered until end of project (April 2012):
Which of the above work good in practice?