i nstitute for software technology graz university of technology austria sommersemester 2011
DESCRIPTION
Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer. I nstitute for Software Technology Graz University of Technology Austria Sommersemester 2011. Outline. What is SpecExplorer ? Overview of SpecExplorer Controllable vs Observable methods Model program example - PowerPoint PPT PresentationTRANSCRIPT
Institut für Softwaretechnologie
1
Johannes Anderwald Graz, SS 2011 Spec Explorer
Institute for Software TechnologyGraz University of Technology
Austria
Sommersemester 2011
Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer
Institut für Softwaretechnologie
2
Johannes Anderwald Graz, SS 2011 Spec Explorer
Outline
• What is SpecExplorer ?• Overview of SpecExplorer• Controllable vs Observable methods• Model program example• Scenario control• Offline vs Online Testing• Case Study
Institut für Softwaretechnologie
3
Johannes Anderwald Graz, SS 2011 Spec Explorer
What is SpecExplorer?• SpecExplorer is a tool for testing reactive, object
orientated software systems
• SpecExplorer is a tool to develop, explore and validate models
• SpecExplorer generates test cases from models
• SpecExplorer can execute tests against an implementation
Institut für Softwaretechnologie
4
Johannes Anderwald Graz, SS 2011 Spec Explorer
Overview of SpecExplorer(1)
• System behavior is described by a model language• Model language is one of those:
– Spec#
– AsmL
– C#
– Visual Basic
• Model program defines state variables and update rules of an abstract state machine
• The states of the machine are first order structures
Institut für Softwaretechnologie
5
Johannes Anderwald Graz, SS 2011 Spec Explorer
Overview of SpecExplorer(2)
SpecExplorer explores machine's states
• Result is a finite graph– Graph can be viewed in SpecExplorer
– Graph is subset of states and transitions
• SpecExplorer generates test cases for the explored behavior
• Test cases are run against system under test(SUT)• SpecExplorer displays successful / failing tests
Institut für Softwaretechnologie
7
Johannes Anderwald Graz, SS 2011 Spec Explorer
Controllable vs Observable
Reactive systems are nondeterministic
SpecExplorer distinguishes input / output
2 Types available Controllable actions
Observable actions
Institut für Softwaretechnologie
8
Johannes Anderwald Graz, SS 2011 Spec Explorer
Model exampleclass Client
{
bool entered;
Map<Client,Seq<string>> unreceivedMsgs;
[Action] Client() {
this.unreceivedMsgs = Map;
foreach (Client c in enumof(Client), c != this){
c.unreceivedMsgs[this] = Seq{}; this.unreceivedMsgs[c] = Seq{}; }
entered = false;
}
[Action] void Enter()
requires !entered; {
entered = true;
}
[Action] void Send(string message)
requires entered; {
foreach (Client c in enumof(Client), c != this, c.entered)
c.unreceivedMsgs[this] += Seq{message};
}
[Action(Kind=ActionAttributeKind.Observable)]
void Receive(Client sender, string message)
requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; {
unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;
}
}
Institut für Softwaretechnologie
10
Johannes Anderwald Graz, SS 2011 Spec Explorer
Scenario Control
Model program is unwound into a model automation
Contains large or infinite number of transitions
Solution?– Apply filter techniques
– Maps model automation into new automation
Institut für Softwaretechnologie
11
Johannes Anderwald Graz, SS 2011 Spec Explorer
Scenario Control(1)
- Parameter selection
- Method restriction
- State filtering
- State grouping
- Directed search
Institut für Softwaretechnologie
12
Johannes Anderwald Graz, SS 2011 Spec Explorer
Parameter selection
Parameter selection limits exploration to a finite set of parameters for the action methods
– Add initial states
– Add states which are reachable from initial states
4 Levels of parameter control- Defaults, i.e boolean = false/true
- Per type
- Per parameter
- Per method
Institut für Softwaretechnologie
13
Johannes Anderwald Graz, SS 2011 Spec Explorer
Method restriction
An action is enabled in the state s if the precondition of m is satisified
Method restriction is applied by strengthening the precondition
Note: Observable actions should be avoided- Why?
Institut für Softwaretechnologie
14
Johannes Anderwald Graz, SS 2011 Spec Explorer
State filtering
State based filtering prunes states that fail to satisfy a given state based predicate
Example: forall{ c in enumof(Client), s in c.unreceivedMsgs.Keys,
m1 in c.unreceivedMsgs[s],
m2 in c.unreceivedMsgs[s]
m1 != m2
}
Institut für Softwaretechnologie
15
Johannes Anderwald Graz, SS 2011 Spec Explorer
Directed Search
SpecExplorer uses non-deterministic search to explore states and transitions
Transitions and states are explored untill user provided bound conditions are met
– Initial states are captured
– Transition exploration choices are based on state weights and action weights
Institut für Softwaretechnologie
16
Johannes Anderwald Graz, SS 2011 Spec Explorer
State grouping
State grouping is used to select representative states of an equivalence class
Used to prune exploration to distinct cases of interest for testing
Example: Bag { c in enumof(Client);
<c.entered,
Bag{<s,m> in c.unreceivedMsgs; m}
>
}
Institut für Softwaretechnologie
17
Johannes Anderwald Graz, SS 2011 Spec Explorer
Offline vs Online Testing
Offline test generation– Tests are generated in advance from a model
– State space coverage / accepted state / random walks
– Value iteration algorithm for negative Markov decision problems
Online test generation– Tests are generated as the model program is
unfolded
– OTF algorithm
– Controllable actions are randomly choosen by OTF
Institut für Softwaretechnologie
19
Johannes Anderwald Graz, SS 2011 Spec Explorer
Case Study(1)
Microsoft Open Protocol Initiative 222 protocols
22.847 pages of specifications
66.692 person days (270+ person years) 350 test engineers in India / China
36.875 requirements 69 % tested using MBT 31 % tested using traditional test automation
Institut für Softwaretechnologie
21
Johannes Anderwald Graz, SS 2011 Spec Explorer
Conclusion
SpecExplorer is a tool to develop and test models
SpecExplorer provides powerful methods for visualizing models
Open problems: Test coverage / suffiency metrics Failure analysis(shorter paths?) Continuation after failure
Institut für Softwaretechnologie
22
Johannes Anderwald Graz, SS 2011 Spec Explorer
Vielen Dank für die Aufmerksamkeit
Institut für Softwaretechnologie
23
Johannes Anderwald Graz, SS 2011 Spec Explorer
References
[1] Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorerhttp://tinyurl.com/3j4ovja
[2] Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testinghttp://tinyurl.com/3nfauuv
[3] Optimal Strategies for Testing Nondeterministic Systemshttp://tinyurl.com/3st57br