bait: adaptive test case execution in practice
DESCRIPTION
BAiT: Adaptive Test Case Execution in Practice. Jens R. Calamé Centrum voor Wiskunde en Informatica, Amsterdam. Agenda. Introduction The Toolset for BAiT An Exemplary Case: The ATM Structure of BAiT Conclusion. Introduction. BAiT: Behavior Adaptation in Testing - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/1.jpg)
BAiT: Adaptive Test Case Execution
in Practice
Jens R. CalaméCentrum voor Wiskunde en Informatica,
Amsterdam
![Page 2: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/2.jpg)
13-09-2007 DWFTT 2007, Enschede 2
Agenda
1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion
![Page 3: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/3.jpg)
13-09-2007 DWFTT 2007, Enschede 3
Introduction
• BAiT: Behavior Adaptation in Testing• Toolset to generate test cases and test oracles as well as
execute tests on nondeterministic systems with data
![Page 4: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/4.jpg)
13-09-2007 DWFTT 2007, Enschede 4
Agenda
1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion
![Page 5: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/5.jpg)
13-09-2007 DWFTT 2007, Enschede 5
BAiT: Test Case & Oracle Generation
• Test generation with TGV (enumerative approach)
• Chaotic data abstraction for input and output parameters of events
• Test oracle generation for Eclipse Prolog constraint solver in order to select test data
• Automatic generation of variable names for the test case
![Page 6: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/6.jpg)
13-09-2007 DWFTT 2007, Enschede 6
BAiT: Test Proxy Generation
• Test Proxy for Java-based SUTs
• Instantiation of test object within test proxy
• Procedure-basedbidirectional communication (publisher-subscriber pattern)
• Mapping of datatypes from specification language to target platform
![Page 7: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/7.jpg)
13-09-2007 DWFTT 2007, Enschede 7
BAiT: Test Execution
• Test execution of pre-calculated test trace (behavior and data)
• Adaptation to the IUT's reactions, rather than immediate assignment of verdict
• BAiT framework provides adapters for user-defined– trace selection and– test data selection
algorithms
![Page 8: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/8.jpg)
13-09-2007 DWFTT 2007, Enschede 8
Agenda
1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion
![Page 9: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/9.jpg)
13-09-2007 DWFTT 2007, Enschede 9
An Exemplary Case: The ATM
ATM specification (partially) ATM test purpose (completely)
![Page 10: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/10.jpg)
13-09-2007 DWFTT 2007, Enschede 10
An ATM Test Case
Photos: Wikipedia and ECB
!getPin(1234)
?pinCorrect
!getAmount(100)
!initBalance(1000)
?retFifty(2)
?retFifty(1)
?retTwenty(2)
?retTen(1)
?
!initPin(1234)
![Page 11: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/11.jpg)
13-09-2007 DWFTT 2007, Enschede 11
An ATM Test Case (2)
May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Pruning planned test trace.May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.RunTest findTrace
FINER: Examining: [init],initPin(Pin),getPin(PinUser),pinCorrect, initBalance(Bal),getAmount(Amt),[tau],retFifty(Fif),emitBankNotesMay 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl solve
FINEST: init(G1),initPin(G1,G2,lparam(nat(Pin))),[...][Pin in {[-Infinity .. Infinity]} => 1234]May 25, 2007 3:06:21 PM nl.cwi.sen2.bait.steps.TraceImpl solve
FINEST: init(G1),initPin(G1,G2,lparam(nat(1234))),[...][...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Trying: [init],initPin(1234),getPin(1234),pinCorrect, initBalance(1000),getAmount(100),[tau],retFifty(2),emitBankNotesMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
FINE: initPin(1234) -> OK[...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
FINE: retFifty(2) -> NOT OKMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Expected: retFifty(2), received: retFifty(1) -> Trace failed, trying alternative.
1. Precalculation of possible test trace
2. Iterative data selection
3. Stepwise execution of the trace…
…until the IUT deviates from the precalculated trace.
![Page 12: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/12.jpg)
13-09-2007 DWFTT 2007, Enschede 12
An ATM Test Case (3)May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Trace deviated; executed so far: [init],initPin(1234),getPin(1234),pinCorrect,initBalance(1000), getAmount(100),[tau],retFifty(1),retTwenty(2),retTen(1), emitBankNotesMay 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Pruning planned test trace.May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Adding retFifty(1) to trace stub.[...]May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Adding emitBankNotes to trace stub.May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Test finished; executed: [init],initPin(5),getPin(5), pinCorrect,initBalance(1000),getAmount(100),[tau],retFifty(1), retTwenty(2),retTen(1),emitBankNotesMay 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: The test case ended with verdict PASS.
5. Appending received system output to trace.
6. Solve test trace with pending system output;
assign verdict or calculate new trace.
4. Pruning not yet executed part of test trace.
![Page 13: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/13.jpg)
13-09-2007 DWFTT 2007, Enschede 13
Agenda
1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion
![Page 14: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/14.jpg)
13-09-2007 DWFTT 2007, Enschede 14
Structure of BAiT: The Tools
• BAiT: Behavior Adaptation in Testing• Toolset to generate test cases and test oracles as well as
execute tests on nondeterministic systems with data
![Page 15: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/15.jpg)
13-09-2007 DWFTT 2007, Enschede 15
Structure of BAiT: Execution Part
BAiTExecution Core
Behavior MappingConstraint Solver
Behavior MappingImpl. Under Test
Data MappingConstraint Solver
Data MappingImpl. Under Test
Impl.UnderTest
ConstraintSolver
Trace Selection
Data Selection
•Trace class: trace execution management•Step class: communication with IUT•Trace selection from complete test graph (IOLTS with all possible test cases for a given test purpose)
Test Oracle
Test Case(s)
•Variable class: Codec of data to constraint solver, data instantiator and IUT•Data instantiation algorithm(s)
![Page 16: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/16.jpg)
13-09-2007 DWFTT 2007, Enschede 16
Agenda
1. Introduction2. The Toolset for BAiT3. An Exemplary Case: The ATM4. Structure of BAiT5. Conclusion
![Page 17: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/17.jpg)
13-09-2007 DWFTT 2007, Enschede 17
Conclusion
• Behavior Adaptation in Testing helps finding more reliable verdicts in testing non-deterministic systems
• A prototypical framework for BAiT has been developed procedure-based setting with atomic data
• Further work:– Extension of framework
for other settings– Extension of framework
to structured datatypes– Case Study:
Mozilla FirefoxCSS subsystem
![Page 18: BAiT: Adaptive Test Case Execution in Practice](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d58550346895da71c83/html5/thumbnails/18.jpg)
13-09-2007 DWFTT 2007, Enschede 18
Further Information
Background information• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Sidorova, Natalia; Data
Abstraction and Constraint Solving for Conformance Testing (APSEC 2005).• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Towards Automatic Generation
of Parameterized Test Cases from Abstractions. Technical Report, 2006.
The toolset/framework itself• Calamé, Jens R.; Adaptive Test Case Execution in Practice. Technical Report, 2007.• Calamé, Jens R.; The Internals of Adaptive Test Case Execution. Technical Report,
to appear.
All downloadable from www.calame.de/works,published.bib,date,.html
Contact:Jens R. Calamé
CWI, [email protected]