![Page 1: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/1.jpg)
Automated modelAutomated model--based test based test generation and executiongeneration and execution
December 2002
Alan Hartman
![Page 2: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/2.jpg)
AgendaAgenda
• Project Overview• Motivation• Methodology• Tools• Experiments• Current Status
![Page 3: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/3.jpg)
Project OverviewProject Overview
• Automated model-based test Generation and Execution for DIStributed systems
• Methodology and tools for model-based testing• Open interfaces• Mixture of academic and industrial partners• Three phase timetable of experiment and
development• November 2001-2003
![Page 4: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/4.jpg)
PartnersPartners
• IBM Haifa Research Lab• Oxford University• VERIMAG/IRISA• Imbus• France Telecom• IBM UK• Intrasoft International
![Page 5: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/5.jpg)
MotivationMotivation
• Testing is 40-70% of development cost
• Early bugs cost less than late bugs
• Famous disastrous bugs:– Therac-5 radiation therapy
controller– Ariane 5 spaceship– Pentium floating point bug
Source: Applied Software Measurement, Capers Jones,1996
%Defects
this phase
Coding UnitTest
FunctTest
FieldTest
PostRelease
% Defects found inthis phase
Perc
enta
ge o
f Bug
s$ Cost to repair defect in this phase
$250
$14,000
Introduced
(Apar $15-40,000)
Introduced
in
in $25
$1000
$130
in85%
![Page 6: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/6.jpg)
AGEDIS MethodologyAGEDIS Methodology
Specs FSM Model
AbstractTestSuite
ExecutableTestSuite
Model
Trace
Generate Translate
Run
DesignBugs
InterfaceBugs
CodeBugs
![Page 7: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/7.jpg)
BenefitsBenefits• Starting from specification
– Involves testers early in the development process – Teams testers with developers– Forces testability into product design
• Building behavioural model and test interface– Finds design and specification bugs - before code exists– The model is the test plan - and is easily maintained
• Automated test suite generation– Coverage is guaranteed - increases testing thoroughness– Matches coverage goals to testing budget– Zero test suite maintenance costs
• Automated test suite execution– Finds code and interface bugs– Includes a framework for the testing of distributed applications– Reduces test execution costs
![Page 8: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/8.jpg)
AGEDIS ArchitectureAGEDIS Architecture
Use
r In
terf
ace
Model
Generation Directives
Execution Directives
Compiler Intermediate
Format Encoding
Generator
Analyzer
Visualizer
Execution
Abstract Test Suite
SuiteExecution
Trace
![Page 9: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/9.jpg)
AGEDIS Process FlowAGEDIS Process FlowObjecteeringUML Modeler
XML IF ATS SET
AML2IFCompiler
Model Simulator
Model Traversal
Execution EngineAML Profile
UUT
FeedbackTools
![Page 10: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/10.jpg)
ModelingModeling EnvironmentEnvironment
• The AGEDIS Modeling Language:– UML Class diagrams - structure– UML Object diagrams - snapshots– UML State diagrams – behaviour & test purposes
• Annotated with an action language – IF• Currently use Objecteering UML modeling tool• Tool profile to convert to XML• General purpose XML to IF compiler
![Page 11: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/11.jpg)
ModelingModeling EnvironmentEnvironment
![Page 12: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/12.jpg)
AGEDIS ArchitectureAGEDIS Architecture
Use
r In
terf
ace
Model
Generation Directives
Execution Directives
Compiler Intermediate
Format Encoding
Generator
Analyzer
Visualizer
Execution
Abstract Test Suite
SuiteExecution
Trace
![Page 13: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/13.jpg)
Test Generator BackgroundTest Generator Background
• Based on GOTCHA and TGV• GOTCHA
– uses Murphi specification language – explicit traversal of state space– extensive coverage criteria
• TGV– language independent simulator– focus on distributed applications– explicit test purposes as sequences of interactions
![Page 14: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/14.jpg)
Test Generator StructureTest Generator Structure
determinisation
IF
synchronous product
AUT
TD
IF/AML Mapping
compilation compilation & simulation
controlability
Aut2ATS ATS
![Page 15: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/15.jpg)
The ATS FormatThe ATS Format
• XML format – contains all the information needed for execution
– Can be produced manually or automatically
• Model description
– classes : constants, types, control & observable signatures
(a special class is defined for the tester)
– object identities
• Test Suite = set of test cases
– « interaction graphs » between the tester and the SUT
– associated verdicts (Pass, Fail, Inconclusive)
![Page 16: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/16.jpg)
AGEDIS ArchitectureAGEDIS Architecture
Use
r In
terf
ace
Model
Generation Directives
Execution Directives
Compiler Intermediate
Format Encoding
Generator
Analyzer
Visualizer
Execution
Abstract Test Suite
SuiteExecution
Trace
![Page 17: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/17.jpg)
Test Execution EngineTest Execution Engine
– Input: ATS and Test Execution Directives– Output: Suite Execution Trace (XML)– Multiple platform (e.g. Linux, Windows) support– Multiple external interface (Java,C++,C) support– Test distribution + centralized logging– Interactive test case execution– GUIs to view execution progress, inputs, and outputs
![Page 18: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/18.jpg)
Execution Engine ArchitectureExecution Engine Architecture
N e t w o r k
Test SuiteDriver
I/O
HostManager
ProcessManager
1 1 on each host 1..N on each host. ..
Object
Object
![Page 19: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/19.jpg)
Experiments with the MethodologyExperiments with the Methodology
• File System– Duplicate testing with and without tools– 20% less resources, same bug detection quality
• Non-standard GUI Application– Failure
• Java garbage collector• Automated GUI testing
![Page 20: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/20.jpg)
Current StatusCurrent Status
• It works!• Objecteering profile gives easy access to AML• Compiler is well-structured for future
developments• Test Generator creates multiple test paths with
fewer inconclusive test cases• Execution Engine demonstrates the benefits of
abstract models for versatile testing of distributed SW
![Page 21: Automated model-based test generation and execution...Project Overview • Automated model-based test Generation and Execution for DIStributed systems • Methodology and tools for](https://reader033.vdocuments.us/reader033/viewer/2022060816/609522648f9aab08616fd634/html5/thumbnails/21.jpg)
Future PlansFuture Plans
• MQSeries Experiment• Productivity tools• Integrated working environment• Finished Tools• France Telecom & Intrasoft Experiments• Exploitation Activity