state 4lachi'(tes realnlc-bnc.ca/obj/s4/f2/dsk3/ftp04/mq63309.pdf · tools to perform...
TRANSCRIPT
SIUTATION TESTIXG OF HERiRCHIC.4L FJNTE STATE 4LACHI'(TES JS THE RATIO^.^ ROSE REAL TDIE TOOLSET
A thesis submitted to the Department of Compiiting k Information Science in conformity mith the requirements
for the degree of Master of Science
Queen's Cniversity
Iiingston. Ontario Canada
September '200 1
Copyright @ Bradley Michael Graham. 3001
Bibliothèque nationale du Canada
Acquisitions and Acquisitians et Bibliographie Services senrices bibliographiques
395 Wellington Street 395, rue WdKingtOn Otfawa ON KlAOM OaawaON KlAON4 Canada Canada
The author has granted a non- exciusive licence allowing îhe National Libraq of Canada to reproduce, ioa., distriiute or seil copies of this thesis in miçrofom, paper or electronic formal.
The author retains ownership of the copyright in this thesis. Neitha the thesis nor substantiai extracts fiom it may be printed or otherwise reproduced without the author's permission.
L'auteur a accordé une licence non exclusive permettant a la Bibliothèque nationale du Canada de reproduire, prêter, distribuer ou vendre des copies de cette thèse sous la forme de microfiche/iihn, de reproduction sur papier ou sur format électronique.
L'auteur conserve la propriété du droit d'auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprim6s ou autrement reproduits sans son autorisation.
Abstract
Reactim systems are a class of systems characterized by the maintenance of a continuous and timely interaction with their environment. Testing reactive systems cliffers from testing sequential systems: instead of checking simple p a i s of inptits and outputs. seqiiences of inputs and outptits have tn btx checked. Testing techniques cannot be basecl strictly on a functional view of the .ioftware rather they musc be baseci on sorne history preserving view such as one of the finite state machine formalisms.
Mirtacion testing is a heuristic technique originally clevelopetl to encourage the construction of tiigh qiiality test mites for seqtiential software. [t offers a potentially highly automatable. hiit also c.oniptitationally rspensive way of creating thorough test suites. The success of mutation testing is largely dependent on a set of miitation operators definecl for the language clescribing the software.
This thesis ewliiates the feasibility effectivcness of perfoming mutation testing of rextive systenis rnorlelerl hy active nbjccts tvhose behavioiirs are definecl hy hierarchicat finite srntp ma- chines (HFSM) as implenientecl in the Rational Rose Real Time toolsct (RoseRT). This is nchierecl by proposing a set of mutation operators. building tools to perforni mutation testina. pcrformirig esperiments on a sniall set of HFSMs. and analyzing the results and obserntions nintle rliiririg the eperiments.
This thesis presents preliniinary eviclence indicating that mutation testing of RoseRT HFSSIs is botii feasible (n-ith tool support) and can resiilt in the creation of faiilt revealing test secluences-
Acknowledgement s
I would like to thank my thesis supenisor Dr-Terry Shepartl for his Friendship. ideas. giii<lnnc~. and iissistance in cotripleting this thesis.
t tvoultl Iih CO rleclicatc th& thesis to my children Benjamin ancl Selena nncl to rny wife Iinchleeri without wtiose love. support, and encouragement it n-oiiId not have heen possible.
Contents
1 Introduction 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Motivation L
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Problem 1 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Objectives -
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Contribtitions 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Synopsis 4
3 Software Testing Background 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Prcliniiriiiries 5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Test Ornclcs 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Test .4 cleqtiacy 6
2.4 Classification of Test Data .i rfequacy Criterion . . . . . . . . . . . . . . . . . . . . . 8 2.4.1 L'sage Baseti: Test cases corne From the espectetl usage profile . . . . . . . . . S
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Black Bos Techniques S . . . . . . . . . . . . . . . 2.4.3 M i t e Box: Test caes corne from the source c:o<le 10
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 EvaluatingCriterion 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . '1.5.1 Summary LS
A Detailed Look At Mutation Analysis 16 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Mutation Testing 1G . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Nutation Theory 17
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Applying Operators 18 3.4 Mutation Operators: Siruulating Progrmimer 1Iistakes . . . . . . . . . . . . . . . . . 20 3.5 \Veaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Selective Mutation 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 A Xotion of Fauit Size '1-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Weak Mutation 26
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Summq- 27
4 Testing State Machine Based Software 39 4.1 Finite State Machines (FSBI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Protocol Testing 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 EFSM nntl HFSM 3'2
4.4 Testing Estendeci hierarchicai FSSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.5 Statecfiart Slutation Operacors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.6 Surnrnq . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5 The RoseRT Toolset 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Overview 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Structure .. . 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Behaviour 43
3.4 Transitions and Triggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 - - 3.2 Active Object issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.5.1 Pnonty inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-32 Spontaneous Events 46 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6 Mutation Testing of RoseRT Models 48 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 hliitation Operators For RoseRT 48
6.2 h Proposed RoseRT CSM Slucation Operator Set . . . . . . . . . . . . . . . . . . . -50 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 FSSI Operators Xi
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Coverage Operacors 51 6.2.3 C'SLL-RT/RoseRT Featrires hased Operator Set . . . . . . . . . . . . . . . . . 52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Sumrnary 53
7 A Mutation Testing Systern for RoseRT [MTS-RTJ Using TXL 5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. 1 Orenîew of the hplementation -M
. . . . - - . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Performing Mutations .... 56 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.21 TSL ar
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 . 2 TSL Grammnr 58 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 T-U Transformations 59
. . . . . . . . . . . . . . . . . . . . . . 7.3 Caps~~le File Format .... . . . . . . . . . . . 60 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Basic Slutatiori Process 61 - . . . . . . . . . . . . . . . . . . . . . . . r . ù Esample Uutation Operator Program . T-IT 63
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 S u r t i m q 64
8 Creating and Running Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Testing Prowss and Automation
. . . . . . . . . . . . . . . * . . . . . . . . . . . . . . . . 5.2 Testing RoseRT Models ... . . . . . . . . . . . . . . . . . . 8.3 Thc RoseRT Generic Test Hrirness (RT-TcstHarness)
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 The RT-TestCaseSlanager Progam 5.3 RT-SescRunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Data Xnalysis: RT-SttperDiff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y.7 Hunting for 11i1t.mts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . $.Y Finciing Faiilts in the Original . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Surnrriary
9 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Capsules Siimmarq-
. . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Expianation of Table Elements
. . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 E-xperiment 1: CeU Phone Component . . . . . . . . . . . . . . . . . . . . . 9.3 Luperitrient 2: CeU from a Garne of Life Sfodel
. . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 kperiment 3: TratFic Light Controiier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Esperirnent 4: Cririse Control
. . . . . . . . . . . . . . . . . . . . . . . . 9.6 Expecimcnt a: Radar Control Cnit (RCG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Ek~eriInent 6: Pop Machine
. . . . . . . . . . . . . . . . . . . . . . . . . 9.8 E.qeriment 7: Electronic \Varfare Jammcr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9 Other C a p d e s Esatnined
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9.1 Radar module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10 Ind idu id Operacor Discussion
. . . . . . . . . . . . . . . . . . . . . . 9.10.1 Covenge Operators, CV T. CSE . CSS 9.10.2 Action DeIete Operators . .WT . .AD E' ADX . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.3 TRD: Transition DeIete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.4 SDL: State DeLete
9.10.5 internai Transition Hiitory Operators ES (Internai To e l temal) . STI (es - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ternd To internai)
. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.6 TAT Transition Alter Target
. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.7 SCP: exchange Choice Point
. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.8 TCP: Transition Choice Point
. . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.9 TGT: Trmit ion Guard True
9.LO.lOTCk Transition Change Action . . . . . . . . . . . . . . . . . . . . . . . . . 93 . . . . . . . . . . . . . . . . . . . . . . . 9.10.1 1 TTG: Transicior: TriGger ruchangeci 94
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 .IL Operator Siimmary 95 9.12 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.13 Initial Test Sequences 9s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1-1 Killing mutants 94
9.15 Equivalent SIutants . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . . . . 100 9.16 Finding Fades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LUC) 9.17 Strong vs Firm SIiitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LOI 9-18 Oracling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 9-19 Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
9 . 19.L Xiimber of SItitnnts . . . . . . . . . . . . . . . . . . . . . . . . . ... . . .. 103 9.192 CPC Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 9.19.3 Friiilts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4
9.2U PreambIcs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 9.21 Testing yoiir owm code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L05 922 Siimniary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10 Conclusions 107 10.1 Siunmary of Resiilts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 IO.2Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LOI1 10.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 10.4 Concliitling Remnrks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
10.4.1 Design .\ Iiitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LL2 10.42 Operator Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LI2
A Testing Glossary 117
B Example Capsule Text File 118 3.1 Bandling C++ Within Capsule Fies . . . . . . . . . . . . . . . . . . . . . . . . . . . LlS B.2 Capsule File for Cruise Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
C TXL Grammar Files Used For Parsing RoseRT Capsule Files L28 C.l Grammar file: SIarkC'pCapsuIe-grm . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
. . . . . . . . . . . . . . . . . . . . . . . . . . . C 2 Grammar file CapGrnmmarVien.grm 13-t C.3 Granimar file: protocol.grni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
D RoseRT Capsule Statemachine Behaviour 140 . . . . . . . D.1 Top state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... 140
D.2 Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-LO D.3 Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
D.3.1 Kinds of transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 . . . . . . . . . . . . . . . . . . . . . . . . . D.3.2 initiai point and initiai transition 141
. . . . . . . . . . . . . . . . . . . . . D.3.3 Transitions to and fiom the initiai state !41 D.3.4 Xo initial transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 D.3.5 Group transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
D.4 Choice points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L42 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.41 Choice Point Example I-12
D.5 Aistory hierarchical state machines . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-E . . . . . . . . . D.5.L Continuation kinds s h d o w histo- deep h i s t o - and defauit 143
D.52 B i t o ry Esample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 D.6 Transition se,- ents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
D.6.1 dunction points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Z U D.6.2 Joining transition segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
. . . . . . . . . . . . . . . . . . . . . . . . . . . . D.6.3 Continuation Kind Property 144 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.7 Tran5ition SeIection Rules 145
D.Ï.1 Transition Selection Rides &le . . . . . . . . . . . . . . . . . . . . . . . 146
E Capsule Statemachine Diagrams 148 . . . . . . . . . . . . . . . . . . . . . . . . . . E.1 Experiment 1: Ce11 Phoue Component 143 . . . . . . . . . . . . . . . . . . . . . . . . . . € 2 Experinient 2: Cc11 from Came of Life 149 . . . . . . . . . . . . . . . . . . . . . . . . . . E.3 Experiment 3: Traftic Light ConcroUer 1.50
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.4 Esperiment 4: Cruise Control l5-L . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.5 Experiment 5: Radar ConcroI Cnit 156
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.6 Experiment 6: Pop Machine 157 . . . . . . . . . . . . . . . . . . . . . . . . E.7 Experiment ï: Electronic Warfare . larnmer 123
List of Tables
4.1 Proteiim/FSM Mutation Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Proteum/ST Mutation Operators
8.1 Data hnalysis Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 E'cperimenc 1: PBX Ce11 Phone . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Experïment 2: Ce11 from Grune of Life . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Experirnent 3: Traflïc Light Controller
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Esperiment 4: Cruise Coritrol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 E'cpcriment 5: Radar Control L'nit
9.6 E~periment 6: Pop &tachine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Experiment E EWS Jammer
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8 Euperiment 8: Radar . . . . . . . . . . . . . . . . . . 9.9 Percentage of test sequences required to kill mutants
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10 Firm and Strong mutation results
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.1 Continuing Junctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Terminating Junctions (to h i s t o ~ )
List of Figures
2.1 Criteria Subsuniptiori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L:!
5.1 RoseRT SIotlel Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . -41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 RoseRT Class Diagram -E
5.3 Stnicture Diagram for SomeCapsuIe . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 .. .4 C S ~ I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5
7.1 Directon Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 . . . i 2 RoseRT Sliitation System:Essentiai Workfiows . . . . . . . . . . . . . . . . . . . . . . a i
7.3 RoseRT SIiitation System:Essential Workfiotvs . . . . . . . . . . . . . . . . . . . . . . 61 7.4 TAT example state machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.1 Skeleton Test Harness Class Diagam RoseRT SIutation System:Essential Worlcfiow . GS 5.2 RT-TestHarness Class Diagrani . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fi9 8.3 RT-TestCasehIanager . . . . . . . . . . . . . . . . . . . . . . . . .... . . . . 71
9.1 Continuation Transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y0
0.1 Xo Initial Transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-1:' D.2 Choice Point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-13 D.3 An Esaniple Cse of History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-4 D.4 . Junction Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14s D.5 Transition Selection Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
E.1 Ce11 Plione TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-18 E.7 CeIl Phone Conncctccl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 E.3 CeII From Game of LXc TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i-19
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.4 Traffic Lights TOP State 1.50 E.3 T r f i c Lights initializirig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.6 T r a c Lights FIashingAmber 131 E.7 Traffic Lights Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 E.S Traffic Lights Transitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L.53
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.9 Traffic Lights Green 153 E.10 Cruise Control TOP state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 E.11 Cniise Control On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5-1 E.13 Cruise Control Active . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.55 E.13 Radar Control Unit TOP State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.56 E.14 Pop Machine TOP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 E.L5 Pop Machine initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1% E.16 Pop Slachine Operating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.58 E.17 EWS Jammer TOP State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 E.18 EWS Jammer working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Chapter 1
Introduction
This thesis d l evaliiate the feasibility and effectirencss of performing mutation testing of iictive
objects modeled by hierarchical finite state machines (HFSbIl as implementetl in the Rational Rose
Red Time toolset ( RoseRT). This will be achievcrl by proposing a set of mutation operators. building
tools to perform miitation testing, performing esperiments on n sniall sec of HFSMs. ancl anaiyzing
the results ancl obsenations made during the experiments.
1.1 Motivation
Reactive çysterns iire a class of systenis whose main feaciirc is a tirnely interaction with ttic c r i 6
rotiriient. Ititcraçtion occttrs when the sgstem respoods to an environmental stimulus by performing
sonie output action. Subway control. patient hospitai monitoring, communication protocols. ancl
proccss controllers are esamples of such systems. The software cotitroiiiig reactive systems is often
niodeled by some finite stace machine formalism. Testing and d ida t ion of finite state machine basecl
specifications is done in generai by reachability andjsis. many t g e s of simulation. and by several
test sequence generation methods [Fabbrit 981. Yewer CASE tooIs such as the Rationai Rose R e d
Time tool can be used to design reactive systerns as coIlections of collaborating active objects n-hose
behavïours are motleled by HFSSIs. These CASE toois dso pronde automatic source code gener-
ation Gom the graphic design. HFSM have ex~ended scate variables. guarded transitions. choice
points. hierarchy. and may use timers to initiate events. Tt does not appear that conventional state
machine testing techniques are su5cient to handle the increased complexitj- of EtFSM PIÜiierburg
Testing's usual purpose is to reveal the existence of errors [Myers 791. From thi point of rien: n
successful test is one that reveals an error in the software. Generating successful tests is a rMcuit
problem that is impracticd in most cases. It usuaUy requires that a tester assume a <les!srructîve
attitude toward the software and then try to find or guess al1 of the w q s in which ~ h e software can
be made to fail. However it is impossible. in practice. to think of al1 of the ways a piece of non-tri~iaI
software rnay fail. Thcrefore systematic methods of testing software have been developed to guide
testers.
Nutation testing is a heuristic testing technique that offers a potentinlly higiily nucomatable. hrit
d so compiitationally espensive may of generating fault revealing test cases. In practice, mutatiuri
testing appears to find manu of the ivays that a piece of software can fail. Much rcsearch lins heen
donc (since 1978) on source code mutation tcsting. However the large compiitationd costs. iirriong
d i e r factors. liave preveritecl its wiciespreatl use.
The major rnotimion for this rcsearch is to investigate chc cost and effectiveness of iisirig rnu-
tatiun techniques to tcst active objects modeled by KFSMs.
1.2 Problem
The rr.sc!arch prablem acidressed bu this thesis is describecl as foIlows: 1s mutation testing a IeasibIe
iitid effective' nnay of testing reactive software modeled by hierarchicai finite state machine (HFSSII
formalisrns and created through gaphical eciiting tools stich as the RoseRT toolset? Siore speciti-
cally are the weaknesses of HFSM mutation testing as prohibitive as the weaknesses OC source code
mutation testing? And are the strengths of source code mutation testing as strong ivhen they are
applied to HFSM testing?
1.3 Objectives
The objectives of this thesis are as fo1Iows:
Provide a brief background on the general problem of softnare testing.
' EBecrive at revealing errors in a pmgram.
O Provide a thorough background on conventional source code bascd mutation testing.
Provide a brief background on testing finite scate machine software.
O Propose a set of mutation operators for use on HFSMs as implemented aith the RoseRT
toolset.
O Builcl tools to automate the process of applying mutation operators to RoseRT HFSJIs.
O Biiilcl a tool to facilitate the creacion of test sequenceç for RoseRT HFSSIs.
O BiiiId a generic test harness to reduce the cost and t h e reqiiirccl to test RoseRT BFSSIs
O Builcl i001s to help in the analysis of miicacion testing resiilts.
O Pmforni preliniinary esperiments CO evaluate each of the proposecl mutation operators and CO
wxiuate rriutatiori testing of RoseRT HFS3Is.
1.4 Contributions
The original contribiitions of this ttiesis inc1ucic the foIlowing:
O Critical analpis of the background Literature on mutation testing.
O Proposed mutation operators for RoseRT capsules and a prelirninary analysis of eadi operator.
O CEG for RoseRT capsuIe text files.
O 5Iutant creation toob basecl on TSL.
O Test sequence creation tool (RT-TestCaseS1anager)-
O Generic test Iiarness capsule ( RT-TestHames).
O Test results ,malpï~ tools.
O Development of an efficient procedure for performing mutation testing on capdes.
O Xnaiysis of the feasibility of mutation testing of RoseRT capsules.
O Identification of some areas of future rvork.
1.5 Synopsis
This thesis is rlivided into 10 chapters as foiiows:
This chapter introtluces the problem. defines the objectives. and lists the contributions of the
t hesk .
Chapters TNO. Three. and Four present background material relatecl to software test in^. niu-
cation analusis. and finite state machine testing.
CIiapter Five presents the RoseRT toolset.
Chapcer Six presents the proposed RoseElT mutation operators and their rationde.
a Chapter Seven contains an overview of the TSL based mutarion systcm and clisciisses the
application of mutation operacors to RoseRT models.
Chapter Eight covers creating test cases. running thern against the mutants, and iuidyzing
che results.
Cliapter Xine prcsents the resiilts of niucation cesting experinients concliicted in order to t m i -
iiate the operators and the feasibility of niucation cescing in RoseRT.
Chapter T m contains the conclusion. and discusses future work.
Chapter 2
Software Test ing Background
This chapter provitles a brief background on the generd subject of software testing. It provicles some
clefinitions and cliscusses the rneaning of correctness befoce t iming to the subject of test adequacy.
classification of testing techniques and how they cem be cornparecl. [t provides the context for
Chapter 3's cliscussion of mutation testing.
2.1 Preliminaries
In this work the terni prograni is tised CO refer to any individuaiIy testable piece of software. wtiicti
ma>* take the forn~ of a single function. a module. or a coniplete program. -4 test case is a single t-rihie
(possibly a vector) of progam input, which enables a single execution of n progrruu. X test set or
test suite is a finite collection of test cases. Testing definitions ÏmplicitIy assume an unrcalistic but
simple "functiond"program architecture: a batch program which is given a single input. cornputes
a single result and terminates. Each cornputation is a function stnctIy of the program's ciment
input. not on prior program States. i.e. there are no sequences of inputs. .Uthough this assumption
is iinreaiiitic in generd. it has proven to be a uselur simplification in esploring and communicating
ideas in testing theory [Hamiet 941. Testing softs-xe in whkh the current output is a Function of
both the curtent input and of previous inputs (often described by h i t e state machine formalisms)
requires test cases that are time sequences of input vectors. Lesting finite state basecl software
is ciiscussed in chapter 4. rlppendk A contains Further definitions and additional terms used and
abused in the testing iiterature.
2.2 Test Oracles
A fundamental assumption of sohware testing iç thac there esists sonie mechanism. separate €rom
the program under test (PUT). for cletermining whether or not the resiilts of a test eseciition are
correct. Siicti a niechariisrn is referrecl to as an oracle. K e c m consitler some process in n-hich test
cascs are taken as inputs hy both the PCT and the oracle. computations are performed. ancl resriits
are comparecl.
Itleally a e Kant an automateci oracle which always gives the cnrrcct ansaer. Often howewr.
oracles are people. who mostly compute by hanrl thc expectecl results slioidcl he. This is n
tlifficiilt and error pronp process, thtis hiinian oracles can make many rnistakes (e.g. Xmmatin et
al. [.4rnrnann 9-11). -4s n resiilt rvheii rhere is n iliscrcparicy bectvceii clic oracle and clic prugrarii.
chc rcsult reportcc1 bu c h ornclc rnust often be verificd before clecicling thnt the PCT is in error.
Hiiman oracles often use whatever specifications exkt for the PCT. However ic is often the case thac
specifications (if the' esist) contain errors hr ther conipouncling the oracle problem.
2.3 Test Adequacy
Testing is plaguetl bu problenis of complcsity. Tt is impossible to completeIy test a prognni n-ich
evcn moclerate complesity. One could. in theory. show program correctness by riinning the program
on e v e n possible input (exhaustive testing) and chccking the resiilts. tiowever. for any non-tririal
progam the set oFpossib1e inputs is so large. possibly infinite. that exhaustive testing is not feasible'.
For example. if a progarn ha three 16 bit integer inputs. (ranging over 63336) exhaitscive testing
woultl reqtiire running the program over lOL.' times. ifeach run took one microseconcl. this still takc
over S years to test. Red software is much more cornplex WC can never be sure that any morlerrrteiy
sizeci piece of software is correct. The best that can be clone is to perforrn lenough- testing to gain
coufitience that the program works and d l continue to work as specifietl. ï h e question becomes
%on- much testing is enough'!" The a m e r is bound up in the notion of test ndepacy.
Since a program cannot be e.xhaustivelj- tested. much of the aork in testing theory involves
findimg subsets of a program's input (Le. test case selection) that na choroughIy test a pro.pni.
The questions to be annrered are [Goodt 751:
':\ pmgram that could be euhaustively cested could be replaceci by a finite table that contains dl of the .ansivers!
a 1s there a way to test a program which is equivalent to e?diaustive testing?
O Is there a practical approsimation to e.xhaustive testing and if so how gootl is the approxirna-
cion:'
A Test selectiori cntenorr (or test criterion) is the bnsis for test case selection. A tcst cricerion
specities the set of conditions chat must be satisfiecl by a set of test cases. in [Good+ 731 the nuthors
rlefined an ideol test set as a test set that succeedsJ only wtien a program concains no mors. A
critenon for an ideal test is both reliuble ancl uolid. ,-\ criterion is reliuble if al1 the sets of r t s L vases
chat satisFy thc criterioti clecect the siune errors. In other worrls it (lofs not mattcr which satisfying
set is selected. every set will detect the sarile errors. .-\ critcrion is u d i d if for any wror in t h
prograni there is some set sacisfjing the criterion chat will reveal the error. [C;ood+Ï3] prowtl tliat
if a testing criterioti is both txlirl and reliable. and if a tcst set satistjing that criterion fin& no faults
in a program. chen the prograni contains no errors. Homevcr it has been shon-n that no dgorichm
exists that rcill dctcrmine a valid criterion for an arbicrary progani. Thcrefore the answer to the tirst
question is no: there is no way to test a non-trivial program chat is equident to eshaustive tescing.
Further the reliability requirement is too strong to be itiecl to clefine practical tcst ncleqiincy rriteria,
Findiig a criterion that is both reliable and valid and chat can also be satisfieti bu i l nianageable
number of test cases is usuaiiy not possible. Therefore the focus of test arbqitat~ rritcrin researrh
is on fiuding practicd lipprosiniatiotis (heuristics) to the icled criteria.
In practice a test atleqiiacy criterion can be used as a stopping nile to cletermine whether or not
enough testing has been done. For esample staternent coverage states chat enoiigh testing Iias h t m
done when every statement in the prograni under test h'as been esecutcd at Ieast once. Wiether
o r not this is a good criteria is a separate issue. however it is a criterion (i.e. it's a Lieuristic). A
criterion c m also be use CO measure the quality of testing. In the case of statement coverage the
percentage of statements covered by a test set gives a measure of the degree to nhich a test set
h a satisfied the statement coverage criteria. in industry chi measure couid be used as a stopping
criteria. For esample. stop testing when 85% statement coverage has been achieved. The term
adequacy criterion is often used to mean testing method.
-hocher role of a test critenon concerns the observations to be made during testing. For esarnpIe.
statement coverage requires chat the tester sornehow determine if each statement in the program
Lias been esecuted. This usualIy requires instrumenthg the program under test.
''ln this sense succe& means that z test euecures without revealing an emr .
2.4 Classification of Test Data Adequacy Criterion
The goal of a test criterion is to find a stibset ai the possible inputs to ri program that rviren feri to
the program as test cases arc to rewal fauics in the program. Since no algorithm exists co
fitid chese subsers. testers ~ e l y a n a number of techniques and heuristics that work wel1 in practice.
but largely Iack an undcriying theary The testing literaturc ciiscusses many ways EU cI;issih. these
techniques. h comrnon way of organizing testing technicps is to cntcgorize tiicni iu being risage
b a e d techniques. black box techniques. or white h m techniques.
2.4.1 Usage Based: Test cases corne from the expected usage profile
-1 usage or aperatiand baser14 criterion cari be based on stocfia.çciç infriruiriticiti about hon- t l ~
software id be used in the field. An input probability ciensity futictiori is neetlctl. cxprcssing
the probability chat somc input will actualIy occur in tise. Operational testing then proccerls bu
seIecting independent randorn points [rom the input cfomain accorcling tc, this ilistribrition. Ysrially.
an accurate operationai profile is far too detailed to obtain. and even a crude apprnxiniatioii retptires
considerable effort [Hamiet Nb]. Operational profiies ate often trsetl s i t h the grid of qtiancifying
the reliability of the PUT.
2.4.2 Black Box Techniques
Black box testing is also knorvn as functionai analysis. It is baser1 on the idea that a progam imple-
ments a function that maps d u e s from its input ciornain to d u e s in irs outpiit tfomain. W t h black
box testing the source code of the program under test is not lookecl at, rncher test cases iire tisually
i leri~ed in tetms of the program's mapping as described in the program's specificncion. Boaever
some bIack bos techniques can be nsed in cases where a program does not have a speçification or
where the specification is incomplete.
The folIon-ing techniques use characreristics of the input domain in attempting w hi1 represen-
cative sets of test cases-
4Cssge based is &a d l e d operatiooril testing and is sornctimes called mdum testing Rudom t a t i n g c m also be IUMI in the tvider context of statisticd testing from any disriburion. rarher chan the nne considermi tiere whicli is based on opentional use [Hamiet N b ] .
Equivalence partitioning
The idea behind equiwlence pnrtitioning is to partition the input domain into a small number of
.-eqiiimIence'- classes. Testing a single element from an equivalence class is incended co be equi\-alen~
to tiaving testeci al1 elenients of that class. In a sense it attempts EO b d reBabIeS test cases. This
heuristic is often b s e d on the system design description. The clomain of each input ~wïabIe that
p l u s a role in one of the components is clivided into 2 or more sets of values. These sets of wIiies rire
the equivdence partitions that are further classifiecl as being either d i d or inmiid. -4 specification
Iikc riach hook has u 6-digit number coidd resiilt in 3 equimlence classes: one vaiid. and 2 invalicl [iliet
931. .-\€ter finding the e q u i d m c e classes. test cases are constriictecl siich that as many. (possibty
d l ) ec~i i i~~~lence classes as possible are -coverecI".
Equi\dence partitioning suffers from some obvious weciknesses:
i. the assumption that the program beliaves equivdently for al1 data from a given dass is not
crue in general[Vliet 931. This is because no algorithm exists to finrl the reliabk partitions for
an? non-trivial program.
2. Testing c.orribinations of ecpiitdence class members is not considered. If a coniponent is a
fiirictioti of two twiables. x anci y. ancl if botti s and y can be partitiritied into three eqiii~tlence
p,xtitions each then there are 9 possible mys to combine exemplars from the 6 partitions.
Same of these cornLinations will probably reveal errors that are not cletectable by just itsirig
the minimum 3 test cases that are adequate for eqtridence partitioning adequacy.
3. Without iooking a t the structure of the PUT. it is probabiy impossible to dctennine dl of the
eqiiivaience classes [[Jalote 971: equivalence class boundaries rire tlefined by the structnrc of the
code.
FinalIq- it is often worthxvhiie to use the equitdence classes of the output. For an eqiiitalent
output cIass. the goal is to generate test cases such that outputs Lie in each of the output equi~aience
ciasses. This is more difficult than finding input partition classes. however output classes have b e n
found to reveai errors chat are missed by simpiy considering the input [.Jaiote 971.
:Reliabie in the sense OF [Cood- 751.
Boundary Value Analysis
Boiindary d u e anaiysis is a variation of equinlence class testing and is basecl on the observation
that software often fails on boundan: \dues (eg . off by one errors). Rather thcm testing some
rantlom element from each equivalence class. boundary d u e anaiysis concentrates on the extreme
d u e s from each class. Cornmon boundary idues are I. 0. 1. nidl. ancl Iists witti 0. 1. or r r w x
elernent(s). Extreme values from the output classes c,m also be tested.
Botinciary mlue analpis suffers from the same weahesscs as cqtiivdencc partitionirig. L i t i i s ~
~ffeçt graphing [JIeyers Ï91[.Talote 971 seeks to ansxer chc combinatorid tleficiency of etpi~alence
nntl boitnrlary value anaiysis bu tcsting cambinations of cquivalence class elements. honever it does
not scnlc well. As the software g o m in size the number of test conibinations grows exponentidly.
Thc combinationai problem is not rcstriccetl to cause-effect graphingl racher it is another facet II€
thc intractability problem pensive in tcsting.
BlackBox Testing Observations
Sirice I h c k bos testing does not reIy on the implementation. test cases CIO not become invalid if and
nhen the iniplenientation is changed (as long ris the specification cloes not change). [n ncldition.
test cases can be devetoped beforc or in parrailel n'itli the progr~un. On the other Iianti specifications
are often incorrect or incomplete *ut1 are iisiially wi t ten in natiirnl langiiage n-herc nrnhigtiity is
iiievitable. Even wlien a forrtid description Ianguage is usetl. it is likely that sonie pottioti t i f ctit*
specification has been missed or neglected: incompleteness is always a prolkri . Fiirttier. :r sp~citi-
cation itself c m becorne intractable because it is not possible to specik every possible n+ay in hi ch
software can be tisecl or abtised. Finall-. even when specifications are the resdt of a collaborative
rer~tiircttietits elicitation phase involring the customer. completeness is elusive. Ciistomers c m usii-
aliy tell whether a prototype is. or is not. n-hat they m n t . but they are unable to spec- exactly
what they n-ant until it has been buiit. Specification problems contribute approximately 30 percent
of dl bugs in software.
2.4.3 White Box: Test cases corne Çom the source code
Khïte box testing criteria. aisa h o m as g l a s box or cle,u box testing, can be dividecl into m o
classes: structurai and fauit-based criteria. Fauit based criteria are further ciassiiied as b e h g eicher
error seeding or mutation testing.
Structural Testing
A large portion of the testing literarure focuses on structural testing techniqiies [Zhu 9ïIFliet
931[.laIote 971. As a result there arc many ccchniques known. With stnrcti~rd testing. test cacç ;ire
generated based on the actual code O€ the PCT. The tester c m then observe the extent cn which
die test cases esercise n prograni's structure. The underlyirig prer~iise of strucciirat testing is chat it
is riecwisary to krinw ttiar id1 thts parts of sorne stnictural aspect (for naniple statcrnents, branches)
of 3 progrnni have bwri tuercisecf at l e s t orice. 111 practice this rnay bc tlifficiilt or impossible to
achieve.
Codc cowrage cesting is b ~ ~ c t l on the numbcc of i~istrrictions. stntements. hrnnrhes. or pnt1is
that are c~eciiteci at mn time. Code coverage c~an be cIisctrssec1 on the hasis of il prngnni's control
gaph. In clic concrol graph. nodcs rienoce actions (staterncna. and stntenicnt btocks). wIiilc the
directed edges connect actions with siibsequent actions. -1 path is a sequcncr of nodcs connected
ttiroiigh etlgs. Cyr-lcs in thc grapti correspond to Ioops in the program. Prograni esectition reqirires
following n certain path ttirough the p iph . When a node has muItipIc otrtgoing cclgcs. test cases
are designed to select among them (these are br,mches).
in the ided case. test cases traverse al1 possible paths. This is cdletl di-paths coverage and
k eqiiident to eshaiistively testing the progam [kliet 931. In general ail- pattis coverage is not
possible for the fol1owing reasons:
1. A loop often resuIts in an infinite number of paths.
2. For a branching progiïm. even in the absence of Ioops. the number of possible paths increases
eqonentially with the number of branching instructions.
3. There rnay exist some infeasible paths. manifesteci as branches that never euecute. The inka-
sible path problem is undecideable .and is the main probtem with d path anaiy5s methotls
@amiet 941.
The best that can be done in the presence of these problems is to find a coverage that espresses
the degree to which test data appro-xhates t h ideal coverage. Xany such coverage metrics hat-e
b e n studied. The most obvïous is to sMply count the number of nodes traversed. This is c d e d
statement coverage and is often used in industry (possibly because of its ease of comprehension, use.
ancl measurement). However statement coverage is rather weak as it is easy to h d test sets hi ch
give 100% statement coverage yet leave undetected faults in the program.
X srronger coverage constructs test cases that ensure that every br'anch in the control vaph is
tnken at least once. This is ail-edges or more commonly branch coverage. Again 100% coverage does
not gitarantee the absence of faults.
Another category of coverage is cdecl data-fion; analysis in wtiich the control graph is irsecl to
esamine the clefinition 'anci uses of variables dong esecution paths. X mriable gets clefinecl at ;i
certain statenient if it is assignecl a new vnliie at that statement. \:ririables are then siibsecpmtlj-
itsed in one of tno wqs: P-uses are predicate uses such 'as those in the conditionai part (if if
statcrnents. rvhile al1 other uses are C-uses (computation uses). A path from a "definition" to a
.'iise" in which the variable is not redefinecl is calleci a definition clear path.
In the dl-u.ve.5 .vtrritegy, test cases are constructecl that traverse definition clear paths between
cach clefinition of a wriable and each subseqiient use. This ensures thnt cach possible use of n
clefinition has been tested [Rapps+ 351. There are several other wiriants of rhta Hom- strate+ siich
as d l P-uses. dI C-uses etc. that c m be seen in Figure 2.1.
Figure 2.1: Criteria Subsumption.
in practice coverage methods may not be cost effective. Some code staternents' such icj ~ r r a r
banders. may be ctificttlt to execute and also be uniiiely to occiir in use6. Aclding test cases to
cover such statements may not be worth their expense.
Fault Based Criteria
Faiilt b;setl adequacy criteria measures test adequacy accortling to the ability of a test set to cletect
known faults in a program. Error seeding and mutation testing are examples of t h i cnterion.
Error Seeding
Error sceding is a technique for estimating the number of faiilts in the software. It \vas origirinted
by CIarlan Slills in 1972 [I<night+ $51. The original iclea ans to estimate the reliability of ii piece of
software basetl on an estimate of the number of faults remaining in it. Artifcial faults are seetled into
the progam. <and the testing is etduated based on the number of seeclecl faiilts found. Error seecling
is based on the assumption that artificial errors are as ciifficuit to detcct as naturally occirrring
errors. Cnfortiinately artificial faults are not necessarily representative of natural faults ancl this
assumption has proven to be untme in general [Gardiner 991.
Mutation Testing
Sliitation testing is the process of systeniatically constriicting small syntnctic variations. called
mutants. of a progam and then determining if they c m be clistingished by esisting test cases. An
example r~iutation wouid be to change a greater than or eqiid operator to ;i grcater than optmtor.
Each mutant that is distinguishecl by the test set is termed -killedU. the rerriairiirig niiitarits are eithcr
equitdent to the original program or ineqiiitdent but rindistinguished by the test set. Ec~iii~lilent
mutants are d isc~rfed while trndistinguished mutants require the addition of new test cases. The
funclanientd premise of mutation testing is that in practice. if a program contains a real fault. there
miIl tisually be a set of mutcuits (representing artificiai faults) that can only be M e d by a test
set that also detects the reai fault (Offuttt001 [Geist-t-921. Kowemr the fauit reveaiing ability of
mutation testing is highly dependant on the set of mutation operators used to create the mutants-
Slutation testing has remained an academic endeavow. however it is experiencing a great deai of
interest because it provides a directo quantitative memure of the quaiity OF test data For a program.
"or example creating a test case to invoke an enor handler for a rnemory allocation may be difficutt to do.
Mutation testing approximates boundaq testing: bounclaries are the edges oF regions on nhich
mutants are killecl [Hamlet 94. For example in boundary analysis a pretlicate e-qression may define
a botinclary on the input data. Killing al1 mutants of the pretlicatc espression implies thac the test
cases have arleqiiately testecl the botindaries cfefined by the predicate.
2.5 Evaluating Criterion
Hon do we compare one test criterion to another? One ml1 kriown n.ay CO compare test criterion is
through the subsumption relation. A test criterion Cl subsumes another test criterion C2 if wery
set of test cases that sacisfies criterion Cl also satisfies Ci?. For erample. if a test set ha covercct
every branch in a provani (satisfied the branch criterion). then that test case is giiarnriteeci to have
also coveretl cvcry statement. One s q s that hranch coverage subsumes statement coverage. The
subsiimption criteria can be clisplayetl in a hierarchical chart stich as figure 2.1 [Gardiner 991.
Only some mcthods. those n-ith a common basis can be itsefiilly coniparetl iising siibsiinies. For
example. variants of path coverage can be compared. Hosever. niariy methocls are incomparable -
neither sirbstimes the other. Further the siibsumption relationship has a serioiis Han.: siibsurriptiori
is only valid under LOO% coverage. Branch coverage subsumes statement coverage. in other ~ o r c l s
if every branch in a progam is covered then every statement is also covered. Howe~er if some test
set Ilas a brruich coverage of 99% subsumption s q s nothing about statement coverage. The 1% of
itncoveretl branches rnay lead to a miich I'uger proportion of rincovered statements. However. in
generd. a high branch coverage iisudly implies a high statenient coverage and therefore a stroriger
test set.
Regardess of a criterions position in the subsumes hierarchy no single testing technique is guar-
anteed to expose di possible failures. Eventually d l of the ciefects a methocl can fintl are forrntf
ancl the methods usefuiness (for esposuig faults) is enclerf [Hamlet 941, It is generdly agreecf chat
an effective testing strate= musc include several clïüferent test methocls. Different techniques tent1
to reveai clifïerent types of faults. Therefore the iwe of multiple techniques usualIy results in the
discovecy of more faults.
Endiy some tvork indicates that human factors are more important than any theoreticnl factors.
Sonie esperiments [Hamiet 581 show that the most important factor in software testing ma- be
the ski11 of the tester. This fiuther clmages the usefulness of iising the subsumes relationship. IF
a technically more powerful criteria (Le a criteria -4 that that subsumes criteria B) is harrler co
use. then an rinskillecl tester using it rnay not test a program as well as n skilletl tester tvho hases
his testing on the subsumed cnteria [Hamiet 891. Siormver the attitude of the tester may be an
important factor. It appears chat most successful testes (i.e testers who create tests that reveal
rnany faults) adopt a destructive attitude tomcl the PCT LJalote 971. It is argued that the cleveloper
of a program d l find it difficult to adopt a clestructive attitude towxrd the program that he has
created. therefore a cleveloper shoultl not test their onm work. Fiaily Hamlet has proposecl that
perhaps it is the process of iising a methodical test technique rather than the techniqiie itself thnt
rnakes testing siicccssful. He ~heorizes that bu forcing testers to pay systetriatic attvntiori co die
software. testers are tnorc successful ac finding ktults [Haxilet SS].
2.5.1 Summary
Testing is a costly business. Testing dong with clcbttgging and fixing the cocle consiinlei on nwrngp
about 50% of the time and money budget of a typicai software cleveloprnent projcct [SheparcI+ 011.
Much of the cost is attributed to the person-hours spent in gcnerating test cases. riinning clic tests
and comparing the results. Most test adequacy criteria are untlecicleable. therefore no algorithm can
be usecl to generate appropriate test cases. Most often the generation of test cases is carried orit
rnanually bu the tester(s). hspecting the resuIes of test nins is aiso tiine consiiming.
With the rising cost of personnel and the decreasing cost of CPC cycles there is a strong moti-
vation to automate as rnuch of the testing process as is possibleÏ. Mutation testing (to be ftirther
discussed in Chapter 3) has several features that are anienable ta autoniation. It is also consiciered
to be among the strongest testing techniques with respect to its ability to reved faiilts in a prrigram
[Offut+ 96b!.
'However. automating the entire testing pmcess is neither feasible. cost effective. or an effective way of revealing faults
Chapter 3
A Detailed Look At Mutation
Analysis
This chapter takes a detailed Iook at mutation testing bu tliscussing hon- niutation testing is prr-
lormed. the theory supporting it. and its strengths anci weaknesses.
3.1 Mutation Testing
Mutation testing is a systeniatic method of obtaining a thorougli test set. During mutatiou testing
mutant versions of the ptogram under test (called mutants) are created bu seeding the origina1
program with faults. one faiilt per mutant in accordance uith a predefined lauguage specific mutant
operator set. Test riata is then used on the mutants mith the goal of causing each mutateci version
of the prograrri to fail ( a mutant fails if its output differs from that of the original program). This
is called kiiüng a mutant. W e n a mutant is killed it is considered dead and is not iiseci in hrther
tests. A test case that c m diïtingtikh ( H l ) a mutant W considered useltrl. others are not.
Mutants that are not kiiied by any of the evisting test cases fdI into one of three cntegories.
First. the mutant couid be functionaily equident to the original progrmi. in wtiich case it c:annot
be cListin,@hed from the original. Such mutants are discarded. Second, the mutant is killabIe but
the test set is inadequate. In thii case the test sr: must be augmented by the addition of a nen-
test case. F i d i y the ori,@al program may be fauity in mhich case the mutant may represent the
correct version or may reveal the fauit.
A live mutant directs a tester to the exact statement chat musc be reached. The tester then uses
inforniarion about the Iive mutant to generate the nea test. In the esample of substituting -! =* for
'<'. these t ao operators a g e e on the ahole range where the '<' espression is TRL'E. If test clatn
only falls in thï range then it cannot distinguish the mutant. The Live mutant implicitly poses the
question: "the test data given so far says it doesn't matter whether or not this operator is .! =' or
.<y - . why is chat?'' The ansn-er d l guide the creation of a ne%- test case or possibly reveal an error
in the original program [DeSIilIo 781.
Once enough ncw test crises have heen aclcled. ail of the live niutants are esecuted against al1 of
the ne%- test cases (an? test case may. coinciclentalIy, MI other live mutants). The iterative cyrle
tif acltling new test cases and esecriting them against the live mutants continues iintil either no live
mmants remain or the mutation score is high enough to satisfy the tester. The mutation score for n
test set is the ratio of cleaci mutants over the total number of non-eqiiivdent mutants. -1 mutation
score of 100% is cdled mutation-atlequate and satisfies the mutation test aclequncy criterion. In
practice miitation scores of less than LOO% are often consicleretf goocl enough. However. assumin::
100% mutation aciequacy can tic achievecl. Fhat cloes this say about the correctness or qualit? of
the progrni so testeci'! The answer depencls on several assiunptions.
3.2 Mutation Theory
l h t a t ion analysis is preriicatecl on the existence of a class F of faults chat can he irijectetl into
n prograrn. It provides a set O of operators. knom as mutant operators cach of wtiich mocltiIs
one or more of the faults in F. Each proqamrning langiage has ics own set of mutant opcrntors.
ln tirircltvare. testing is baset1 on fault modeis that closely model actuai hardware fauits (such as
stuck at 1. stuck a t O faults). The hu l t model provides a set of simple faults that can be combined
to mode1 any fauIt in the hardware. However. since software is more comples thaa hardware. no
hardware inspired fault model exists for software [Cai 981. Testing techniques like mutation must
use heuristics to establish the fault class F and its corresponding mutant operator set 0. Hoaewr.
if rnutation testing can Hi di of the mutants of a progam. then that program c m be asured to
be Eree of the errors represeated bu the mutant operator set 0. Hence the effectiveness of mutation
testing ciepends on the accuracy and compieteness of the fault model as modeiied by the mutation
operator set. in principle. mutation testing based on an exhaustive fault mode1 (Le. ail possible
faults considered) would constitute a perfect test method. Kiiiing al1 the mutants would show that
no other program agrees with the program under test on the test set. hence the program under
test must be correct if any prograrn is [Hamlet941 [Budd+ 821. Bon-e~er. it is impossible to predict
al1 of the ways in which software may fail. therefore sohware faiiIt classes as modeled by mutation
operator sets are based on Desses (heiiristics) about what kind of faults can he introcliicecl into a
program.
Mutation testing assumes that faults in n program are the result of mistakes matle bu program-
mers. .A mistake made by a programmer may leacl co one or more faults locatecl in the tcxt of a
pïogram. Since a fault in the text of a program can take many forms. mutation testinp further as-
sumes that proganniers protliice nearly perfect programs in which mistakes are syntactically small
tleviations of the intended program. This is called the competent programmer hypothesis [DemilIo
YS]. In reality this hypothesis does not hold. Although a fault may be a syntactically small devin-
tion in a sin& stacement of a program it could just as n d l j im be spreacl ovcr several LinesS. Tu
overcome this problem mutation testing proposes the coilpling eflect hllpothesis. which States thar
simple and comples errors arc coupled. This implies that a test that kills a simple non-equitalent
mutant will aho kill a complex non- ecpivaIent miiram. The coupling effect hypothesis has been
srrpporterl both empiriçally [.-\cree+ 791. [DeiIillo-t 791. [Offut 921 .and theoretically [IIorY-L]. Baseti
un these hypotheses rrwstg niiitation uperators rire selecteri €or tlieir irbility to niotlel srnall syntactic
faults chat rnight bc the result of a mistake made by a programmer. This leacls to the probtern of
discovering what kind of smdl syntactic faults a programmer is most likely to niake.
3.3 Applying Operators
XppIying a mutation operator to an original source code stacement results in what is called a first
order mutant. in general one application of a mutant operacor can generate more than one tirst
order mutant. For esample repIacing a logical operator niith every other possibIe logical operator
resdts in severd distinct mutants for each logical operator in the on$d progam. It is &O
possible to appIy a mutation operator to a mutant program. These -mutants of a mutant" are
ralled second order. Because the number of mutants groas expanentially it is iisiiaiiy not practicai
to use anything but k t order mutants- This decision is justified by the competent programmer
-1t could even be a fault of omission. 3Stuch of the mutorion Iiterature implies that mutation operators are designeci strictly to simulate programmer
crrors. Kowever, this is not the case; mutation operators are amïved at thmugh various means.
hypothesis in n-hich it is argued that first order mutants simulate cedi t ic programmer errors %hile
higher order mutants do not [DeMillo 781. More importantly empiricril m r k by Budd and Ltathirr
support thii restriction to first order mutants. Budd sho=ed that higher ortler mutants do not
provide significant advantage with respect to the construction of a better test suite [Buclcl 801.
S[athiir cites experirnents in which test data chat ~+,?s able EO ciistinguish mutants created bu simple
operators. was also able to diitinguish iip to 99% of the liigher order mutants [Mat+ 911. It appears
that tests that cletect simple faults collaterally detect any large faults.
The theoretical foundation of mutation testing ultimately rests on the vdidity o i the campecent
programmer hypothesis and the soundness of modeling faults by simple syntactic changes to a
program. These appear to be weak foundations: the conipetent programmer hypothesis does not. iri
generril. holtl while simplicity means that operators niay fail to yieltl hiilts that are represctitatiw
of iritligerioiis faiilts. Sonie stiidies indicace that simple niutarits can exhibit bdiavioiirs sirnilar to
the behaviours of trrie faiilts [Darad9951 however there are other stutlies that indicace the opposite.
Fiinher. while achieving a mutation score of 100% shows that the program cfoes not have .an? of the
faults modeled by the mutation operator set the meaningfulness of this vaiile is tlepenclent on the
cornpleteness of thc fauit model.
.Uthoirgh the iintlerlying theory of mutation testing appears to be weak. in practice mutation test-
ing h a turned out ro be one of the strongest known atiecliiacy criteria [Butlrl SO][Offtitt 36b][Fr'rankl
- 971. Xlthough a direct comparison via the stibstimes relationship is noc possible. mutation cestuig
appears to be as good a t revealing errors as data How testing. [Franklf 971 [Offrit+ 96bI. Uutatiori
cesting approsirnates other cesting hcuristics: in Wong et al's [Wongt 971 study of T l C operacors.
sonie operators aere identified as being able to:
1- force a tester Co select test cases horn different parts of the input doniain
2. select test cases From nearby parts of the input clomain of each scaiar reterenco
3. construct test cases to esamine points on or near predicate borders
4. review the logic of the program under test by mutating di Iogicd operators
5. obtain statement coverage
6. force each arithmetic expression to take on a negative, zero, and positive \-due
Mutation testing encourages the construction of test data that distinguiçhes mutants, iiihen test
data fails to distioguish the mutant. and when the tester cannot show that the mutant is ecluitxlent
to the original. then the e-xistence of an error becomes highly probable [Wong +97]. A difficiilt to hiIl
mutant tiirects the progarnrner's attention to poorly covered and possibiy poorly nritten sections
of the code. -1 mutant that is difticult to kill may in facc indicate untestable code, Le. code that is
tao cornplex to test. Such code shoidd in most cases be ren-ritten.
3.4 Mutation Operators: Simulating Programmer Nlistakes
The mutation operators of current mutation systems were cleveloped primarily iising the researcher's
on-n programming esperience and consiclcration of typical programmer errors. For cxrimple die C
mutation operators rvcre *hc result of -long delibcrations aniongst the authors in which ieveral
aspects of the C langiage and program developrnent of C were examined" [=\grawal+ 391. Thc
hlothra system @lehIiüo 881 uses '11 operators that were derivctl from sturiics of -tupical crrors"
made by Fortran progr,ammers. [t has been arguecl that esisting mutation operacor sets are itnlikdy
to be sound. as thcy ~ e l y totally an the authors cspcricncc tvhich rnq be hiasecl and insotriplctc"
,anci they may have tliscarclecl certain types of mutation operacors or else simply not (:onsideret1
certain types if ttiey had no experieiice tvitti them [Iiini+ 991. Kim et al. argue that a so~uid
operntor sec must be generaterl throttgh some systematic mechotl such as the rhemical incliistry's
HXZOP (Hazard and Operability Studies). They have used HAZOP to generate "rigorous" mucation
operators for the Java language and are currently building a Java mutation system to validate tlieir
daim. However the HAZOP-Java case is the esception and most operator secs are the procfuct of
the researcher's intuition.
Some mutation systems also inclircle operators that enforce statement coverage. branch coverage.
domain coverage. and extreme values criterion v o s + 981. Although these latter operators do not
simulate programmer mistakes they are included in order to enforce other testing heuristics and
have the goal of improving the fault revealiig ability of mutation testing. ;Uthough the rationale
behind some operatos is global (i.e. mutating reiational operators). mutation operators are generdly
Ianguage dependant. Operators for two Ianguages can differ due to the differences in their respective
syntas. and due to features that are intrinsic to a particular language. For esample some of the
M d 3 operators deal 6 t h that Ianguage's unique tasliing feature. The foilotving paragraphs describe
some esisting operator sets.
The Mothra system from Purdue University and Georgia Tech. is probably the most well knom
mutation systern. It automates the process of mutation testïng Fortran 7'7 programs by creating and
executing mutants. managing test cases. and computing the mutation score. Each operntor is rep-
resented by a 3-letter acronqm. For euample. che -array reference for array reference replacement-
(.AAR) mutant operator causes each am-. reference in a progam to be replacecl by each other clis-
tinct am- reference in a program. Similar naming conventions are used for other mutation systems
(Proteum's C operators have 4 letter acronyrns). Slnchra's 22 operators are goupecl into S classes.
which are further grouped into 3 levels:
1. Statement analysis: This level corresponds to craditional statement aniilysis tests. i.e. trsting
thnt al1 statenients are executed. i d that every stntement has an effect.
( a ) replace each statement by TRAP (execiiting the TRAP statement niIl kill the rnrrtant).
ttiis ensiire statcmcnt coverage:
(h ) replace ench statement by CO-iTISC'E:
(c) replace each statenient in ii subprogrm by return:
(cl) replace the targer label in each GOTO:
(e) replace the label in each DO statement.
2. Predicate .an<[ domain anaipis: Thii IeveI c m be icjecl to create test data to ~xercise al1
preclicate houndaries ancl data domains.
(a) take the absolute d u e or negatke absohte value of an e-xpression:
(b) replace one arithmetic operator by another:
(c) replace one relational operator by another:
(ri) replace one logical operator by another:
(e) insert a un- operator precehg CU expression:
(f) alter the talue of a constant: alter a DATA statement.
3. Coincidental correctness: This IeVd provides test data that e l î a t e s coincidental correctness
enors (smdl errors in the original program that have gone undetected because. by the nature
of the test data. the program h a ~ coincidentdy produced correct resuits).
(a) replace a scalar variable. array reference. or constant by another scalar variable. ar-
reference, or constant:
(b) replace a reference to an array name by the name of another may.
Fortran is an old language and lacks many of the features found in later and syntactically richer
lnnguages siich as C and Ada. -4s a resuit the 4Iothra system uses on- these 22 operators that were
tlevelopetl ,and refined over a 10 vear period poas+ 981.
CSas et al. propo~ecl mutation operators for -1daû3 [Voast 981. The Ath language hris many
features not found in Fortran. This required studying how to miicate programs that iise:
srrong typirig and information hiding,
0 dynartiic tlnta structures and rtatic long lived data stornge striictures.
cornplex control structures sucb as exception handling and tasking,
0 conciirrcnt processing
As a result of this study 65 operators were clefinet1 in 5 categories [\*oas+ 981:
0 operanil ceptacernent operators whicli cause each operand to be replacetl by each other syn-
tacticdly Iegal operand
0 statement operators which modify entire statements and modik the cantrol structures of the
Ianguage
0 expression operators which niodify matheniatical operaton and entire espressions
0 Coverage operators: Decision coverage is nchievecl by replacing cach dccision by TRLE and
FALSE. Condicion Coverage is ncliievetl bu replacing each condition bu TRLT and FALSE. Tt
is possible to iise both decision and condition operators to obtain rnuitiple condition-decision
covcrngc. mhich requires that the entire tmth table for a decision be covered.
0 tasking operators which are specific to the lda53 tasking constructs
Since the h d a mutation operators have never been irnplemented by a tooI the d i d i - of this
operator set has not been empiricaiiy evaiuated. Mutation operators have d s o been proposed for
C[Xgawal+ 891 p e l a m a r o t 9Tfl COBOL [Dehlillot SSII and .Tava Kim-+ 991-
3.5 Weaknesses
The niain problems of mutation testing are the large cornputationai and human resources required
to carry it out. Mutation testing is expensive in terms O€ both cime .and space complesit. BuclcL
anaiyzed the number of mutants generated for a program and found it to be roughly proportional co
the product O€ the number of data references tirnes the number of data objects p u d d SOI. This is a
large number for even sniaiI progarns. For esample a 4 line program usecl in [DcSIillo 781 gcneratps
40 nuitants. Each mixant is tquiwlent in cornplcxity ;incl size to the orignal program ancl musc
h c riin ngainst eacti test in the test set tintil it is kllecl. Sonie of these n;iit.mts tvill noc teminace.
some are eqiiivalent. and some are iindistinguishable from the ociginai program by the curent tcst
set. Eqiiklent iiiiitants are another major probleni. Sorne empirical data indicatcs that aboiit 10%
of al1 mutants are equiixient [DeMidIo 891. The eqiiidence problem is undeciclable and is donc
in practice by the tester making a decision based on looking at the cocle. If the mutant is jurigerf
ferror prone) to he non-equivalent then the tester niust create ii new test case to kiil it. Sonir aork
tins been clone on nutomated equidency checking with a succcss rate of about 50%. The miication
liternturc contains severai ideas for rediicing the prohibitive computntional costs of mutation tcsting.
The riext section discusses severai of them.
3.5.1 Selective mutation
The cost of esecuting mutants can be recfuceci by retfucing the nuniber of mutants t h nwcl to h ~ f
cseciiteci. Researchers have proposecl random mutation and seIective mucation'" [Offitt+ 937 as t~vo
techniclues in ahicli only a subset of the possible mutants are esecuted.
Acree [Acree 801 and Budd [Budd 801 evamined the idea of using O&- a s m d randomIy selectecl
subset of dl possibIe mutants generated in Mothra. Both experiments shoned chat a test set that
is adequate with respect to a randomiy seIected 10% of al1 mutants in Mothra has a mutation
score of over 99% n-ith respect to d mutants. However. randody sekcting mutants ignores the
fault cletecting capability of individual mutant types. Biidtl's M t cletection e-xperimcnts found
that mutants generated n<th respect to one mutant operacor may be more effective in detecting
certain 5-pes of fauIts chan mutants generated %<th respect to another operator. This suggests that
n-hile mutants are selected for esamination. they shodd be weighted dXerentIy dependiig on their
respective fault detection capability This has become knom as selective mutation.
'"This a h appears in the lirerature as constrained mutation.
Selective mutation origindly proposed by 4Iathur [Mat 911 is an approximation technique that
cries to select only mutants that are truly distinct from other mutants. The main advanrage of
this approach is that it does not requùe the generation of al1 mutants. The problem is in Gnding
a n effective smdl set of operators. X good selection should not o n - recluce the eseciition cost but
should d s o maintain good fault detection capabilit.
In one analpis O f i t et al. showed that test chta that kilis al1 mutants created by only 6 of the
5Iothra operators niIl usually kill almost ail of the mutants created by d l 22 of the original operators
[OMrti 961. LSing the obscnation that different mutant operators generate mutants a t clifferent
rates they decideci to perform e.qerinients in which the S most industrious operators were not iisec1.
This is called thii X-selective mutation. Tliey then comparecl selective to strong mutation. They
found that test sets that were 100% selective mutation acieqiiate were almost 100% strong niiitation
adequate (8 out of 10 test sets were 100% iitleqiiatc). The exact results are:
a 2-selective mutation yielded about 20% compiitational savings with 99.99% niirtation nricquacy
-1-seIective mutation yielded 40% compucational savings with 99.5% strong rnttcation arIeqitacy
r 6-seIective mutation yielded over 60% computationd savings with 99.2% strong mutation acl-
cquac.
This early data indicates that selective mutation can Save up to GO% of the computational cost airh
a negiigible loss in mutation acIequacy. To tinderstanrl why this occiirs the idea of fault size xi11
bc considereci in ttie next section. Both 4Iathiir [Mat 911 and Offut et al. [Offut+ 931 stiggmecl
ttie ttvo tnutacion operators to be oniitted are SVR (scalar variable replacement) .and XSR (an-
for scdar replacement). These operators generated the most mutants because SVR replaces evwy
scalar in a progarn with every other compatible scalar founci in the progarn. .UR replaces eve-
scdar variable in a program nith every array reference of corripatitile type. Adclitinnally they fotind
that the S%R and ASR operators produce mutants that are easily kiiietl by tests that kilt d e r
types of mutants.
3.6 A Notion of Fault Size
X mistake (not necessanly a coding error) made bj- a deveIoper m a - lead to one o r more fauits
located in the text of a program. A fauit may be an isolated mistake in one statement of a program
or it may be spread over several lines. One could Iook a t a fauit as being the dierence between an
incorrect prograrn and some correct program. -4 fault has both a syntactic .and semantic measure".
The following section is based on [Offut+ 961.
The syntactic nature of a fault can be described by the number of changes requiretl to correct ic.
-1 fault also has a semantic aspect: every program P can he viewetl as having a specification S thrit
clefines a sec D (the input domain). a sec R (the output range). and a mapping from D to R. The
program gives results basetl on a superset of D. If the input is not in D. P may protluce output
chat is not in R . The semantic characcerization of a fault vicws the faulty prograni as containing ri
computation that produces incorrect oiitput mec some siibset of the input clornain,
The size of a fault is the clifference between a correct and incorrect version of the progrmi. The
sjntactic size of a fault is the minimum number of statements or tokens that neecl to he changecl to
get a correct progratn. The semantic size of a Fault is the relative size of the sub-clomain of D for
ahich the output mapping is incorrect. hlthough the icleal use of sem'ancic size would be basetl on
a usage clistribution that assigns a non-uniform probability to each input. it can bc approximacecl
by a uniform distribution (each inpuc having equd probability). With respect to size the syntactic
and semantic characterizations of a fauit are very clifferent.
For a syritacticdly sn~all f d t . one ~oken or one statement m.?'- be incorrect. For n scnianticaIIy
small fault. Pb behaviour on a very smdl portion of D is incorrect. Clearly a small syntactic fiiitlc
cati protluce a large scrnaritic faiilt. Sirnilady a tnajor syntactic fault rnay affcct only a smnll portion
of the input doniain: a sniall sernantic fault. O l course a large syntactic fault c m result in a large
seniantic fault. Hoaever. the interesting class is ahere a srnaII syntactic fault procIuccs a srrial1
semantic fadt.
SIiitants created by mutation operators that produce Large semantic faults n-iU be e'q- to ki11.
[Budd+ 521 discussed the concept of program neighbourhoods. A neighbourhood of a progtam P is
a set of programs chat are semanticaliy -closeE to P. We aan t to use syntactic mutant aperntors
to produce mutants chat are dose ta P. since such sernanticaiiy small mutants shotdd be harder
to !dl and thus have the potentid to lead co higher qudity tests. SIoreover mutants that are
s m d syntactically but large sernanticaIly generate noise: the? add difficulty to the mutation process
without increasing the testing t d u c of the resdtiog test cases. This is sliown by the fact that aiany
mutants are trivialIy Wied by almost j - test case that reaches the mutated statement.
l l~here are other ways to define fault size, for a m p l e the consequence of a fault occurring.
In ternis of semantic fault size it appears chat selective mutation is ming the mutation operators
that tend to generate small semantic faults. Zi'nfortunately it foil~ws that such operacors niII aIso
produce a proportionally higher number of equivalent mutants as indicated in [Offut 461. i7iewcd
in terms of semantic fault size. hard to kiü mutants are mutants with very smal1 fault size vhiie
equitalent mutants have semantic fault size zero.
3.6.1 Weak Mutation
.As tiefinetl so far. rlistingriishing a mutant requires that the oiitpiit of the mutant differ From the
output of the original program. This requires nmning the mutant to completion (nsstiming oiitpiits
only occur on completion which nlthough untnre in generd is the consemtive xsumption iised in ~ h i s
section). Weak mutation is proposer1 as a way ro reduce the coniputadoniil cost of rii1it;itiori tcstiiig
(iioa referrecl to ris strong iiwtacion) by suggesting that a nriicruit be considetecl ciiitinguislieci if the
data state of the mutanc cliffers focni the data scate of tht. origind irnmediatdy nfter the esecrition
of the mutatetl stacement ~ o w r i e n $21. The iclea is chat whether or not the niiitant is n-eakly
killeri. execution of clle mutant can be terminated immediateiy after the data state conparison thtis
reciucing the computation tinie. With ttie introduction of weak niutation cesting, original niutation
testing became known as strong mutation. Weak mutation leads to somc intercsting observations
about wliat constituces a fault.
if a mutant is Mlecl in the strong sense then it will be killed in the veak sense bu the s.me
test (stroug ~nmation subsumes ive& mutation). However the opposite is not necessarily trile. a
niutation may alter an internai Stace. but subsequent execution mq- ignore. correct. or in some other
n.q mask the error state. For esample a mutation may have the effect of crttising zrn extra iteration
of a loop. Thii ail1 be detected under weak mutation. as the internai data states wiII rliffer. hnwever
the mutant may be equivalent. The foiiowing two C Fragments iLiustrate this case:
for (x=lx<lO;xf+) count = x*cost[xl:
r e t t m count:
for (s=Os<'>O:s++) counc = X*COS~[S]:
r e t m count:
Origind kërsion Mutant Version
C l e q both versions d yield the same result, hence ne have a ~ct ioaa i ly equiialent mutant.
-Uthough t h codd be cIassi6ed as a programmer e m r . in general Ke are not inte~ested in finding
this kind of "faultn". On the ocher hand embedding a simple print statement n i th i i the loop n-ïii
cause an extra line to be printed. In this case the mutant would be ciistinguished under strong
mutation testing (assiirning printed oiitput is compared).
Strong and weak mutation testing are evtreme ends of a continuum. in PVoodn-arcl+ 881 Wooci-
ward and Halewood introduce the idert of firm mutation as an extension to we,& mutation. Xith
weak mutation the ciata state is compared a t the end of the rnmatecl statement. nith strong rriii-
tacion the cornparison is made after program termination. Firm mutation compares the program's
data state (and possibly ruiy intemediate oiitputs) at some intermecliate time after eseciition of the
mitant. Cornparison poincs are user seIectable and muiilly corrie iiftcr the eseciition of a coniplete
progrnm block such as a for Ioop body. Corisider a mutation operator applieci to a stacement in ttir!
bot[! of a ioop where the loop is part of a proceclure calIed by the body of a progam. Then the data
Stace coulrl bc conipareci immediately after esecution of the rnutatecl statement (weak rniitation).
immediately after tennination of the loop. irnmediately after the rettirn frorn the procediirc. or after
program termination (strong mutation).
The original Slothra -stem did noc nui compiled versions of the mutants. Rather it usecl a
piirpose biiilt interpreter riinning on a slowi3 computer. Rit t i 'an interpreter weak mutation ti
rcIativeIy e a d y to perform. Programs can be nin to an arbitrary line. haited and the tinta stnte
examineci. This is also possible with a compiled progam howver it is not ,as easy ancl is firrther
complicated by code optirnizations. The real computation savings cornes by recliicing the riirinirig
tirne of a relütively slow interpreter. With a cornpiled prograrri running on liigh speecl conipriters
ninning tinic may become less important than compiling tirne and the possible benefits of wriik
miitation testing are diminished.
3.7 Summary
This chapter has discussed the process of performing mutation testing, its theory. strengths ancl
weakness. and some ideas for overcoming its deficiencies.
Mutation adequacy appears to be a strong test criteria in that mutation adequate test suites are
good a t reveaiing faidts in a program. Aithough computationallj+ expensive to perform. research
"In an extreme case the program could be sensitive ta the performance penalty of an extra loop iteration. in which case the tester is interested in this kind of problem.
L 3 Ç I o ~ by tadafs standard.
inclicates that by using weak niutation a large increase in efficiency can be obtainecl withoiit a large
sacrifice in the adeqiiacy of the test set. One of the objectives of this thesis is to decemine if the
liuman and compucation costs that are the major neaknesses of mutation test are as significant
when mutation testing is applied to the textual descriptions of finite state machine.
Chapter 4
Testing State Machine Based
Software
Ciiapter 2 stated ttiat the problcni of testing software couId be approached in a varietu of w a p . d i c h
tvere categorized as being usage bueci techniques. black box techniques. and white hox techniqttes.
These are general prrrpose rnethods. in the sense that tliey arc (lesigrictl to work for jiisc abolit a-
type of soltmre [Bernhard 941. There dso e?ust sotne spccidied testing eechniqites tnrgetetl at
jpecific typrui O€ systerris siich as ttiose based on h i t e state tnachines. This chapter tlisciisscs sorric
[if die craclitionai inethods usecl to test software specified by tinite state tmdiinti furrnaiisrris. It
concludes xitti a look at recent work in ivhich mutrrtiori testirig has Lieen propusecl iw a rnetiind of
testing HFS31 specifications.
4.1 Finite State Machines (FSWI)
F S K s rnodei systems chat can be described by stimuli and operations on state. Stimuli (ohen caiied
evencs) are inputs from the oucside n-orkd. wtiiie operations are the FSSiS responçes to the stimuliL".
FSM based systems are ohen cdled reactive or erent drit-en ystems. With Eunctionai software the
output of a compucation is a htnction of its current input. Qïth FSM bbased softme the output of
a computation is a function of both the current input and the current state a€ the -stem. where the
LJSome of the literature refers to stimuli as r e c e i d messages or in bound signais. ivhile resporises are olten cdlei generated messages or out bound signjls.
curen t state of the system depends on the historj- of the software's inputs. The input to a FSM is
a possibly infinite sequence of input events, each including a certain number (possibly zero) of input
parameters.
FSM software can fail in al1 of the ways that functional software cm. Additionaiiy F S X s c m fail
because of sequencing errors. in [Chow 781 sequencing m o r s are clivicied into three classes. Assume
X ancl A' are tno minimal15 FSbIs with the same input alphabet. A' is the correct version. The
error classes are:
1. Operation errors: -4 is said to have operation errors (also callecl output errors or outpiit fa~ilts).
if h is not eqiiivalent to -4' and A c m be modified to be equi~alent to A' bu changing the
output function of A (without adding or cleleting states in A). Output errors are relacively
ensy to cletect. hriy test suite based on a transition tour of A (trnversing cadi trimsition r>f -1
nt l e m once) is guaranteed to detect dl outpiit errors [Bach+ 941.
'1. Transfcr Errors: A is sait1 to have transfer errors if X is not cquivrilent to A' and X XI^ tie-
tnociifetf to be ec~uiwlent to A' by ch'wging oniy the nest state Function of X (withoiit acicling
or deleting states).
3. Extra states (missing states): A is said to have extra (missing) States if in order to nitde A
eq"\alent to A' the number of states in h must bc reducccl (increasecl). Since X and A' are
mininial. ati tinecpnl number of states implies that A and A' are not eqiiivalent.
By making some restrictions on the structure of a FSJI Chow provecl that it is possible to gerierate
test chta paranteed to cletect the above sequencing errors [Chon- 781. His method. noa calleil
the wrnethod, is both vdid and reliable in the sense of [Good+ 751 and provides an aigorichni for
generating a test suite. The method is based on resuIts of automata theory and is limited 6- the
foliowing assumptions:
I. the FSM is completely specified.
2. the FSM is minimal.
3. the FSM starts in 6xed initial state,
4. and every state is reachable [Chow 781.
lSA minimal Stace machine is a state machine whose behaviour coiild not he matched by another scate machine that h& a fewer number of states.
Chow's assurnptions restrict the method to deaiing strictly ~ 5 t h the possibIy abstract controI struc-
ture ofa FS41 design (without these assumptions the generai problem is undecicfeable [GilI 621). This
impks that there are no counters or control mriables manipulatetf in the operatians that intiueucc
the secpencing olaperations. Yote that these assurnptions do not in gcneral hold for more powerfril
moclcling cechniques basecl on estended FSMs. hierarchical FSlls. or Statechiirts.
Chow's rv-methocl although tdid and reliable c m require a large number of test sequences.
Researchers working in the field of protoc01 tcsting have developed other test derivation niethods
For FShls airned at reducing the size of the test suites. Among these niethociç. the T-method. D-
method. and L-rnethod. providc algorithms for gcnerating test suites that are tisndly sniailer t11;in
those produced by the w-method. Both the D and C method are guarantecrl to finrt nll oiitprtt nntl
transfer errors [Bernhard 94. However the T-method. although proclucing the srnailest test suites.
is orily gtiarnntced CO detect dl transfer errors. Cnfortiinately al1 three of these methods are limitetl
by the same asstiniptions as the wnicthotl. Although the resdting suites are smalIer than those
gcneraterl by the w-mcthod they can stil1 become large.
4.2 Protocol Testing
Generally spe'aking. protocols: specify the rides useri by a coilection of computer syscenis that arc!
cnrnmunicating on sorne network [Bernhard 941, The protocol (conformanceL~ testing problem is
the problem of decermining if an implementation correctly impkments a FSM specification. Con-
formance testing is based on the idea that many dierent implernentations wilI he tested against a
singIe specification. This is expected when an organization defines a standard (e.g. the OS1 protoco!
stack specification [IS7.5961) chat is used by ma- different irnplementers. Thuç the mettiods iised in
conformance testing are highly dependent on the esistence oEa precise specification [Bocht94]. Since
the cornpleteness of the FSJI Eest suite generation methods ciiscussed above is limiteci bu Chorrfs
assumptions and since comp1eu protocols can vioIate these assumptions other tescing methods for
protocols have been eqIored [Bocht !Ml. Among them are some fault-based methods.
As with hardware testing, the fadt-based testing of protocols is bxed on a fauit model that
defines di of the fauIts that are to be detected. Chow's sequencing errors are a nattud basis for
çuch a protocol ladt model. Each Fadt is rnodeIed by a mutant O€ the specification. A mutant is
t6Protacol testing concentrates o n testing €or confamance of a protocol to its specificarion. This is i Iittle diffecenr €rom testing to reveal raults.
a FSM obtained by applying the following four types of operators on the specification. They are
basecl on ChoK% sequencing errors [Petrenkof 961:
T'pe 1: alter the tail state of a transition (a transfer fatilt)
Type 2: alter the output of a transition (an output faiilt)
Type 3: atltl a transition: and
Type 4: atltl an cutra state or delete a state.
These operations model possible programmer mistakes macle during the trnnslation of an FSSI basecl
specification into a program. Fault-basetl identification of mutants is most naturd in the contcst of
cliagnostic testing where the goal is not only to detect erroneoiis behaviour. but also cleterniine hon-
the iniplemeritation coulcl be correctecl in orcler to make it coriform to the specification. To date
tliert! lias been no enipiricd support cletermining whether the assiimptions of this or ottier protocol
faiilt models are realistic [Boch+ 9-41.
4.3 EFSM and HFSNI
The common interpretation of '.state" for softn-are systems is that each state represents one rlistinct
set of al1 ~zlicl ~zltics of al1 program variables. Even with simple progams containing few primitive
mriables. this interpretntion leads to an unmanageably large numbcrs of states". One way to retluce
this complexity is to use an estended FSM (EFSB[) model. in which the specification of a system
module. represented as a F S X includes the foilowing additions:
O .\clditional state variabirs: States becorne abstract view of the object that tlefine its qualitative
hehaviour. mhile mriables pertain to quantitative aspects. In this case a change in a variable
d u e does trot rilways imply a state change [Selic 94.
O Both the input and output interactions mal* include parameters.
Transitions may have enabling predicates and transition actions.
- The enabling predicate (often called a p a r d ) must be satisfied for the transition to be
taken. It depends on the present state (FSill and evtended variables) and possibly the
input pairameters of the stimuli.
LÏFor rxarnple. a sirilgel6 bit integer has over 65000 states.
- The action defines the output producecl. including the effective oiitpiit parameters. antl
possibly sorne updates of the additionai state variables. This iç ciifferent from the simple
FSII mode1 in mhich outputs clo not have parameters antl do noc tipdate state nriables.
Figure 4.1: HFSX
.ki extension to the representation of EFSIls and FSJIs is to introduce hierarchy. Wcti a
liierarchical FSM (AFSM) .any FSM state can have sub-states. Figure -4.1 shows a HFSM ahich tias
3 top level staces (Sl. S3. S3). The top-level States S3 and S3 dso contains three stibstates (S2L.
STL. S23 in S2 and S31. S32 in S3). Assume the current statc is SI. LVhen transition t l is taken
the nelT top-level state becomes S2. Hoaever nithin state S2 siib-state S22 becomes the ciment
sub-state. if transition t3 is taken then regardless of the mhich siib-state of S2 that the machine
is in the machine ni11 move to top-level state 53 and within state S3 the current state n-ill be S31.
An HFSM cati tie converted to an ecpitdent FSM. Figure 4.2 sLotvs the -tiattened' version of the
HFSSI of Figure -4.1. Xote t h che flattened version required the addition of 11 transitions.
Rierarchicd. concurrenti" state machines, cailed Statecharts. nere first introduced .mcl system-
aticaily studied bu Hare1 to mode1 reactive systems parel 871. Hierarchy recluces risual complesïty
whereas concurrency increases the rnodeling poaer. EFSJL HFSM. and Stateciiwts al1 increase the
representational power of FSWs. Honwer this increase in representational power makes the tescing
activity more ciiicult. ,Mi of these representations can be viened as compact notations for FS-Yfs.
'JConcunency in Sratechms is represented by ";uidn stares. rvhich are nor used in the UML-RT/RosenT and thus are not covered in this rwrk.
Figure 4.2: "Flattenecl' HFSSI.
hlthough it is sometimes possible to unfold an EFSWs into a Hat FSMs (possibly for sottie rrsciiig
activity). it is seldom done in practice.
4.4 Test ing Extended hierarchical FSNI
If certain lirniting assurnptions are made about the forrn of the prerlicates nncl actions in EFSlIs.
thcn the anaiysis of the behaviour of the specification and systetnatic test selection remains decid-
able [HigaSJI. In this case one of the test derivacion methocls for FSXs (which guacantee cornplece
sequencing ercor fadt coverage) can be used to generate test sequences. Ln generaI Liowever. partic-
ulariy vhen actions may incliide Ioops, the question of deciding n-hich input parameters shouIt1 be
iised in order to force the selection of a particuiw transition beconies undecideable (similar to the
question of deciding the executability of a given branch of a program). Fu-ther. specifications .anci
designs based on EFSM/HE'SM models are seIdom minimal or complete. Since the test sequence
generacion methods that are idid and rdiable for FSM are neither vdïd nor reIiable when applied
to HFSMs other test criteria have been suggested. Offut has presented four test criteria chat can be
used in EFSM specification based testing [Offut+951:
1. Transition coverage: the test set musc satisfy every transition in the specification graph. This
is analogous CO branch coverage.
2. Fi111 preclicate coverage: for each predicate on each transition. the test set must include tests
chat cause each clause in the predicate to result in a pair of outcornes where the talue of
the predicate is directly correlated with the d u e of the clause This relies on clehitions Eiom
DolfSB [RTCA 921 and is sirnilar to .CIC/DC. This subsumes transition covernge.
3. Transition-pair coverage: for each pair of adjacent transitions in the spccification graph. tlic
test set musc traverse the pair of transitions in secluence. This is meant to check the interfaces
among States. This subsunies transition coverage but is meant CO test the software differencly.
4. Complete sequence criterion: the test set must contain tests that traverse .-meaningfiil se-
quences" of transitions on the specification graph. where these sequenees are diosen bu the
test engineer based on e.xperience. domain knowleclge. anri other hun1.m-basecl kno\vicctge.
Thesc criceria are btack-box in nature. clepending only on the specifications. buc are partly niorinteci
by notions of stnictural coverage.
4.5 Statechart Mutation Operators
Fabbri et ai. have studied mutation testing of both FSSI and Statechart basecl software speciiications
[Fabbri+ 981 [Fabbri+ 98bl. in [Fabbri+ 981 an operator set for FSLI hasecl specifications m a
deriwcI anci a cool calied Proteurn/FSMLS w u built. Their mutation operators are basecl on the
error classes deEned by Chom- and on heuristics (guesses) clevised. by the authors. about typicai
programmer miscakes made during the creation of FSMs from specifications. The operators. Listecl
in table 4.1. aIso guarantee other minima1 testing requirenients such as covering ail transicions.
Table 4.1: Proteum/FSM Mutation Operators
LgProtetirn stands for Program Testing wvith ~Iutation.
The authors state that an empirical study was to be condiicted whose piirpose woultl be to
e d u a t e the cost of applying mutation testing to FS& and to tietermine an essenciai rniitntion
operator set. however results have yet to be publishecl.
In [Fabbri+ SSb] the FSSI operators of table 4.1 were espancled to cover Statechart basecl spec-
itications. The new operator set is based on their earlier work aicl on the intrinsic featiires of
Stntccharts. The Statechart opcrators are dividetl into three cacegones:
1. FSM mutation operators. These are the sarne operators shown in table 4.1.
2. EFSM mutation operators. These operators are basecl on sonie of the mutation operacors for
the C language and for boolean expressions.
3. Statecharts-fcaturcs-bûsecf opcrators, These opcrators consicler intrinsic Statechart featiires
siicfi as history. broaclcasting and parallelism. The definition of these operators n-as also
inspirecl by C h ~ n - ~ s error classes. interpretetl accorduig to the Statecharts concepts.
2. 1 arc missing 3. 1 event miçsing 4. 1 event extra
S. 1 output exchanged
.5. 6. 7.
9. 1 state missing EFSSI operator set
1. 1 ex~reçsion deletion
event exchangecl destination eschanged output misshg
2. 1 t3oole.n expression negsrion 3. term associrrtivity shift 4. 1 arithrnetic operator by arïthmetic operator 5. / relational operator by relational operator
logical operator by logical operator
9. 1 variable by constant replacement 10. 1 constant by requiretl constant rcplaccment Il. 1 constant bv scaIar variable rwlacernent
Statecharts feature based opcrator set 1 transition's history tIeIeted
transition witli Liistory bu transition replacenierit history missing h by h* replacement tl* by h replacement 11 extra li* extra in(s) conclit ion missing in(s) condition state replacement not yet(e) condition niissing not yet(e) conditiori event replacerrierit exit(s) event missing exît(s) event state replacement entered(s) event missing entered(s)event state replacement broadcasting origin transit ion replacement broadcasting destination transition replacement
Table 4.2: Proteum/ST Mutation Operators
-4s mith the Proteum/FSSI workan operator set %as detined and a tool prodiiced (Proteum/ST).
Table -12 lists the Proteum/ST operators. &?th this tool mutations are applied to textiiai repre-
sentations of Statecharts m-ritten in SSL langage (Statecharts Speciiïcation Langage). As in the
Proteum/FSM case no empirical results are yet a d a b l e .
4.6 Summary
The traclicional mechods of testing FSSI presentetl in this chapter were for the most part clesised
a i t h the intention of testing the correctness (conformance) of the specification to implemencation
translation performed by programrners. Testing a RoseRT generated program is somewhat different
in that RoseRT is iffied to develop a specification and then to autoniaticaiiy generate the source
code for the application. Effectively the specification becomes the implementation. This diapter
11;s presented material that is useci as the basis of performing mutation testing of RoseRT HFSMs.
However before clisciissing mutation testing of RoseRT progranis it is n e c e s s q co c,Ae n more
dctailecl look aE the RoseRT tool, This is the subject of Chapter 5.
Chapter 5
The RoseRT Toolset
This chapter gives a brief introduction to the RoseRT toolset. [t then hriefly cliscitsses bot11 the
stniccural and betiaviotiral motleling of active objects as implemented bu the toolset. liiitation
testing of active objects requires a detailed understanding of how HFSMs are iised to define ru1
accive object's hehaviour. This cliaptcr provitles those cletails.
5.1 Overview
RoseRT is ii rnodeling tool riesigned for specifying, visualizing, tloçur~ientùig. aritl aiiturri;rting th!
construction of coniplcx. evcnt-driven. and potentially clistribirtecl r d - t i m e systems [Lyons 981. tt
is nlso an exploration tool in chat it c ~ m be iised to explore the design o l a project. Tiie rnocleüng
langiiage iisecl iri RoseRT is bascd on Lnified Siotleling Language (LXL) frir R e d Tmc CSIL-RT)
and the R 0 0 5 1 (Reai-ïime Object-Oriented Blodeling langage, originaiiy developed by O b j e c T i e
[SeIicf 941). One can rnake a distinction between two kinds of reaI time systems: time-&ken anri
event-<iriven. in time-driven systems events and outputs occur a t periodic rates b a e d on reaI-time
clocks. In event-driven r e d - t h e systems an output is a response to an event that must occrrr wichin
a specified bounded time. RoseRT is targeted a t real time systems of the event-*en End- RoseRT
appiications are constmctetl nfth the aid of class. object. sequence. collaboration, Stace and ocher
diagrams. most of thern based on L . L .
RoseRT supports the iterative development of r e d - t h e systems at hro Ievels of modehg. At
the schematic level. ciesigns are expressed in the C'hIL-RT visilal langiiage (formerly R0051). At the
tletail level C or Ci-+ source statements are addecl at various points in the model. For example each
state transition can have an associated code segment (siibsequently cailed an action) that is esecutetl
ahenever the transition is taken. Similarly. states can have associated entry and exit actions that
are executed when the state is first entered (entry action) and ahcn the state is eventually left (exit
action).
-1 RoseRT mode1 is esecutable: it is possible to compile. run. and clebug the mode1 on the host
rlevdoprrient machine. L-sin:: the obsenability ~ p t i o n ' ~ of the toolset. it is possible to graptiically
observe the rtiotlel 'as it nioves througli states. hlthough RoseRT is primarily a modeiing and
sinruiation tool. it Li possible to generate a complete and deliverable application strictly tising the
toolset.
Figure 5.1 shows the laycrs involvetl in runtiing a RoseRT modcl. At the top is the RuscRT
mode1 consisting of Lioth schematic and detail level elenients. The Services library is a Iraniework
siippliecl by RoseRT. It is siniiiar to a virtiid niachitic and provides the following services:
O 11echanisms that support the iniplementntion of rlynarnically creatctl concurrent comm~inicat-
ing state machines.
O Thread management and concurrency control.
O Timing,
O inter-thread 'and inter-process communication.
O ObsertabiIity and debugging of a running model.
At compile t h e C f + code is generated. based on the model. then cornpiled. The compiled object
code is then linked with the RTS iibrary to produce an esecutable program. L-nder W i d o m L T
this is an .ese file. Other formats are possible and depend on the target real-time operating syscem
(siich as VsWorks or Q 3 ï ) .
"'Oboermbility is carriecf out by using an expandeci version of the RoseEtT services l i b r i q in n-hich communication betwveen the taolset and the wecuting application is performrd via a TCP\iP connection.
I Rose RealTi nie Model !
LI Rame ULo<Ju Timing U F [ _ l State / 1 Madiine 1
Servi ces Li brary
Figure .5.1: RoseRT Mode1 Environment.
5.2 Structure
The structure of a L..iL-RT mode1 iç clescribecl in ternis of Capsules. Ports. and Connectors. Tliese
arc extensions added to the base CML Inngtage through its scereocype rnechnnism. F i g m 3.2 sho~vs
a class diagram for a simple LXL-RT system wtiilc figure 5.2 shows ;L stnicture cliagrani containing
these three core stntctirral cornponents. Base L X L elements such as cIasses c m also be usecl in n
LXL-RT design (DataCIass in figure . 5 2 ) .
Capsules are an extension to a Gl lL class. The- are highly encapsulated active objects. possibly
n-ith rnethods and data. Each capsule c m be considered a logicaiiy concurrent process chat interacts
with the environment through one or more b o u n d q objects called ports. Capsules have no public
methods or data- MI manipulation of a capsule's data (which is protected or prixtte) occius as a
response to messages arriving on the capsule's ports. Collaboration diagrams are used to describe
the structurai decomposition of a Capsuie class. The capsde .'SomeCapsulë in figure 5.2 has 1 chta
actribute (of type -DataCIass"). one method ("CaldateTieToGo'}, and three ports. Ports are
aiso inclicated by an icon in the lowest cornpartinent of the capsule box. The Log and T ' d g ports
are supplied by the services library. A log port provides the abiity to m i c e to the standard output
device while the timing port d o m a capsule to request timeout e\-ents. The -CamniandCorunis-
R + C l o g : Log a+ I NewPcRf : Erning .+ I CommandComnis: ComrnandProtoçol
W ~ P ~ O ~ O C O ~ S
CommandProtocol
+aStaItUp O cast : int = 100
--
Figure .5.1: RoseRT CIass Diagram
protocol role is a port objeçt that instautiates the interface clefinecl by the ~~CornrnanclProtoct~l"
protocol. As shown in figure 5.3. capsules can dso contain other (sub) capsiiles as part of tlieir
structure. Such nestirig can Lie carriecl to a r b i t r a l depth.
A port is a physical part of the implernentation of a capsuie that mediates the interaction of
the capsule with the outside world-it is ;ui object that implernents a specific interface [Lyons 9SI.
Ports reaIize protocols. which define the Row of information (sibgds) between connecteci ports of
capsuless. Ports dIom capsules to export multiple different intedaces: each possibly tIesignet1 for a
specific roIe. Figure .5.2 contains a CommandProtocol protocol which SomeCapsuIe fias instantiated
as its CommandComms port. The class diagram for the CommandProtocoI shows the signals that it
clefines. There are four -in' signals (arrows entering the box icon) and 1 out signal (arrows leaving
the bos icon). Ports are either conjugated or not. For a conjugated port the in signais become out
signais. while the o.ut signals become in signais. Two ports are compatible ruid c m be çonnected
together when the in signals of one port are the out signais of the other.
Connectors are used to join the ports ~f coiiaborating capsules. They represent the channels
through which messages (signais) Bow. Conneccious c m be made staticaiiy a t design time or cly-
Figure 3.3: Structure Diagram for ÇorneCiipsiile
naniically at riln tirrie. -4 RoseRT nioclel consists of a netaork of collaborating capsules (ancl classcs)
communicating wÎth each other strict- through messages sent nnd received on their ports. Figure 5.3
shows a single connector which represents a connection becween a containirig capsde's Cornmiuirl-
Cornms Port and a contained capsule's SewPortl port.
5.3 Behaviour
The behwiorir of a simple capsule is defined by a capsrile's RFSN (CSM). CSMs .are basetl on
ROOM Charts [Selicf 941. ahich in turn were based on Hareik Statecharts parel 871. The niain
difference berneen Harel's Statecharcs and capsule state machines is that capsule state t~iiicliirie~ ilo
not include and stace2'. X CSSI specifies the behaviour of a capside in teaction to cvents receivctl
frorn its ports. In sipals deline the stimuli that cause transitions in the CSSI tu be taken. Out
signals define the responses that cm emitted by the capsule. In the context of capsules the term
signal is sjnonymoris ~ i t t i event and message.
Complex capsules contain an internai network of coiiaborating subcapsules joined by connec-
tors. These subcapsuIes are capsdes in their ovn right. each n-ith its own optionai CSM. and c m
themselves be decornposed into subcapsules. This type of decomposition c m be carriecl to ahat-
ever depth Ïs n e c e s s a aüowing the modeling of arbitrariiy cornplex hierarchicd stnxctures and
behaviour,
21Concurrency in Stiitecharts is modeleci by and States. ROObt md UhIL-Kï rnodel concurrency via capsules.
5.4 Transit ions and Triggers
Wien a capsule is rcsting in a state. it is ready to proces events. .In event (&O cded a message
or signai) arriring at one of the capsde's ports ma!* trigger a transition. in the RoscRT toolset
twents rn- be assignai one of 5 priorities. An event wiI1 on1y trigger a transition if it satisfies the
triger specification of any of the outgÔing transitions attached to the ciment state or any of its
parent states. If multiple transitions are enabIed then a scoping ruIe is iiçed to rIecidc which nf the
canclitIate transitions d i be taken (when a transition is caken it consumes the cvenc: therefore n
single eveuc triggers a single transition). The sc~ping rule specifics chat transitions be t.vnluatec1 iit
the lowst possible ~ubstate first. if none of these transitions are enabled for the ciirrcnt ment then
die triuil;itioris of che subs~ate's itnrirediace pareiit arc evaluarcd. IF norie of ttiese are etiabletl tiieu
t!vdiiation concitiues uuti1 the top level state is reached. IF none of its trarisiciotis are enahlecl tken
clip ewnt il; cliucar(lerl and there is no effect on the CSM.
A transition iwuaiiy takes the CSBI fiom one state to another. Hon-cvcr transitions can s t m
and enci on the same stilte. Each transition may have an action abject associatecl nit11 it. An
action object is a C or Ci+ code bIock similar to the body OF n c las methoci (chiring the code
generation process. actions are transIated into cIass methods). -411 signais sent by a CShI (Le. the
statc machine's outputs) must be codcd in the rietai1 Ievd in an action object. X transition code
segnietit tia access to ail uf its capsule's vxiables (n-hich are the CShI's estendecl variables) and
any ai~totriatic mriabIes instmtiaterl in the transition's code block. .4dditionally the niessage that
caused the transicion is stored in a predefined extencled state m-iabIe (a void pointer rianiecl rtdatn)
so chat it can be acçessed by the transition code ifriecessary (for ex'uariiple ic cari be tticd to ilcttw~iiric
the source of the ruessage and to recover the parameters. if a- p a s 4 in ttie message).
Figure -5.4 depicts a tien- of a HFSM in which composite scates are shom dong n-ith tbeir
decompositions (trie toolset does not show KFSMs this say: each composite state is shon-n in its
o m window). Whcn a capsule is created. it is in the TOP state. ï h e TOP state cannot have sibling
states: it is the root of the state machine hierarchy. h e d i a t e l y foiiowing instantiacion a capsule
receives a meation" went that tnggers an initia1 transition. In figure 5.4 t h k is the transition
iabeled -Tln. The initid transition is the start of a new concurrent thread of execution. From n
design point of viexv al1 capsdes c m be treated as concurrent objects each running on their onn
Iogical thread. in Iater stages of dewloprnent logical ttrreads can be assigned to separate physicai
threads or d t e m a t i d y grouped into a smder number of physicd threads (for efficiency remns).
Figure 5.4: CS11
Tlie initiai transition is triggered by the "creation" event. With the excepcion of thc initial
transition al1 other activity is triggered by the arrital of events at one of the interface coniponencs
(ri port). -4 simple crigger spccification consists of thrce parts:
The name of the triggering signal of the event chat causes the transition.
The name of ttie port on tvhich the event is espesteci to arrive,
An optiond giiard condition. -4 giard is a Boole,m expression that is evaliinteti rlynarnicaIIy
when the event is scheduletl for processing. ft is entercd irito the tooiset i is ;i C i - rettmr
statement. If thc gciard ecdiiates to TRUE the crarisition is taken. r)cliertvise t h rwrit is
ignored. if no grard code is present the giartl rlefairlts to TRLT.
X single transition cari have more than 1 trigger. The semantics of these cornpound tr isers are
sirch that satisfying any one of the simple triggers d l cause the transition to occur. Final- there
is a universal trigger chat specifies a shorthand way of sa-yÏng that any incoming signa1 event ni11
cause the transition. h universal trigger may aiso have a guard associated n-itii it.
In the toolset each Ievel of the HFSM is shonn separately. each composite state in a leveI is then
displayable as a state machine in its onn windon-. This is camed on to arbitraq- depth. Each state
c m have a code segment associated nith state entry and state exit. For eYLLrnp1e ahen transition
TI in figure 5.4 &es state SI is entered. This causes the entry code for S I to be euecuted. SimiIarLy
vhen transition T2 is triggered? the exit code for SI wi l i be euecuted. Transitions t int nit across
multiple States are partitioned into mdtiple transition semgmem. each of which may have it's on-n
code block.
5.5 Active Object Issues
5.5.1 Priority Inversion
-1 capsule is an active object &ose behaviour is detined by a HFShI. Cspsiiles eshibit riin to
completion sernantics. This means that once a transition chain h a been s tmed another transition
c h i n for chat capsule d l noc be starter1 imtil the current one is coniplete. 1 transition chain is
coniprisecl of al1 of thc actions tiiken in response to the arrivai of a single event. In figure 5.-L when
an event triggers the transition T2 a transition chah consisting of four actions is initiated. Firçt the
exit action for state S1 is executed. folIowed bu the action code for transition T L transition T3. and
findly by the entry code for state 521. Once this chain h a been startecl it cannot be internipcecl
by the a m d of a new evcnt at one of the capsrrlc's ports. This may lead to priority inversion in
the case where a high priority message arrives whik a capsuIe is in the rniddle of a transitioii ctiaiti
triggerecl by a Lower priririty et.cnt. The benefit of nia to completion sernantics is that within a
capsiilc there is no concrirrency. This avoids having to protect a capsule's data structures against
race conditions.
5.5.2 Spontaneous Events
Passive objects modeled by FSBls onIy reacc to stimuli from the environnient. in addition to this
hehai-ioiir. active objects modeled by HFSJI. such as RoseRT capsules. m q initiate actions. in-
clitding self-actions. in RoseRT motieis che latter causes apparently spontaneous behaviour by a
capsule which is iisudly the resuit of that capsuieS t h e r e-upiring. and less ohen due to some other
intemipt. e.g. an device hidden by the capsule. Obvîously the addition of spontaneous interna1
events adds to the compIexity of adequately testing a capsule.
5.6 Summary
This chapter has presented a brÏeF overcie~ of RoseRT motleIs in terrns of their structure <ancl
hehmioiir. It has also introduced the idea of active objects. A thorough discussion of a cool as
compler as RoseRT is beyond the scope of this chesiç. X more detailed description of RoseRT statc
rnactiine behaviour is prcsentecl in .Appendi. D.
C hapt er 6
Mutation Testing of RoseRT
Models
Ctiapters 2.3. -1. and 5 presenced ctie backg~und information necessary to consider mutation cestirig
of RoseRT CS& This Chapter proposes and discusses an initial set of 16 rriutation opcrntors for
cssts.
6.1 Mutation Operators For RoseRT
The work by Fabbri cc al. is concernecf wich mutation testing of FSM and Statechart specificntions.
However. with RoseRT the mode1 entered into the toolset is more chan a specEcation. With the
addition of detail level code and the code generation facilities of the toolset the specification becomes
the implementacion.
In the n e z section mutation operators that operate miinly on the schematic levei elemencs of
RoseRT CSXs are defiaed. Schematic eietnents are those elements that are presented graphicdlj-
b - the tooiset. They roughiy correspond CO elements of the state machine contror structure (Le.
States. transitions. guards. stimuli. and responses). A t the detail Ievel any mutation operator that
is reasonable to use on C or Ctt cm be applied to a RoseRï Model. Hoivever this thesis o d y
looked at schematic level mutation spedic to the b?uiL-EU? language as impiemenced in che Rose
Real-Time tooiset. Therefore dthougb entire code blacks such as transition actions zrc mutated
(cleletecl or exchanged), no mutations are applied a t the source code token level of the detail code.
Further no mutations of structure diagrams. protocols. or optionai capsules are performed. Whereas
some operators (Fortran and C) are partially based on studies of typical programmer errnrs. ~hese
RoseRT operators are not (no studies are arailabIe). instead the RoseRT operators are baset[ on:
a Chow's error classes.
a Proteimi/FShi/ST operators.
a uperators useil in tradicional progranunhg Ianguage mutation syxemi-'.
- niainly iriterested in coverage operators
intrinsic featrires ancl weahcsses of the RoscRT toolset that can Icad to rnistakes.
a discussions nith iiscrs of thc tool:
- stiiclent uscrs of the cool (3)
- An R\iC faculty member who taught the cool as part of two offerings of a course. He
WLS able to provide information on obsenations of common sttitient errors.
- in the cnd these ciiscussions tfirl not influericc the choice of operators presented in this
thesis.
a personal experience in using the tooI over an 18 month period for courses nncl other projects
O The itlea chat mutant operators shouId mimic semmticall~ srnail faults as disctrssecl in chapter
3. Operators chat result in semanticaiiy large faults aiIl be of little use'" as such errors n+out(l.
in practice. be foimd during development.
Althoiigh studeut users arc signüicantly cfifferent than professioual users of a- tool, the rcarcity
of both professionaiiy cleveloped models and professiond cievelopers meant that only t a o of the
capsdes used in this thesis were professional1- developed. This is acceptable in the contcst of this
thesis. ,as the main intention is to obtain some preliminary resuIts n-hile evduating the feasibity
of implementing mutation cesting wïth RoseRT. Future mrk. if deemed worthwhile. ivouid require
information on errors committed by experienced developers and access to RoseRT modek developed
by them.
'-Specificdly Fortan, C, and Ma83 operaton. "In fact they wouId be detrimentai to mutation testing as they would increase the cornputationd burden on the
mutation systern.
6.2 A Proposed RoseRT CSM Mutation Operator Set
The operator set proposed in this tliesis is dividecl inco three catgories: FSN operators. covcragc
operators. and RoseRT incrinsic operators. %te ~hat ci& is not a rigd classification as sonie
operators could qualify for rnembcrship in two of thc classes. Wiereas mutation testing researchers
often imply that mutation operators mimic programmer errors no such daim iç made here. Sonie of
~hese operators are basecl on C h a t s secpencing errors. sonie are designed to obcain cuverage. soine
are intended ta siniu1at.e programmer mistakes. and some are clesigpci rio simulate subtle rnistakes
iinintentionaliy encuurnged by the RriscRT tooisct.
6.2.1 FShl Operators
The definition of mutation operators for this category &-as baseci on the error classes i ~ s i x l in Pru-
teum/FSS[ ~ h i c h in turn are hasecl an Choa's error classes arid on tieuristics about cypiçnl titrcirs
iriarie by prograinniers rful-ing chc crciltion of FSXls.
1, TRD: Transicion Deletc. Detete each transition in the CSM. This operator produccoi I miicmt
for eve- transition in the rnodel.
1. TTG: Transition Trigger exchangecl. Replace a transition's trigger n-ith cve. nthcr pnssibk
trigger. The RoseRT literature uses the tenu triggcr to refer to the stirnuhi~ everit r:;itrsing a
transition to be taken. Consider a CS11 with T s i rnpFi triggeretl transitions tuid E possible
triagering ec-ents. This operator ni11 produce on the orcler of (E-1) ' T tiiutarits.
3. TAT: Transition Alter Txget. Alter the target stace of a transition. Siïth a CS11 the target
mets are: of a transition is defined in tcrms of the transition's source state. The possible ta,
O The parent of trie source state.
O Anp sibling of the source state.
O The source state itseif. This is cded a transition CO self in the RoseM Iiteratirre.
Ttie TAT operator changes the target state of a transition to be each of the possible alternatives.
It is impossible to give a formuIa for the nurnber of TAT mutants generated for an a r h i t r q
capsule. The number of mutants depends on the hierarchical structure of the capside.
"A simpIy triggered transition responds to a single event. X cornpiex tnggered transition responds to muhipie diEerent eveots.
One of the Proteurn errors is wrong dort state- RoseRT models do not have a start state per
se. rather there is an initial transition that originates From a pseud+state. With RoseRT the
wrong stort state error is performed using the T-IT operator to chauge the target state of the
initial transition to each of the originai target's sibling states and to the parent if the parent
is a substate with an exiting transition.
4. ADT: Action Delete Transition. For every transition with an action. delete the action. This
operator protluces 1 mutant for every transition in the CSSI that has an action associatetl a i t h
it.
.5. ADE: Action Delete state Entry. For every state in the CShI with an entry action. ticlete the
action. This operator produces 1 mutant for every state in the CSM chat has an entry action
associateci with it.
6 . ADX: Action Delete state exit. For every state in the CSSI rvitli an esit action. clclccc the
action. This operator protiuces 1 mutant [or every state in the CSM that has an exit action
nssocirited with it. The ADT. XDE, and AD-Y operators correspond to the o n t p t missirig
error or Table 4.1 and are riesigned ro End action (output) errors.
7. TCA: Transition Chauge Action. For every transition action objecc in the CShI replace the
action aith every other trarrsition action in the CSSI. In this thesis each action in che CSM
is assumed to be unique. This assumption hoIds in the rnajority of cases. however there coultl
be a few exceptions. This asstmption 1s macle in order CO sirnpli- the iniplementation of
this operator. The cost of iising this tvrong assuniption is chat more mutants rire generntcti.
Liowever thcse Fe%- excra mutants are copies of t'xisting mutants and rio not neetl cstrrr test
sequences to be killed.
S. SDL: State Delete. Delete each state in the CSM. This operator produces 1 niiitant for ewry
state. Deleting a state in a HFSM also deletes aIL of that states sub-states and al1 of the
transitions to and from the deleted state. [ntuitively thii will ofken be a large sem,antic error.
6.2.2 Coverage Operators
These operators are inspired by the Mothra trap operator in which each statement of a Fortran
program is replaceci with a Trap c d terminating the program. in 4Iothra this enforces statement
coverage. With RoseRT these operators are based on the belief that di transitions should be
esecutetf at l e s t once, al1 states shoultl be reachable. and that there shodd be no terminal scates
in the niodeI. The necessitu of guaranteeing no terminal states is application dependent. Although
il continuoiisIy running reactive system uually would riot have terminal states. the existence of a
terminal state is not always an error. However it is probably tvorthtvhile to identify 'an- terminal
states in the software then clecide whether they are intentional.
1. CVT: CoVerage Transition. For every transition ir i the CS31 change the tra.~sition's (pcissibIy
empty) action to output a terminate message. iiilling al1 of thcse mutants ensures transition
roveragc. This operntor protliices 1 mut'mt for eveq transition in the CSSI.
2. CSE: Coverage State Entry. For evecy state in the CSSI change the state's (possibk empty)
cnt- action to output a terminate message. Killing ail of these mutants ensitres ttiat al1 States
in the CSSI are reachable. This operator produccs 1 mutant for evc- state in the CSSI.
3 . CSX: Coïerage State Exit. For evecy state in the CS>[ change the state's (possibIy ~rnpty)
action to outpiit a terminate message. Iiilling al1 of thcse mutants ensures that there are no
rwrninnl statps in the CSM. This operator prodiir~s L miitant for evecy transition in the CSM.
6.2.3 U-vIL-RT/RoseRT Features based Operator Set
TIiese operators adtlress the intrinsic features of CSIL-RT <anci the RoseRT todset. Sote that sanie
of the operators listed in the other classes (such as CSE ancl CSS) ancl coiilrI be incliitied here
tion.e\.er tEey fit more naturally into othcr classifications.
1. TGT: Transitiori Guard True: For every guarded transition in the CSJI. change the guarcl ta
true (i-e. the transition is always enabled). One could also change each guaril to aiways return
falçe. however for guarded trarisitions. this has the same effect as deleting the transition ahich
is perfomed by the TRD operator. This operator generates one mutant for every ginrded
transition in the CSM.
2. TCP: Transition Choice Point: For each choice point in the CSM. change the expression
controiiing the choice to dm-ays be True and always be false. Choice points alivqs emiuate
a booIean expression in order to seIect one of m o possible outgoing transitions. Boolean
expressions are entered as Cf+ code. This is the only operator considered in this thesis
in ahich detail Ievel code is mutated. However the mutation is simple in Ehat source code
tokens are not mutated. rather the choice point's code segment is completely replaced by the
statement retuni TRUE: or retum FI4LSE:. This operator generates two mutants for each
choice point in the CSII.
3. XCP: exchange Choice Point Transitions: For every choice point in the CSAI eschange the
triie and false transitions. This lias the effect of negating the boolean expression controlling the
choice point. This operator generates one mutant for every choice point in the CSAI. Althoiigh
this appears to he a cletail level mutant the operator is actually applied a t the schematic level.
4. XTI: eSternal To Internai: For c iwy externd self transition in the CS11 change it CO an
interna1 transition to seIf. Transitions to self haw an intcrnal-eutcrnal Hag associatecl n-ith
them. ff esternal is selected then any scate esit actions within scope are eseciitecl. followeti
by the internai transition's action. foilotvett by any state etitry actions within scopc. if the
internal flag is selecterl then only the internai cransition'ç action is euccurcri.
$3. ITX: Intemal To eXternai: For every incemal cransition to self in the CS51 charige it to ;in
ctsternal transition to self. This is the cornpienient of STI. 00th the STI and ITS opcracors
rriirriic subtle rnistakes niade bu iisers of the toolset. The toolset encourages this niistdx iri
tliat seleçtirig bctwetln internal or esternd is accornplishecl by selecting a check box on one
pane of a thrce pane dialog box. It is easy for a developer to miss or neglect ttiis sritting.
if none of the states n-ithin the scope OF the transition have entry or exit actions then the
internai-estemal selection cloes not atfect the behariour of the CSM.
6.3 Summary
These operators can also be divided into two categories depending on the number of mutants they
generate. Linear operators generate mutants in ünear proportion to the number of states anci
transitions in a CSLI. For example the TRD operacor is Iinear. It produces L mutant for ench
transition in the CSbI. Yon-liiear operators generate m u t a m at a geater rate than the linear
operators. For example the TTG operator generates mutants in the orcler of T f h > . where T is the
number of transitions in a CSM and tr ig is the number of possible triggering signais ( in sibaals)
that cari trigger a transition in a CS3I.
There are possibly an infinite number of 1~x4s CO mutate a program whether it is state hasecl
or not. The operators proposed in thiç chapter do not constitute an eshaustive set rather they are
exploratory and intended strictly as an initial investigation. The nevt chapter discusses hon- these
mutation operators arc appiied to the teutuai representation of CSMs.
C hapt er 7
A Mutation Testing System for
RoseRT [MTS-RT] Using TXL
This chapter begins with an overview of the entire mutation testing system for RoseRT [MTS-RT]
chat as implemented as part of this thesis. The rest of this chapter is concerned witti explaininj;
what TSL is and how it was useù CO apply mutation opcracors CO RoseRT niodcls.
7.1 Overview of the Implementation
The developnient of tooIs to support anything but explaratory mutation testing is mandaton since
the number of mutants quickiy becomes large for even small program. The tools deveioped for
this thesis correspond to the qpical mutation testing activicies of mutant generation. test case (test
sequence) creation. program execution. and resuit analysis. Testing rvas perfomed at the capsule
Iex.el. i.e. each test session was targeted at testing the state machine of a single capsule. The correct
operation of the SITS-RT is dependent upon the diiectory structure s i d a r to the one shonm in
[igue 7.1. The actuai directory structure is defined in an initialization 6ie. The basic workfiow of the
system is shonn in tigure 7.2. Several intermediate steps have not been shown in order to Ïmprove
cIxity. inputs to the system are the RoseRT mode[ and an initial populatiou of test sequences-
The initial test sequences rnay be baed on transition tours. snz-itch covers. or any ather heuristic.
In practice, home~er. they usualIy corne Erom the tester is boking at the speciücation or the
capsule to be tested. Subsequent test sequences are gneratetl in order to kill the remaining live
mutants. The overdl proceçs can be broken into three phases. in the first phase mutants are creaced.
in the second phase test cases are created in the thirrl phase the mutants esecrited. and che results
'his chapter describes the fint phase. Chapter S describes the remainder.
Figure 7.1: Directon. Stnictrire.
7.2 Performing Mutations
h11itation operators were applied to the textual representations O€ RoseRT capsules tising TSL (to hr?
described later). Rose RT modeIs are saved in plaintevt disk files callecl petd files (.rtmciI estension).
An esample RoseRT capsule file is includecl a s Appendiu B. These Files .are formatteci as a linear
sequence of sections. Each section describes one of the major eiements of the mode1 sitch irs a
capsule or protocol. Petal files are writcen in accordance with a Rationd p r o p n e t - context free
grarnmar (CFG) that c m be described n-ith BNF or TIL. B3T' Grammars are adable for the
earlier ObjecTime R00M models from which RoseRT is derived. however no RoseRT gramman
werc made avaïiable by Rational. X working T'CL granimar wns developed for the RoseRT capsule
section through the e~amïnatiou of severd empIes , The complete RoseRT capsule gamniar as
tieveloped for thk thesis is inchdeci as Appendiu C.
P h n n Mutahm 1 MTS-RT
Mutant Files P est Sequeme P Note th& the onginai capsuie ir nui through the system just like an^ athér
'. \
\ \
~nfm&hfmmthe resuitr is fedback &O the test scquence 3neaJ\ation pmcess in ohs,to kiii remaUruig L w *tant5
w 1 4
_ Rwdt Files R e d Cornpaison Mutatisaa Score
- RT-SuperDa
Figure 7.2: RoseRT SIutation System:Esseritial \Vorkffows.
7.2.1 TXL
TSL (Tree ïransforrn Language) is a generd piirpose programming Ianguage clesigned to support
transformationai programming [Cordu+ UDI. TSL is useful for transforming an input. describeri by
a CFG. into an output. described by the s m e or possibly clifferent CFC;. X TSL progarn consists
of a collectiori of one or more rules that describe by c-ample how tto transforrn the input inco EIIC
output. TSL prograrns can aIso be used to extract various pieces of information frorn riny source
file describable by a CFG.
A TSL program operates in three phases- F h t the entire input is tokenized and pused iri
accordance with a supplied CFG. The output of this phase is a parse tree. in the second phase
transformation niles are applied to the parse tree to produce a new (transformed) parse tree which
m q confom to a different grammar. -Us0 during this phase intermediate files can be nritten in
order to extract interesting data. The finai phase unparses the transformed tree by perforrning an
inorder d k of the transformed passe tree and mithg the Leaves to produce the output [Cordyf
001.
7.2.2 TXL Grammar
T-YL g r m a r s are specified in a uocation simiIar to Eutendeci Backus-Nauer For~ii (B3T) [ C u r d ~ t
001. The basic unit of a TSL grarnmar is the define statement. h d e h e statemerit specifies an
ordered set of alternative forms for one non-terminal type. corresponding roughly to the set of
prodrrctions for n single non-terminal in a BXF pamrnar. In deîïnitions. literds and user definecl
keywortls mrist he prefked with an apostrophe in order to resolve conflicts with TSL's keyn-arcls
and prerlefined non- cerniinals. Each alternative Form of a definition is specified as a secluence oE
~erminai syrnbois and non-terminais. :\Iternatiws are scparatccl bu rhc vertical bar synhal. Non-
cerminaIs appearing in the body of a definition are encIoserl in scjuarc hrackets [ ]. For exaniplc one
definition for a state maciiinc transition could be:
def ine transition ' ( 'object 'Trans [stringlit]
'source [stringlit] ' target [stringlit] [opt aaioa-objectf [opt event-guards-List]
'1 end def ine
This definicion would be referenccd in other definicions or TSL rides as the non-tcrtni~ial sytiibol
[transition]. The follun-ing tes% corresponds to a transitioa object taken from a RoseRT capsoie that
matches this ctcfinition of a transition:
(object Trans "t-scan2jam1' source " :TOP :s-operating:s-scanning : Junctionl" target " :TOP: s,operating:s,jamming: Junctioni"
1
[stringhl ù predefined in TSL see beIori-. Xocice chat the esample docs not h a ~ e an instance
of either the [opt actionabject) or [opt event@ardslist] non-terminais. These non-terminais are
declareci optionaI by the opt keyword.
The foilotring List shotvs some of TU'S predehed non- teda ls :
id . h y identifier be,@ming nith a letter or underscore and contuiuuig with any number of Ietters. digits. and underscores.
[number ] An>- unsigned integer or real nurnber.
[stringlit Xny double quotecl string beginning and ending ~ 5 t h a ciouble quote and consisting of a- number of characters in between. For example ~:TOP:saperating:sscanning:.Junctionl~ is a legal [stringlit] non-terminal.
[charlit ] Like [stringlit] but beginning and entling with single quotes. For example -+'
Xon-terminais can be modSed by prefkuing them with opt. repeat. or list as follows:
[opt X ideclares tliat the presence of the non-terminal S is optional. [ri the transition clefinition given above both the [actionabject] 'and [eventguarclslistl nonterminals are optional.
[repeat X 1 States that zero or more repetitions of nonterminal S will be niatched.
[repeat X+ 1 states that one or more repetitions of nonterminal S d l be niatched.
[list X ] States that zero or more repetitions of S separated by a comma will be matchecl,
[list X+ 1 states that one or more repetitions of S separatecl bu a comma will be matchcd.
7.2.3 TXL Transformations
In the second phase the input parse tree is transformecl into the ciesirecl riiitpitc parse tree bu the
application of a set of transformations (specified by TSL functions arirl riiles in the TSL progan~) .
The basic TSL rule defiiiition has the following forrn:
rule name argument [argument,typel replace Ctypel
pattern by
replacement end function
R-here name is the mle name. and [type] is the non-terminal target of the rule. The tgpe cletermincs
the scope of the rule, Le. which nodes of the parse tree the nile will match. The pattern clefines
the particdar type and shape of the sub-trees that must be matched for the nile to proceetl. The
replacement represents the subtree that will be used to replace the subtree that n*as matched in
pattern. The d e d l perform the replacement o d y if the pattern match succeeds othernise the
pane tree is left unchanged. -1 d e may &O take an argument of ~ p e [argumentzype].
The folowing is an example of a function that matches a transition object and repIaces it bu
a ne%- transition object in &-hich the target of the transition has been changed to that supplied by
the NewTarget argument. Narne, Src, and Tgt are variable names bound to the elements of the
hct ion 's pattern. The convention of lead capitais has been used for variable names.
59
function alterTarget NeuTarget Cstringlitl replace [trans-oh j ect]
' ( 'object 'Trans Name [stringlit] 'source Src [stringlit] 'target T g t [stringlit]
'1 b Y
' ( 'object 'Trans Name 'source Src 'target NewTarget
'1 end function
Rules are applied reciirsively. if a rule's pattern matches a sub-tree of type [SI anct the replace-
nient is a new sirb-tree that is also matched by the rule's pattern then the nile mil1 be applietl to the
replacement. This continues until the new stib-tree (alwqs of type [SI) does not match the riile's
pactern. Fiinctions arc rules that are not applied recursirely-
Rules and functions r n q also contain variables. constructors. and cleconstriictors. Vnrinbles are
names boiind to sub-trecs of a particiilar type when the variable is introducetl. Constnictors are
useif insicic rules to builcl intermecliate subtrees for later use in the ride. Deconstructors are usecl
insicie d e s to break apart (match) large subtrees into their constituent parts (in accorclance with
the grammar).
It is aIso possible to write rules that do not perform a replacement. rather they n-rite sorne
portion of their scope to a file for later use or analysis. TSL is a miich richer progamming langiage
than çan be tlescribecl in this section. For an in ciepth description see [Corclyt 001.
7.3 Capsule File Format
The telaual form of a capsule file is organizecl as a Linear secluence of sections. :i complete capside
test fde is included as Appendk B. The behaviour of the capsule is ciescribed in the stateniacfinc
section. The statemachine section contains a single top IeveI state object which is the parent of
al1 States in the capsule's FiFSSL State objects contain a k t of their contained state and pseudo
state objects foUoved by a List of the transitions bemeen the states and pseudo states in the List.
State objects may &O contain a section describing their optionai entry and exit actions. Pseudo
state objects correspond to the states. junction points. initial points. and choice points founci in the
behaviourai level of a RoseRT capsule. Junction points are an implementation detail chat permit
transitions crossing state boundaries to be comected. - b y state object may contain its own pseudo
0 TXL Markup and Info Extraction: TSL programs are tised to perform a marlrvp on the
cripsule Ne. Slarkups are used to iiniqueiy iclentifr. each pseuclo state object and transition
object in the capsule. X rnarkup consistc of inçerting a '<' symbol lollowed bu a unique
identifier before the nonterminai and inserting a '>- symbol after the nonterminal. -1 second
TSL program parses the markedup capsule file in order to produce several ciifkrerir. l i t s of
information. These lists are stored in separate files and include:
- Iist of transitions
- Iist of States
- l i t of al1 pseudo state objects
- list of protocols and their signais used bu the capsule
- l i t of guards
- list of choice points
- list of transition action objects. which are the code boclies associated n-ith a transition.
0 Mutat ion Programs: For each mutation operator there is a mal1 control program Iivritten
in C++) ancl a TSL program. Each control program invokes its correspontlin,~ TSL program
iisirig information from one or more of the information files as parameters. A single invocation
id the çoritrol prograrn will execiite its TXL prograiri multiple tirries, once for eacli nultarit co
be gnerated. Somc concrol progams also mite a log mhich correlates the rntitated elernent
with the file rianie of the mutant.
TXL Remove Markups: EacIi mutant capsule is processerl bu another TSL prograni n-hic11
rernovcs the rnarkiips.
Coucatenation: For each miitatetf capsule a coniplete mutatecl RoseRT rnodel file is creacerl
by concatenating the pretiu. specitic mutant. and SUEY filris. The resuiting mode1 files are non*
ready for use by the RoseRT code generation program.
The program for each mutation operator is i n ~ k e d Gorn the comniand Line. A more sophisticated
mutation system wouid ailom seIection of desired mutation operators via check boxes on a G U form.
however in this system individual operators are selected by ntnning or not ninning their associated
control programs.
7.5 Example Mutation Operator Program - TAT
The TAT operator alters the target state for a transition. The control program tat.exe follonis the
following procedure when invoked:
1. BuiId a list of al1 of the transitions in the capsule by reaclig the transition list file.
2, biiilcl a list of al1 states in the capsule by reading che state L i t füe.
3. For each transition in the capsule:
(a ) BiiiLl a lisc of al1 Icgal targets for the nirrent transition. Legal transitions incIiirle al1 sibling states, the transition's source statc .mcl possibly the parent scate. The parent state is only possible if there is an explicit transition frotn one of the sibling states to its parent. This requirement is necessary because chaiiging the target state of s traiisitic~n requires that the target state aiready have <an entry junction point. Transition c3 in tigrre 7.3 is stich a transition. The original target state is not includecl in this list.
(b) For each entry in the target list:
i. The control program invokes the TAT TSL program to find the crirrent transition and replace its carget with the target supplied. The unique identifier for the ciirrent transition. the new target. and the fiIe name into which TXL niIl wi te its oiitpttt are passed to TSL as command Iine parameters. The file name d l be T.-\TSS.cap. The 'i corresponds to the number of the unique identifier of the current transition. the Y corresponds to the list indes of the new target. TAT-4-7.cap inclicates that transition number 4 has a new target which is the rth entry in the L i t .
ii. Log the mutation in the TAT log Ne. This fik is Iielpful in creating test seqtiences to kill live TAT mutants: the tester does not have to look at the capsule file which is considerably larger.
Figure 7.4 shows both Icwls of a two level statemachine. Suppose the current transition is t'll whose
source is S27 and tiirgct is S21. The IegJ target states for the TAT mutation operator are S23. S21
(this sould be a transition to self which is Iegal), and S1 via the jtinction labellet1 t l .
It is usualiy not possible to view the mutant with the m e n t version oE the PITS-RTI toolset.
One of the sections of a capsule file is the stateview section. Ki section recorcls the physical Iayoiit
of the state machine elements as they are entered bu a programmer. if the mutant file is loaded
into the toolset the ciifference between the statemachine and stateview sections may either cause the
coolset to crash. or be ignored in whicti case the stateview incorrectiy shows the tinmutated siate
machine. in order for the tooiset to correctly display the mutated state machine it is necessq
to alter this section or remove it compIeteIy. Removing it results in the toolsec creating a ne&-
graphical vie=- based on the state machine section. However aii formatting is lost and the resutt
is an incomprehensible mess that mus be reformatted mmua.iiy- The alternative is to dter the
stateview section. This wouid require the development of anodier grauuuür and more TXL progünis
#'
4" fi,, , , t ' " !fl * - - ,... ?.!<*':.'
- .. , ., - , - . . . " , .. < .,. ,,.;;;,. . . .. -*.
., :;>..: -2. '. -. , , . , . . " . ..b'.,?p.. : '
Figiire 7.4: T-AT example çtate machine.
effectivelu doubIing the work effort. It w s decided that havïng correct stateriews n;u- not
the effort requircd.
The architecture of the MTS-RT system is essentiallu that of a batch sequentid systern. AIthough
not particularly user friendly, a batch architecture is neil suitecl for prototyping a çustern consisting
of many s m d components. Each component cati be dereloped and tested in isalatian. Cade changes
when necessary are isolated. and the products of intermediate stages c m be easiiy esaminecl.
This chapter has provided an overview of the entire LITS-RT mutation system. and an e+uplma-
tion of how mutation operators can be applied to the textual version of CSMs using TX.
Chapter 8
Creating and Running Tests
This chapter describes the process of creating test cases. nmning test cases against the mutants.
analyzing the results of execution. and tliscusses what to do when a fault is found in the original
program.
8.1 Testing Process and Automation
Regarclless of the testing strategy (test adequacy criteria) usetl. the basic testing process consists ol:
Creating test cases based on the testing criteria.
Running the program under test and Leding it the tesr; cases.
Observing the mn-time effect the test cases have on the program. such as rneasuring coverage.
Comparing the actuai program outpiits against the expected outputs.
Recording the results in test logs and anaiq-ting the Iogs.
The feasibility and d u e of automating some of these steps depends on rnany factors inciuding the
type of software being testeci. the necessity of being able to repeat tests in the hrture (perhaps
regression testing). and the avaïiabiiity of an autornated oracle. For esample the value gaïned from
automating the testing of a singIe release web apptïcation is probabIy not wonh the cost. On the
other hand it may be weil m r t h the c o s of automating the testing process for a long livect muhi-
version system such as telephone mitchhg software. It is usudiy possible to automate the processes
of nmning the program against each test case, and recordmg the results. A program that cloes this
is a test driver. A driver cdls the program being tested and keeps crack of hon- it performs. It
shoulcl:
O Set rip the environrnent needcd to cal1 the iinit being tested. This m q involve creating and initidizing global variables.
O Invoke the prograrn under test against each test case or sequence of test cases. The test cases for these invocations could be read from files or be ernbericletl in the driver% code.
O Record the results and possibly check their correctness against the expectetl residts. and n ~ i t c the outcornes to a log.
in addition to tlrivers. testing often involves the use of stubs. -4 driver simulates the pan of the
program that c a b the iinit iuitler test. Stubs sinidate parts of the program c d e d by the unit onder
test. -4 stub can:
Check the reasonableness of the environment provicled bu the d e r .
O Check the reasonableness of the arguments passed by the calIer.
O !dodi& the arguments and the environnient. followed by returning values in siich ;i rvay tliat the caller can proceeri. It is best (obviously) if these effects are consistent with specification of tlie unit the stiib is siniulating. A goai of tcsting rtianagcrncrit is to scticcltilc testing so ;ü: to retliice the niimber of stubs reqiiired.
Testing reactive software mocieled by active objects such as RoseRT capsules aften requires chat
the timing of messages bc controlled. For a passive prograrn morieled by a FSM. timing is nor
controlIed by the FSM. rather it is controlled by the tcst harncss. With an rictivc object timing is
controlIed both bg the active object and the test driver. Active objects may generate their ottn et-encs
in response to iriterrupts and timers. In the case of a capsule. some transitions ni- be triggered
by an internai event such as a timeout. an intemipt, or a message Gom the Game service. In swh
cases it mq- be necessary that the test driver evplicitly create a situation Ï n which the internai event
c m occur. This requires that the time sequence of inputs to the capsule be controiIed bu the test
driver- The ganularity (events occur at intervals measured in miHisecon&. seconds. minutes etc.)
and precision (the acnrracy of timing measurements is measured in milliseconds. seconds. minutes
etc.) of the timing depends on the specification or requirements of the application.
8.2 Testing RoseRT Models
Often a RoseRT capsule is testecl by using the obsenability feature of the toolset. This feature allon-s
a cleveloper or tester to manualIy inject s iga l s into the ports of a capsule and observe ancl recorcl
the responses. This methocl. while good for developmerit. is very labour intensive ancl tests c-aiinot
be ensily nor aiitomatically repeated. I n oit-iised alternative is to cnnçtnict a clriver capsii i~ char
esercises the production capsule nith a limiteri nrimber (ohen one long sequence) of -harcl-cotlecL"
stimuli. This methocls weaknesses are the tediormess o l conscnicting many correct driver capsules
and the lirnitccl numher of twts ttiat cari be performeci hy a single ststic ciriver capsiile. To siipport
mutation tcsting a clrivcr is necderi that can:
be easilg (minimal progamming) used to test most capsiiles.
a execute the capsule iinder test (CapL'T) over a large and tlynamic set of test seqiiences.
a control the temporal seqiicncing of messages sent to the Cap UT.
a record the stimulus-response history of the CapCT for siibsequent analysis.
\Vith these criteria in niincl a test sequence creation tao1 (narned RT-TestCase.CIanager) and test
driver (ricmed RT-TestHarness) were deveIoped in tandern. RT-TestCaseManager is iisccl to create
test sequences that are eventrially read by RT-TestHarness.
8.3 The RoseRT Generic Test Harness (RT-TestHarness)
The RT-TestfIarness is a RoscRT moclel into which a capsule to be tested is imported'" (the protocoIs
instantiated as ports by the CapUT are also required). X skeleton version of the RT-TestHarness
mode1 is used as the basis for each capsule to be tested. The class diagram for the skeleton is s h o m
in figure 8.1.
;\fier importing the capsule CO be tested and any required protocok into the skeleton. a minimal
amotuit of nork m u t be performed by the tester via the RoseRllT toolset (about 10 minutes each
for the capsirles used in this thesis)- Figure 8.1 k an esample of the encl resirlt the class d iab~am
for a completed RT-TestHarness. in this esample RT-Testffarness instantiates tn-O classes and two
capsules:
"The tooiser dlowvs capsules and protocois ro be exporred and irnporred ro and Gom diik files. These featiires support software engineering activities such revision cantml, muse. and t e m cieveloprnenc.
*Capsule> B
Driver
.# ( TestComms
.+ i l a g + i timer # i TestComms
+ l hame <<Pottz> + i aboniïmer
Figure 8.1: Skeleton Test Harness Claçs Diagam RoseRT Mutation System:Essential WorkHows.
O JarnrnerDriwr: .A siibclass of the generic Driver capsule which is minimaiiy modifieci to inter- face &$th thc specific rapsule under test. The code for the sendEvent methocl is generated bu the RT-TestCaseManager application to be described in section 8.4.
CnpLT: an instance of the Jammer capsde which is imported from sonie development motlel. Two intemal self transitions are added as test instrumentacion to this class. This instrrrmen- tation is usecl to return information about uncxpected events. the CapGT's rnirrent stnte. and the Capu'T's temination.
O TestReatler: a class xhich reads test sequence files.
O DataLogger: a class used to record the resuhs of test mns.
Xlso shonn rire two protocol classes- TestProt is a protocol used to perform state queries of the
CapCT. Jarnmer Protocol is imported and cornes with the CaptiT. In this example the CapCT onl-
has one port. Some capsules have rnuitipie ports. In such cases the protocol for each port must be
imported. Yotice that the attributes and operations for the DataLogger. Test Rearler. and Driver
capsules have been eiided.
The test harness nuis according to the foUoning sequence:
Figure 8.2: RT-TestHarness Class Diagram.
Datahgger +databgger
1. Rcatl in etivironiriertt data from ari initialkation file.
restCOm*s
<<port>>
>
2. Determine the name of the current niutant by parsing commancl line arguments. RT-TestHarness is irivokerl bu RT-TestRunner (to be dicussed in section 5.3). Communication betn-een RT- TestRunner and RT-TestFiarness is via command line arguments.
un *Prntoc0la TestPrnt
+OstateReply O +otrap O +ounexpected 0
a +statcRequest O
3. For each test sequence file in the tests riïrectory
(a) Sec if this test sequence was run during a pretious session. This is tlone by checking to see if there is a corresponding results file for this test sequence file.
dapsule~ Jammerûriver
endEvent0 ~Capsule- 0
rackData0 CapUT
m# I JxComms- JammerPrniocol
+DstopJam O +njamStatusRequast O 4
(b) if the test vas run pre~iousIy then restart step 3
+ogoOpwate O +og~5hurdown O +oreset O O+jamStatusReply O
(c) instantiate the CapCT. Instantiating a ne-- instance of the capsule for each test ensures that the CapCT starts in a knam state.
(cl) initiate the data logger for a new test nui.
+ i Jwcornrnr
(e) For each signa1 in che test sequence file:
i. Send the signal (via the sendEt-ent method supplied by RT-TestCaseSIanger to be described later).
ü. if the signal &O specifies a dekiy then wait the specified duration before initiating another event. The deIay feature can be used to test events such as timeouts that are initiated by the CapCT.
K. Othemke mit for and log my responses.
I + i TçstComm:- + i log + i JxCornms + j IookThmughTimer + 1 BITTimcr
iv. dfter a short intend (default is 1 second) send a state request to the CapGT. This del- d o m the CapCT to respond to the most recent test input bu powibly rwving to a new state or sending a message CO one of its ports.
( f ) Close the results file for this test sequence.
(g) Destroy the CapUT.
4. Terminate the RT-TestHarness application.
Note that RT-TestHarness is just another RoseRT moclel from nhich C++ code is generaced.
The generated code is ultimately compiled and Iinked with the RoseRT senrices library ta procluce
a stancl-donc executable. One of ttiese test hamess models is created for each mutant to be tested.
The actual code generation. compilation, and mnning of each mutant test harness is iintler th^
control of the RT-TestRunner program.
Depentling on the CapLT it rnay be riecessaru to further mociify the driver siib-class. For esatriplc
sonie CapL'Ts rnay Iiavc otic or iriore periodic events that pnll the environment (siniiilatetl hy RT-
TcstHarness) for claca. Rather chan requiring the tester to creace test cases chat explicitly respond
to mch polletf recluest it niay be more efficient to add one or more internal self transitions CO the
RT-TestHarness wtiose purpose is to respond to the periociic reqiiests. This wns rarried out in sortie
of the capsiiles testecl in this cspcriment.
8.4 The RT-TestCaseManager Program
RT-TestCaseJIanager is a G U tool deveIoped using the Visual Cs+ and thc JIicrosoft Foiindation
Classes. Tt facilitates the rapid and easy creation of test sequences. Its main window is shon-n
on figure 8.3. Test sequences are stored in separate files in a specifietl clirectory (see fig. 7.1).
The leftmost pane of RT-TestCaseManager display the contents of this directory. Selectiug one
of these tests causes it to be displayet1 in the rightmost pane. One of the outputs Gorn the TSL
JIarhwp ancl Info Extraction phase (see chapter 7) is a file containing a List of al1 of the siqals
the CapLT can accept as inputs. These are dehed by the protocols that the CapCT instantiates.
RT-TestCaseManager reads in this information and displays it in the center pane. X signai can be
added to the end of the m e n t test sequence by selecting it Gom the center pane. If the seIected
signal takes an integer parameter then the value to be sent rvith the signal is <ais0 entercd ( ~ h e
default is void). Other data types are possible but are not included in this version. -ln optional
%-ait time can also be speciiied for each signai. A tirne entered in the vait column instnrcts the
RT-TestHarness to wait that amount of cime before sending the nevt message. Each test case aIso
bas a 6eld for comments. In the test case shom the comment indicates thnt the purpose OC the
curent test sequence is to try and kiii the mutant icientified as T-AT-ilo.
RT-TestCaseMariager ais0 generates the code body for the seniiEvenc method i~serl by the srib-
classed driver cIass (JamrnerDriver in figure 8.2).
in t sendEvent ( int event Index, int sendilata, int uaitTime1 { svitch (eventIndex1 €
case(JXC0MMS-STARTJAM) : { datalogger . appendlog ("sending JXCOMMS-STARTJAM\nW ; JxComms . startJam(sendûata) .sendo ; break; >
... more case statements ... )//end svitch return uaitTime;
1
The sendEvent method correlates the signal data in the test sequence üies (encoded as integer
constants) nith the actual RoseRT library cidi required to send the event. This code is generated
basecl on the contents of the center pane as descnbed above.
8.5 RT-Test Runner
Once the mutants have been created and an initial population of test sequences createcl the ac tud
ninning of the mutants ngainst the test seqiiences c m hegin. Running the mutants is performecl
11'. the RT-TestRunner prograni. RT-TestRunncr is a cornm~md line program that execnites the
following nlgorithrn:
1. Reaci in environment ctata from an initiaiization file.
2. Build a list of al1 mutants by reatling d l of the mutant file nanies in the mutcuits ciirectory.
3 . For each mut~ant in the iiit:
fa) Check whether the current niutanc was huiIt tliiring a previous test session. This is performed by looking for the current mutant's esecutable in the esecmables tiirectory
(b) if the esccutable nlreatiy exists then:
i. itivoke it and pass iti its name (the name of the current mutant) ris a comrnancl line argiiment. The progam being invokecl is the esecutahle version of the RT- TestHarness mode1 which concains a mutatecl CapGT.
ii. Go to step cl.
(c) If the executabte does not esist then concatenate the current mutant capsule fiIe a i th the p r e K ~ and suAiu files ttientioned in chapter 7.
i. invoke the RoseRT comma~id line tools whicti will generate C+t code basetl on the current mutant.
ii. invoke the C++ compiler which compiles the Cf+ source cocle =and links it with the RoseRT services Iibrary. These activities represent the largest computational biircien of the mutation syscem.
iii. Place a copy of the new emcutable into the executables clirecton for future itse.
il-. invoke the executable passing It the name of the current mutant as a command iine argument. The program being invoked is the ~ ~ e c u t a b l e version of the RT- TestHarness which concains a mutated CapCT. (Same as step b.i above).
(d) The executable (RT-TestHarness) mites test resuit files to the resuIts/currentMutant t i i r ec to - currentAIutant is a dieccory name identi-ng the tnutmt. An esample is TATJ2.
(e) Log the run in RT-TestRumer Iog me (this is a single text fiie that keeps a record of dl mutants nia by RT-TestRumer.
4. Close the RT-TeçtRunner log fiie and terminate.
8.6 Data Analysis: RT-SuperDiff
Before running RT-TestRunner for the first tirne the non-mutated version of the capsule file (caiied
cleanCapsuIe) is put into the mutants directory. This is the firçt %nutant" run. Its resitlts are
the results against nhich al1 of the mutant's results are comparecl. At m y tirne (even when RT-
TestRumer is ninning) the tester c m invoke RT-SuperDB. RT-SuperDiff is a program thnt pe r iom
a file cornparison between each mutant test results file and the results of clennCapsule. -4s each
cornparison is petformecf a log fiIe is written. Cpoii completion of al1 comparisons the Log fiIe is
prncessetl aricl convcrtecl into a table which c m be read tlirectly or importctl into a sprendshcet
progran1 as in table 8.1.
killeti Test effectiveness total mutants total mutants run niiitation score
Table 8.1: Data JLndysis Results.
Thii table shows .5 tcst cases executed against 18 mutants. -ln 'L* in the body oE the table
nieans -Live-. The row labeled "kilted" shows how many mutants were detected (Mled) bu che
çolumn's test case. The column labeted -hiIed" s h o ~ s hon- man'- test cases were able to detect the
row's mutant. h zero in the kiiled column indicates a mutant that has not been detected bu any of
the existing test sequences: a iive mutant. The row labeled test effectiveness indicates ho^ mm>-
mutants were kiiietl by the correspondig coIumn's test setluence. The colunm labeIetl proportion
killecl indicates the relative ease with nhich a mutant m s killed by d l of the test sequences. Tlie
mutation score is tlisplayed in the last row. In mutation systenis like hIothra. once a mutent h.as
hcen tletected. it is niarkecl as dead and never nui again. However in thii system we are interescecl
in analyzing the:
0 effectiveness (how many mutants ic c,an rlistinguiçh) of pruticular test cases against al1 miitants. and
0 the scrcngth (how hart1 chcy are ro kill) of mutanc operators against d l test cirses
Therefore deid mutants are not removed €rom the systcm .and crery mutant k eventiially riin against
Ç'VE'Cy tCSt Cirse.
8.7 Hunting for Mutants
By looking a t cliarts like table 8.1 the tester can see which mutants are live. Tlie next stcp is to
ii&l test seqiiencrs niniecl at killing these rernaining mutants. Creating new test secpiences to kill
rriiitants is the niost labour intensive p m of the entire mutation testing process. Testers can use
rxrioiis pieces of inforniation to create new test sequences:
0 The name of the mutant tells which mutation operator has beeri appliecl to ivhicti elcrnent the capsule. For example TFtD 3-1 for says that the mutation operator ti TRD: TRransitiori Delete. the 5 signifies that the transition object with unique identifier 5 was deletecl. The trailing L is alaays a L For TRD since there is oniy one way to delete a transition. Other operators can be applied severd times. For exatnplc TAT. Transition Alter Target. cari be applictl as many timcs as chere are legal aiternative target stntes. In this case the trailing number will inclicate n-hich alternative target is selectetl. Lleally the spreadsheet presentation shoitlcI Iink to this information attached ta a RoseRT CS31 diagram. As this would reqiure considerable fievelopment effort this Eeaturc was not developeci ditring chis thesis. It is left for future work.
a Some mutation operator control programs produce Iog Mes chat show the original and mutaterl capsule element. The TAT operator. for e.xampIe. Iogs the original and mutatecl target state for each transition chat it mutates.
0 The marked irp capsule file c m also be consulted.
&*hile trying to create new test sequences the tester may decide that a mutant is equitalent to the
original capsule. The tester may aiso decide that a certain mutant is too difficult to distingukh.
This may indicate chat the code is untestable which in turn may imply that the code shoulrt be
returnecl to the developers for improvement (design for testability was poor).
Xfter new tests have been added to the system RT-TestRunner is re-invoked. RT-TestRumer
and RT-TestHarncss ni11 only nin new test sequences. Sote that the original program miist be run as
well in order to obtain the expected resuIts for the new test sequences. A strength of mutation testing
is chat while adding test cases to kiU live mutants the correctness of the results of the compiitation
are not important: n e are only interestecf in whether the mutant's resiilts ciiffer froni the original's.
The correct behaviour of the original progrmi should only be checkecl after clie tnutant tins b ~ e n
killetl. This allows the possibiiity ofcuüing redundmt test cases from the test siutc. There is also the
possibility thnt wtiilts crrnting a killer tesr secpiencc a f d t is foiind in thr, original capsula (possibly
heciiiise the iiiutarit is the correct version). The next section disciisses this eventuality.
8.8 Finding Faults in the Original
The mutation literature is poor when it cornes to tdilking about mhat to clo n-hen the original progrnm
ha a faiilt. It is irnpliecl that fndts are fount1 tlitrïng the process of creating a new test to kiH lire
rriutarit. Wien any test is run on the original capside. the result ni- reueal incorrect hehvioiir.
Hoivever this recluires an oracle. a role often played by the tester. Faults may ;&O be rer-ealetl
through the process of exarnining the mutant code in order to generate the new test. This can be
though of as a directed inspection in that the rntit~uit points to the exact program statemcnt to
examine. This is supportecl by Hanilet3 suggestion that systematic testing is sticccsshil het::iiise it
forces a tester to pay systernatic attention to the software [Harnlet 881.
Wien a fault is found anci kxerf in the origind capsule (mhich is part of the RT-TestRarness
nioclel) al1 of the prerïous testing may Iie invaIidated. It is possible that a test case that MIS a
rnutnnt version of the faulty progam may not kir1 the new mutants created alter the fatth has bcen
correcteci.
With this system n-hen a f a d t is repaired. the corrected capsule must be extracteci Gom the
RT-TestBamess mode1 and the mutant generation ptocess must be cepeated as described in chapter
7. The tester must decide whether it is necessary to rerun any of the dead mutants against the Luecl
capsule since. as discussed above. a change to the CapU'T may affect the validity of pretious tests.
i.e. n Lest seqtience that killed a mutant may no longer 'di tbat same mutant bmed on the corrected
version of the capsule. This iç an important decision because generating and compiling mutants is
a major computationai burden of the system.
8.9 Summary
Onc of the problems of automateti testing in generai is that of excessive log files. Sonie of the log
files gcncratcd bu the MTS-RT systetn rire only consulteci when there is n problern with the systemi.
Others are consultetl in order to find information neccssary to kilI livc mutants. [n practice only a
m i d l perceritage of the data lrigged during the mutatiou process is ever consulted. Tliereforc orle
of the problems to be addressecl in future work is how CO efficiently extract the sniall mioiint of
information that is actuaily usehl to the tester.
This chaptcr introtlricetl some arrtomated testing icleas anci describeci the progams iisecl to create
mutants. control the execution of mutcuits against the test sequences. and anaiyze the test restilts.
After cfescribing the programs the proceçs of killing niutanst cwtl finding faults was coveretf. SIany
implernentation details of thc various prograrns have been Ieft out in order to iniprove clarity of the
presentation.
Chapter 9
Results
This chapter presents the result of performing euperirnents on seven capsules ancl cliscrisses the
performance of each of the miitacion operators proposcd in chaptcr 6. The chapter conclucfes trith a
tlisciission of the rnany observations made during the performance of the cxperinients. The niimher
of cxperiments performetl in this thesis is smalI. Thereforc the observations concerning rtiirc;icion
ciperator cffectivencss arc pre l imina . Frirthcr eqerirnents are requirecl to make firm any conclnsioris
c h v t i frorii this w r k .
9.1 Capsules Sumrnary
Eleven capsules were usetl in this thesis. CompIete mutation testing was perforrnecf on seven of them
whiIe the others were either rejected due to poor quality or aere only iised to investigate other aspects
of niutation such as niunber of niucants generaeed, Of the capsules fully testetl. one taken Gom
a moclel provitlecl t>y Rational. tn-O were from examples inclucled nith the toolset. tn-o were Frorn
personaf projects. ancf tmo were ffom graduate student projects. The foilowing sections describe
each capsule and the observations made whie performing mutation testing on them. Each capsuie
description includes a table summarizing botb capsule and mutation details. The statemachine
tliagrarns for the seven tested capsuies are inchded as hppencliv E. Table 9.1 shons the s t i m m a -
of the results for the k s t mperiment. The foilowing section esplains the table.
! Causule Details !I Mutation Details I i transitions states pseiitlestates levels of hierarctiy ports in signals out signals actions
Miscellaneous Yumber of Test Sequences iengths of test sequences longest test sequence
compile time (al1 mutants)
3 32.2
26 secs
14 minutes
Number of Mutants 6
coverage ITS/'CTI TRD TCP SCP TGT
TCh TTG SDL Testing Details mutants generated mutants compilecl cqiiimlent invalid 11 valid 9.5 tlcacl. firm 9 1 mutation score, firm 96% ~Ieatlstrong Y9 mutation score, strong 94%
Tablc 9.1: Experiment I: PBS CeIl Phonc
9.1.1 Explanation of Table Elements
Capsule Details
a Transitions: This is the total number OF transitions in the CSM. including internai transitions
and initial transitions.
a States: This number includes al1 states in the CSM including the TOP statc ahicti is the root
of the CSBI hierarchy.
Pseudo States: This is the total number of elements that are foruid in the RoseRT capsuie Éile
as elements of state lists. It incrudes states. initia1 points. jiuiction points. aricl choice points.
See section 7.2.4.
a Levels of hierarchy: this indicates the depth of the deepest containeci state machine. This
includes the TOP level root state object-
a Ports: This is the number of ports used by the CS31 and therefore the number of ports
through which the RT-TestHarnes communicates wïth the CSM. If the CSAI contains ciniers-
the number of timers is aiso indicated.
in sïpnls: This is the total number of in signols entering the CSJI. This is the nrimber of
tlifferent messages that can be sent by the RT-TestHarnes (ic does noc include timer messages)
as stimuli to the CSM. in signals are defined as p a n of a protocoL
out siqnafs: This is the total number of out signafs chat can be emitterl by the C'SU. This is
the niunber of messages thac can be received by the RT-TestHarness. mit signafs are definet1
as parc of a protocol.
Transition .ktions: this is the nimber of transitions chat have action objects associateci mith
theni. Action objeccs are C++ code segniencs.
0 Siinibcr of Slritants Protlucctl: This section lists each operator. or group of operators if they ' r~
sirriilcar. followetl I)y die riurnber of mutants generated bu the tnuration system.
Testing Details
LIutants generated: This is the total of the mutants listecl iu the criutation cletail section.
hIiitants compile& Tt& is the t o t d nurnber of mutants that successfully conipiletl. Sorne of
the ~riutants generated by the SITS-RT systern are syntacticaily inlaiid and therefore da not
compilc.
Equivalent: this is the nurnber of mutants thac were judged to be equimient to the original.
un-mutated. capsule. Equitdency was determineci througti the course of mutation tcsting
while trying to kiiI Iive mutants.
Intaiid: this is the tocd number of mutants that were either qntacticaliy inraiid (for the code
generator or the compiler). or chat upon inspection were not ali id RoseRT files. The latter
form of ind id i ty occurred if. for example. the SITS-RT system created a mutant by adding
a trigger to a transition that shouId not have a trigger. For example a continuation transition
is not triggered. it is a continuation of a transition that enters a state kom a superstate and
continues to a substate. Figure 9.1 shows such a case. Transition t l l is a continuation of
tr.uisition tl. It is not possible to add a trigger to t21 using the graphitai tooiset however it
cm be done bu the SUS-RT system (the TTG operator). Such a murant is not syntacticalIy
intalid. rather the trigger is just ignored by the mutant. Such a mutants are equitalent.
However since they should not be generated in the first place. they have been classifieri as
invalid rather than equivdent. -4 Future version of the TTG operator roulti look for and
esclude continuation transitions however this was not clone in this version.
I 1
Figurc 9.1: Continuation Transition.
0 Valicl: This is the total number of miid mutants.
killetf to obtain a 100% mutation aclcquncy score
mutants minus the i n d i d and equivaIent mutants.
i.e- the nurnber of mutants that musc bc
. It is eqtral to the number of generated
0 Dcatl. firm: This is the number of mlid rntitants that were eventually killecl by the test
seqirences. -1s riiscussed in chapter S some instrumentation n-as atlderl ta the CapC'T in
order to allow obsen-ation of the CapCT's current state and ariy uiic.spcctet1 messages tIirit it
rcceived. This estra information sas part of the output used to clerermine whether or n a ;i
mutant had been hlled by ri test seqtrence.
SIiitation score. firm: This is the firm mutation score which was arriveci üt by d iv id ig the
number of firrnly clead mutants (e-upressed as a percentage). by the number of d i d mutants.
Dead. strong: This is the nurnber of mlid mutants that nere eventiiaiiy killed by the test
sequences. The instrumentation information cotlected as part of the mutanté outprit n-as not
used in the determination of mutant currectness. O* the observable behat-iour of the CapLT
R-as used to compare the mutant's output to the original's output.
O Mutation score, strong: This is the strong mutation score which aas arrived at by tlividing the
number of strongly dead mutants. by the nimber of mlid mutants (exprcssed as a perccncage).
Miscellaneous
O Sumber of test sequences: This is thc number of test sequences iisetl during the niiitation
tcsting.
O Lenghs of test seqiiences: this is a comma-separated list of the lengths of each of the test
seqiiences iisetl. Lcngtii refers to the number of indigenous messages sent to the CapCT. It
does not iriclude iristriiriieritatiori messages (the only one iisetl in this thesis is the state recliiesr.
message) senc to the CapLT.
Longesc test sequence: This is the length of tirne in secontls it takes to execiite the longest
test sequence on the origilia1 (noti-mutateci) version of the CapCT. This nuniber is s ip i t icmt
n-hcn a capsitlc uses tiniers.
O Proccssor: Threc compiiters n-ere merl in this thcsis. a Pentirtm II '266hIHz. a Celcron 3661iHi.
;iricl n Pentiiim iiI Ï33itIHz. This colurnn inclicates which one ivas iisetl to test this capsule.
O Compile time (ail uiiitants): this is the total time reqirired to generate ,mtl cornpile (not nrn
'any test sequences) al1 of the mutants for the e.xperiment.
9.2 Experiment 1: Cell Phone Component
This niocle1 was siipplied by Rational in respome CO +mail comniunications, The P B S systeni is a
mode1 ttsecl cfuring adwnced rnodeling courses taught by Rational. However. the Ce11 phone capsrile
is a relatively simple capsule extracted lrom the model. Table 9.1 shows the sumrnary of the resiilts.
The first test sequence is ii transition tour that takes ïnto account looping in order to increment
a counter used in a choice point. The transition tour Mled S4 of the 89 kiiiable uiutants. W d e
constructing the second test sequence cn-r, issues were found. a reduntlant transition and some clead
code.
The dead code r i the result of an i f statement in a state exit action n-hose predicate alnays
e d u a t e d to false. The dead code fauit n'as found wtiiie constructing a test sequence to Hi an -ADS
(Action Delete state exit) mutant. The question poçed by the live A D S mutant ms -ïvc cIeleteci
this code segmenl but it doesn't seem to matter, why is that?" The answer was chat it tiidn't matter
because the code does noching.
The redundant transition m s a result of the dead code and was found while trying to kill one of
the TRD (TRansition Delete) rntitants, FVMe constructing a aew test sequence it nas not possibIe
CO kill one of the TRD mutants. This memt chat after deleting the transition rione of the test
sequences were able to tletect a change in the capsule's observable behaviour. On the other liant1
thr coverngp iiperator (C\*T) For this transition w u kiileti iriciicacing tliat the transition is cnkm in
i~tic of the test scquences. The Eact cliat chis transition lias an obsenxblc oiitpirt action malces the
t ~ o obsenztions appcar contriirlictorp. Hrirvever. i~ was iound t t i a~ the transition $vas rctlundarit.
The capsule has two transitions that respond to the s m e event. One is in the top state whilr! the
other is in one of the substatm. When the capauIe is in the sub-statc the inner cransitiori responds
to the event and transitions to a target state %?thin the substiite. When the ctirrent state is the
top state the top statc's trruisition responds to the event. Xlthough che two criuisitious origiriiite in
clifferent Ievels of the statc machine tiierarchy they boch have the same target statc and the snrrie
outprit action. Thereforc by curitroIling wtiidi state ttie niachine is in the tester çan musc either
one of tiic transitions to be taken. But with the inner transition deletecl only the auter transition
can be taken. Howwer since both transitions have the same output action ancl carget state the
absenable behaviow is unchanged. if the dead code in the state's exit action not been tleacl but
rather had an obsenab1e effect on the CShI's behaviour then the rectundant transition woltl no
Ionger be redunclant.
Since hoch of these issues dicl not affect the obsenable behatiour of the capsule, i.e. they (tic1 not
restrIt in incorrect behaviour. tlien by sonie definitions they n i - not be classilied as fadts. However
bot& tfead code and ceclundant code (transitions) can Iead to difficuities in maintenance.
This mode[ had an interna1 timer that used IO second tirneouts. in order to kill sorne of the
mutants it would have been necessary to traverse thii transition 10 times, each tirne requiring a
del- of 10 seconds to don- the che r to expire, Due to the number of mutants it was decided to
change the constant cuntroiiing the number of iterations h m 10 to 2.
9.3 Experiment 2: Ce11 from a Game of Life Mode1
Tiiis niode1 sas iriclucled as an example nith the toolset. This is a single level state machine
sirniilating the life cycle of a single ce11 creatiire in a g m e of life simulation. This state machine hac!
two internai self-transitions. Triggering these transitions either increments or decrements a mpsiile
variable but does riot chcange the -gaphical- scate of the capstile. Whcn one of the ocher cransicions
is taken. the capsule changes state based on the value of the affectetl variable. Killing mutants for
this capsuIe requirecl interleming ~xeciition of these internai transitions with the %are çlianging-
trirn~itions. The tao test seqwrices iisetl tvere bascd on triuisition tours. A single transition tour was
riot possible. Table 9 2 shows the siimrnary of the resiilts. So issiies were foiincl with this capsiiIe.
transitions states pseudo-states levels of hierarchy ports in signds oiir. signals actions
MisceUaneous Yumber of Test Seqiiences Lengths of test seqiiences longcst test sequence
compile cime (al1 mutants)
processor
37 minutes
Pentium U '266 MHz
Mutation Details Number of Mutants ADS/E/T coverage tTS/STI TRD TCP SCP TGT TAT TCX TTG SDL Testine Details mutants generatecl mutants corripilecl cqui\alent intxlirl valid dead. firm mutation score, k m deac1.strong mutation score, strong
Table 9.2: Experiment '2: CeU from Chme of Life
9.4 Experiment 3: Traffic Light Controller
This mode1 was included as an e m p l e with the toolset. This ras the first capsule testeci in rhicii
di top IeveI states have sub-states. This capsule also has a terminal state that is part of the design.
so it n-as not possible or necessq to obtain 100% state exit coverage (CSX). One rather large
semantic faiilt mas found. While atlding a test case to kill a CVT mutant it n-as found that the
timer tnggering the uncovered transition wiis never set. This hult was repaired nhich requirecl the
mutacian cesting procedure to be restarted. Table 9.3 shows a summary of the rcsiilts.
Capsule Details transitions States pseud~scaces leveIs of hierarchy ports in signais out sipals actions
Miscellaneous Yamber of Test Sequences lengths of test sequences longest test sequence
compile time (al1 mutants)
processor
23 10 52 4
2 - 1 timer -1 - n 9
a - - , .a,12.7?7 23 secs
4-1 minutes
Pentium iI 266 5IIh
Mutat ion Details Number of Mutants XDX/E/T cowrage ITS/STI TRD TCP SCP TGT TAT TC.4 TTG SDL Testing Details mutants generatctl mutants compileci
cleacl. firni mutation score, firm cieacl.strong mutation score. s t rong
Table 9.3: Experiment 3: Traffic Light Controller
Because this capsule iisecl multiplc interna1 timer events test sequences took a relatively large
tirne to esecrite nith the Iongest being 23 seconds. This means that ninning a single test sequence
against the original 343 mutants took over 3 hours. The total time required to compile al1 of the
mutants tn-ice (the second time aas after the fault fk) cmuitl run a11 .3 test cases against the 343
mutants n-as in excess of 14 hours.
9.5 Experiment 4: Cruise Control
This capsule was the main controuer for a cruise control rnodel. [t n-as developed severd months
ago by the author of this thesis. This capsule has a periodic event that is triggered once per second.
This event sends a message requesting the vehicles m e n t speed. .Uthough this coiild be viewetl its
just another input-output sequence, because of its periodic nature it decided to motfi&- the RT-
TestHarnes The RT-TestBamess dnwr capsule used in this experiment n-as modified bu addirig an
incernai sclf-transition that responds to the data request from the CapIiT by sending the speecl back
to the CapCT. This 6ees the tester from having to respond to each request individudly. in orcler
to alIow the tester to update the speed value returnetl by the RT-TestRarness a new message was
added to the RT-TestCaseManager. Previous to chis d l of the messages (signds) seIectabIe through
RT-TestCaseManager corresponded to the signals that the CapLT coiilcI receix-e on its ports. The
new message. however. cioes not correspond to signais sent to the CapCT. it just npdates the speecl
ixrinble.
Ttic procedure follon-ed iir testing this capsule was to generate. compile and rtrn n strbset of the
mutants, Ench subsct consisced of al1 of the mutants prociuced by cither a single or n relatecl qoirp of
operators. There were three relatecl groups sets: the action clelete operator set ( A D S . ADE. .\DT).
th^! coverage operator set (CVT. CSE. CSS), and the internal-esternalseIf transition set (ITS. STI).
The other operators werc arltled singly. The first set risetl wiis the coverage set. Aftcr ohcaining
100% rniitation caverage the ADS/E/T mirtnnts were added. hl1 of these mutants were kiled bj-
th^ ~sisting test sequences. Sext the TRD mutants were atlcled. Again al1 of these n-cre killetl by
the esisting test sequerices. Sext the TAT mutants n-ere atldetl followed by the remaining mutants.
In eacli case al1 of the nen- mutants were killecl by the esisting test seqtiences. In surnmary .Ifter
ohcaining 100% miitation covernge no new test cxes were requircd to kill any nf thc siihç~qtrentIy
generated mutants. Table 9.4 shows the summary of the resiilts. Xo issues aerc idcntifierl witti ~ h i s
c;rpsiile.
9.6 Experiment 5: Radar Control Unit (RCU)
This capsule w u estracted [rom an electronic warfare simulation developed by gaciiiate stuclerits
at RSIC. The same procedure used in rsperixnent 4 was followed with this experiment. Kitli L0[1%
mutation coverage (n-hich required 2 test sequences) the -DS/E/T mutants were nrided. -4nalyzhg
tiie live ADT mutants lead to the ciiscovery of tn-O dead code segments. Adding the TRD and TXT
mutants did not prodiice any live mutants. hdding the TC.1 mutants resuited in a single Iir-e mutant
which %-as kiIled by the addition of a thiid test sequence. Adding the TTG and other remaining
mutants did not require the addition of hirther test sequences. Table 9.5 shows the sumrnary of the
resdts.
scates pseudo-states levels of hierarchy ports in signds cirit signalç transition actions
Miscellaneous Srirnbcr of Test Sec~uences letigths of test sequences longest test sequencc
cornpite rime (al1 mutants) 33 minutes
11 Mutation DetaiIs 1 Il Nurnber of Mutants I ! -
hDS/E/T coverage ITS/STI TRD TCP XCP TGT T - T TCA TTG SDL Testing Details - mntants generated 1 .LU0 niirtants compiler1 364 cquidenc intxlitl 33 ! valid j 965 rIeatl. firm SG5 mutation score. finn 1 ~ 0 % r1earl.s~rong 363 mutation score! strong / LOO%
Table 9.4: Experiment 4: Cruise ContmI
9.7 Experiment 6: Pop Machine
This motle1 was developed and tested by the author as part of this chais. Thii capsule tias 3 chice
points nrid a single tirrier. This çapsuk was incIuded in order to inwstigace the ability of a capstile's
tlesiger to test the capsule. As mentionecl in chapter 2. sorne cesting literattire suggests tbat an
arithor of a program crinnot cffectiveIy test his o m work because testing to rereal firuits is basically
a destructive process. It is feIt chat the author of a program is biased torvard riot destroying bis
onn creation and nîil therefore not be as effective at breaking it. intuitive. hon.ercr. mutation
testing stiouId be inunune to auy such bias because the niincisec of =any tester (inclutlmg the capsuleS
author) is to kill mutants. Table 9.6 shows the s u m m q OC the reçults.
8 faults n-ere revealed through mutation testing of this capsule. The CbT. TBD. ml TXT
operators each ret-eded a single fault. The ADT operator rewded tn-o. and the TCA operator
revealed three- EWng mutants resulted in the creation of test sequences that esercisetI non-intuitive
control paths. For esampIe when adding a test seguence to Hi a TC.4 mutant it T;LS riiscovered that
it rvas posçibIe to bu- a second pop for haIf price: X single cm of pop costs one douar. It was found
that by imerting 6 quarters into the machine foverpaying) and then p u s h g the pop select button
Capsule Details ' transitions States pseudo-states levels of hierarchy ports in signals out signals transition actions
Misceiianeous Yumber of Test Sequences lenghs of test sequences longest test seqiience
cotiipile cime (;dl itiiitants)
processor
13.6,3 3 secs
3-1 minutes
. - .
Number of Mutants ADS/E/T coverage ITS/STI TRD TCP SCP TGT T-IT TCA TTG SDL -- -
Testing Detriils mutants generatecl mutants compiIec1
intalid d i c i cleaci. firm mutation score. firm cleadstrong mutation score. strona
Tablc 95: Esperirnent 5: Radar Control Cnit
the ciistorner ~votilcl receivc a can of pop and two quarters in change. Then the customcr only had
to insert two more quarters in order to receim a second can of pop. This type of faiilt is typicaI of
niistrikes niade by software clevelopers n-ho assume chat their software wilI be risecl in cxpected ways
and it is typical of the types of fauhs that a code's developer ail1 have trouble in fincIing plyers
Ï9][.Jaiote 971. dlthough this is onIy a single example. the Fact that such an interesting fauit n-as
founcl (the half pnce faiilt) indicates that mutation testing may mitigate the psychological argiiment
chat an aiithor cannot effectively test his mn software. This is an interesting subject for fiiture
research.
9.8 Experiment 7: Electronic Warfare Jarnmer
This capsule was e-nracted from an electronic d a r e simulation developed by graduace students
at RCIC.
The same procedure used in experiment 4 n-as foUowd with this eqmiment. Xith 100% mu-
tation coverage (which required 3 test sequences) the JLDS/E/T mutants were added. X fourth
,, t a t i o n Details I
I l Number of Mutants 1 4 states pseudo-states hreIs of Liierarchy ports iri i;ignnls oiit signnls transition actions
Misceilaneous Sumber of Test Sequences lengths of test sequences longest Lest sequence
cortipile tinie (al1 niiitants)
Table 9.6: Experiment 6: Pop Machine
15 43 6
27 6 3 1
104 310 LI11 14
6x2 574 25
1-4 ADS/E/T $8 coverage 4 ITX/STI
3 4- 1 timer TRD w
I TCP S XCP 13 TGT
T-AT 1 TCX
processor
test scquence n;ns aclded to HI the 8~ ADS/E/T. mimnts. .kithg the remaining rnutanls cliti not
reqiiire the aridition of Iiirther test sequences. Table 9.7 shoxs the surnmary of the restdts. So
issiies werc fottnd wich chis capsule.
inralirt valid
9 25.26.2.15.10.3.15.2,4
3 secs
36 minutes
9.9 Other Capsules Examined
TTC SDL Testing Details mutarits generateri ~iiiitants compiIed erluisalent
Pericium LI1 ï33h[Hz
9.9.1 Radar module
This c a p d e riras taken fiom a mode1 deveLoped by some gaduate students. Due to the poor qudity
of the code tliiç experiment n-aç not completed. This capstde indicates that mutation testing shouid
not be used on s o f m e that is not at some minimal Ievel olquality. Each issue Eound requires çome
corrective action be made ta the code. foliowed by regeueration and rerunning of aii of the mutants.
This is estrernety expensive in terms of computationd and human resources. Table 9.8 shons a
summ- of the initial results.
deatl. firrn mutation score, 6rm
.540 99.63'2
rleatl.strong 532 mutation score, strang / 9%
Capsule Detaiis tr.msitioiis stntes pseuclo-states levcls of hiernrchy ports in signais out signals transition actions
~Miscellaneous Xumber of Test Seqiiences lengths of test sequences longest test sequence
corripile tirrie (al1 mutants)
Mutation Details , I 1
-
- Y
Y
13 6 39 4
3 -i 5 cimers 6
Number of Mutants
coverage 23
Table 9 . 2 Eqcrinienc 7: E\VS .Jamtner
,
ITS/STT TRD TCP SCP TGT T.kT TCA TTG SDL Testing Details I
9.10 Individual Operat or Discussion
- J
4
4 6.9.15.9 -
I secs
65 minutes
Celcruu 3GGUHz
-L j 13 O O .>
26 4s
-5 I
mutants generated mutants compilecl eqii ident invalici
Ttie tiest sectioti clisctisses obscnations made concerning intiiviciiial operators.
254 230
O 26
9.10.1 Coverage Operators, CVT, CSE, CSX
mlirl WQ
clcatl. finri E S mutation score, firm LOU% cIcntl.strorig
l -- .,sg --
mutation score, strong LUO'X
Thcse opcracors are designed to show chat dl transitions are necessary. to show chat al1 states are
reachable. and to iclentify an!- terminal states in a capsule. Creating test cases to Hl the coverage
operators is relatively e a s . Test sequences that brirr ail O€ the coverage operators also kill man>- of
the other mutants. in some cases .di of them. In the capsuies tested 2 faults nere revealerl by the
CVT operator.
The importance of statement coverage has been discussed in chapter 2. Xthough 100% stacemenc
coverage is not often targeted in industry. it is deemed important in studies of acfec~uacy criteria
and for cr i~ica l"~ softmre for the reason that in order to have confidence in critical software it is
'fiCritical softwue is usually IabeIed as safety, mission, or Iife critical. In generai it refers ro any softrmrc in which the consequences of failure are high. i.e. I o s of life, hedth. money, resources.
pseudo-states levels of hierarchy ports in signals out signais transition actions
Capsule Details
co\'erage rrs/sn TRD TCP SCP TGT T.T
1
lcngths of test seqiiences longest test seqiirnce
Muta t ion Deta i l s I
Miscellaneous Yumber of Test Sequences
compile tinie (al1 niiitants)
Number of Mutants 1 -U~X/E/T 1 9
transitions
proccssor
9
1 [ -4
i secs
States I 4
T C I TTC
34 minutes
Celcron 36GXIHz
Table 9.8: Espcriment fi: Raclar
SDL Test ing Details mutants gencrarecl
necessary to know, tiiirt as i l rriinir~itirri, every fiiriction of the softxare has beeti escrc:istid nt l e s t
once. The rtiiwon for not obtainirig 100% statement covcrag in industriai code baseci testing is ttiat
the importance of iincowrecl staternents rnay not bc mortti the additional cost of covering thern. For
example (deacl) ctebiig code may not be rvorth testing. Often a piece of software wiII have sonie code
that is intencletl only for use cIunng deveIopment. Such instrumentation code may be -deactiva?itecl"
in the procluction version thorough a compile time snitch (Le. #ifclef DEBCG statenient in C).
Aclcling test cases to obtain coverage for such code segments is probably not worth the cost.
4
1-11
In ternis of CSJIs. however 100% coverage appears to be more important. The preliminary
residts of the esperirnents described in this thesis suggest that test sequences that kill ail coverage
operators also kill the majority of the other mutants. By concentrating on killing coverage mutants
first. the number of subsequent Live mutants is greatly reduced.
C'sing mutation to achieve coverage can be inefficienc becaiue each CSBl rcsuits in 2S'T niutants.
=here S represents the number of states and T the number of transitions in the C S X One mutant
is protluced for each transition and two mutants are produced for each state (state e n t - state esït)
in the CSM. Altematively a singIe instrumented version of the CSM couId be produced in n-hich
each transition and state entry-exit has code segment h a Instrumentation code added that recorcls
the fact that the code segment has been euecuted. However this has implications For thnimg. The
mutants compiled 1 124 equi\aIent 1 ..' inidici valirl deacl, ürm mutation score, firm
17 124 74 60%
instrumentation adds a run time cost to every transition cmcl state entry-exit code segment. This
riin time cosc cloes not occiir with mutation operators: the capsule execuces norrnnlIy (Le. there is
rio instrumentation penalty) until the mutated transition or state is execiiteri at which cime cnpsuIe
esecmion stops. In order to avoid the tirne lags produced by instnimentation it m- be n-orth the
cost of creating 2S*T mutants.
9.10.2 Action Delete Operators, ADT, ADE, ADX
Tlic action clelete uperators arc clesigneci to show chat al1 code segments in the CS)[ have an effect
on the behaviour of the capsule. If after deleting an action from a CSM the potentid1i2' ohsenable
behavioiir of the capsule does not change then the action is unnecessary or is deacl code.
In the capsules testecl in this thesis 5 issues were fountl bu these operators. Aniotig these issues
= test cases to were dead code segments and preclicates that never etailiate to mie. While creatiti,
kill Iiw action clelete rnutants. intcresting and non-intuitive paths through the state machine are
tlcvelopcct. The prelirninq resiilts observeci in this thesis indicate that the action rielete operators
;irp hcttrr at revealing subtle (srnall semantic size) friults than any of the other operators used in
tliis thesis. The valiie of these operators is also enhanced becanse they gncrate very few mmutants.
one for each action in the capsule. The iiurnber of actions in a CS&[ is at niost 2S+T (eacti state
can have tn-O actions, one for encry, one for exit. while each transition c m !lave a single action).
however the total number is usually much less.
9.10.3 TRD: Transition DeIete
At tirst glance the TRD operator appears to have the s m e cffect as the C t T operator in thnt a
test secluence that kiki a CVT mutant siiould kill the corresponding TRD mutant. However th-
are not the same as was shown in the first experïrnent.
Assuming that 100% CVT coverage has been obtained. a live TRD mutant ah-s indicates
an issue with the CSM. The C\T coverage indicates that the transition has been cakeri. but the
Iiw TRD mutant indicates that rernovïng the transition does not affect the capsde's obsenable
behavioiir. This is a contradiction. On the other hand issues reveded by the TRD operator are not
necessarii- fadts. As shown in e-qeriment 1 a Live TRD mutant may indicate redundant code.
"It may be necessq- to observe the CapbT for sorne time after the execution of the rnutated elernent in oder for the effect of mutation to become observable-
Although TRD selclom indicates something not s h o w by CVT it can be useful. in chi thesis
two issues nere founcl bu the TRD operator.
9.10.4 SDL: State DeLete
t\lniost al1 of the mutaiits produced by th operator either clici noc compile or ctirew a riin tirne
exception. Dcleting a statc with a contained scate machine dwq-s prorluced a run time crror from
ttic RoseRT code gerierator engirie. Therefore only IeaF states could be rlcletcd. Howevcr rniitants
gcricratetl by tliis operator w r e eil~iily kiUe11 and tlid not reveal any faiilts iirriong the six çiipstlles
tisecl in chis thesis. tt appears chat deleting x Stace is a large seniantic fault. An iniprovetl version
of tliis operator coulti be wricteti in orrier EO reduce the nt~rnber of conipilatiori tmcirs atirl riin titric
exceptions that occur. Such an improveti operator would onIy clelete leaf states. where i i leaf st;itcb
is a state that does not Iiave a contained statemachine.
9.10.5 Internai Transition History Operators ITX (Internai To extemai),
XTI (externa1 To Internai)
These operators do not appear to be very usefii1. They do not generate many mutants and the unes
generatecf are often equivalent to thc original. Eqi~i~alent mutants are protlucetl ahen the states
tvithin the scope of the internal transition being mmated do noc Iinve entcy or exit actions. in siich
cases there is no behat-ioural clifference becween an internal and external transition. S o issues tvere
rcvealccl by chese operators during this thesis.
9.10.6 TAT: Transition Alter Target
This operator rnodels the transfer errors rnentioned in Choa's error classes [Chow 781. It apperirs
that most T-IT mutants are easily Wled by test sequences that kilI aU of the coverage mutants. On
average the T-IT operator generates more mutants than aU of the coverage operators combined but
l e s than the number o l mutants generated by TCX or TTG. During the eqecïments this operator
revealed a single issue. W d e t u h g to kill a TCX mutant the question posed by the mutant n;as
-1t doesn't matter nhether this transition goes to state .A or to this choice poiut. a - is that?
The answer was that by going to the choice point instead of state h the condition e d u a t e d by the
choice point always evaiuated to F-USE. However the FALSE transition leaving the choice point
also nent to state A. Therefore the behaviour of the mutant waç equivalent CO chat of the original
capside: the result of taking either the rnutated or original transition was to arrive a t state A.
9.10.7 XCP: exchange Choice Point
This operator generates few mutants. one mutant for each choice point in a capsule. It has the effect
of reversing the boolean espression controlling the choice point. No issues nere itlentified by this
operator cluring this thesis. However. since the nimber of choice points in the collection of testeci
capsi~les was sniall. a nieaningful analysis of this operator coiilci not be niacle.
9.10.8 TCP: Transition Choice Point
This operator generates two mutants for every choice point in a capsule. It ensiires that t d i brandi
(if ii rhoice point is trriversed. It appears chat TCP mutants are easily killeti anri therefore represent
Inrgc sernaitic faults. They ttierefore do not appear to be very goocl a t rereaiing faults in n capsule.
Hoaever not enough choice points were obsewed in the capsules of this thesis to permit a nieaningful
anaiysis.
9.10.9 TGT: Transition Guard True
Tliis operator niutates the expression bgmditig a transition by Iiaving it alw-s return tnic. Gttnrdetl
transitions seem tu be rare. only one &-as found in al1 of tlie capstllcs examiner1 (luring this thesis.
S o analysis of this operator can be offered on the basis of this thesis.
9.10.10 TCA: Transit ion Change Action
This operator generates on the orcler of T*A mutants per capsule. where il is the riumber of transition
action objects present in a CSM. This is usrially niore than an>- of the other operators used in this
thesis. However rnost of the TCX mutants are triviallq- killed by test cases that kiii the other
mumnts. However the remaining ones can reveai Fauits. in this thesis three issues were reveaIed by
TCX mutants. X good scrategy appears to be to kiii the TCA mutants aFter di of other mutants
have been kiiied. The idea being that the test sequences used to kilI the other mutants niil also Hi
niost of the TCX mutants. Then creating test sequences to kiii the remaining TC.1 mutants may
reveal faults.
TC.-\ also generates many syntacticdiy ilkgal mutants. When the action object for a parameter-
izecl transition uses the parameter then the parameter type of the message must match the parameter
type of the action object. Khen a mutation is performecl in which an action object e-xpecting a pa-
rameter ol type T is ,~ssocintetl with a transition triggered by a message with a clifferent parameter
type the resiilt is syntas error. A transition's parcameter is always accesseci by casting a voict pointer
namcti rtdata to the rlesirecl type. Since the identifier rtdatn is aiways present in the code segment
of an action object that accesses ii parameter it is possible to automate tlie elimination of this type
of syntacticaliy intalid mutants at the mutant generation stage. Hoaever this is Ieft for f~itiire work.
TCX also tends to gcnerate a fen- equivaIent mutants. When tao or more action objects arc the
sanie. which is often thc case if the action is a simple message send. ttien TC:! sri11 generate a pair of
tvpivalerit niiitants for racti pair of eqirident action objects. Hoaever some automatecl eqiiiralency
checking is possible. TSL [Cordyt 001. for example. coiilcl Lie tisecl to compare action objects bcfore
the mutants are generatetl (by parsing the action object and performing non-terminal comparisons).
9.10.11 TTG: Transition TriGger exchanged
Scst to TCA die TTC operator generates the most niritants per CSAi. in a aue wliere a CSM uses
aiany clifferent triggers the ritmber of TTG mutants can beconie dominant. However despite the
large nimiber of TTG mutants. niost are trivially killed by test cases createti to kill other mutants.
-4s with the TC.-\ operator a good strategy is to not use the TTC operators untiI most of the other
operator\ mutants have been killed. The TTG operator did not reveal any issues among the capsules
usecl in this thesis.
Originaily it n;1s decided that TCA would be applied to ail transitions. even those tiiat dicl not
have a tcrigger in the original. The rationale n-as that triggeriess transitions n-ere probably the kinds
of faults a e were Iooking for. This iJ an inefficient po t c - There are many transition segments that
correctIy clo not have triggers. These are continuation se,pents and se,gnents that Ieave choice
points and go to States. XppIjing the TTG operator to a continuation segment d l result in one
syntacticaiiy legai but invalid mutant for each signal that can be appiied to the capside. ThÏs
can represent a large nurnber of inmiid mutants. Tt appears that untriggered trans-itions are rare
in practice. represent easily found faults, and muld be found by the CVT operator in an>- case.
Therefore any future work should not a p p l the TTG operator to transitions that rio not aIready
have a crigger.
9.11 Operator Summary
Typicdy rvhen perforniing mutation testing, killetl mutants are removed from hr ther testing as soon
as they are killetl. However as this thesis was investigating the effectiveness of mutant operators. it
~r-as neccssary to run al1 tnutarits against ail test seqiiences even after ri. test secluence liacl killed a
mutant. -1 mutCant that is distinguishecl by mnny tests is easily killable ancl probnbly rqresents ii
large serriantic fault. X mutant killcci by a lotv tiuniber of tests is probably r i stuall setnaritic fault.
tvhile orle ttiat is (leclarerl eqriivrilent is a size zero semantic faiilt. The intention here wns rn see
dietlier a particiilar operator gencrated rnucarits ttiat tvere on average eitiier tu kill tlian tiiose of
;inother opcrator.
hl1 of the operators produced mutants that were killeri by al1 of the test secluences (inrlicating
large semantic faiilts), mut~ants chat were killed by a single test sequcnce (semantically smaLler
faults). .mil mutants that were killetl by multiple test seqiiencw. Table 9.9 shows tht. riiinilic~r of
test scqiiences (expressetl as a percentage of al1 of the test sequences run against a given mutant)
reciiiiretl to kill the givcn mutants. For esample the table s h o w chat for the pop machine rapsrilc.
tlic 45 (-overage rniitants were killed by an average of 70.3% of the 8 test sequences. The top rnn-
.ihow the nnmc of cach capsule tested in this thesis. In the same ceIl but undermath the namc is
the tiuniber of test sequerices that were run against the mutants. The first column shows the nanie
of the mutant operator(s) and the number of mutants protluced by that operator(s) for pach i,apsule
testerf. Sot ai1 of the operators proposed and used in this thesis are inclurlecl in this table. Oiily
the operators that produced a large number of mutants are includecl. Yotice that the action rleIete
(XDT. ADE. XDS) and coverage (CVT. CSE. CSS) operators are grouped.
On average mutants were killed bu 68% of the test sequences. The game of Iife ceil capsule was
relatively simple which accounts for the relative case with n-hich its mutants were kilIed. There are
a ïea results that are far from the 68% average. these are marked in bold. The average column
summarizes the data for each mutant operator. This coiumn allovs the operators to be ranked
accordiig to the number of test sequences required to kilI a mutant. lccording to the table the
TCX operators produce mutants that are the easiest to kill. R-hile the coverage operators produce
mutants that are the hardest to Hi (in terms of the number of test sequences required). However
the tlifference between the average for any particular operator and the total average of 68% is not
significant. Further. dile to the small sample size. no statisticdly significant conclusion cnn be made.
Thercfore baseci on these results it is not possible to say that one operator produces mutants that
are on average easier to kill chan the miiteants producecl bu any other operator.
Table 9.9: Percentage of test sequences required to kill mutants
Capsule
So.Tests ADT/E/'i Mutants Coverage .CIutnnts
1 T.T Mutants
' lfiitants TTC Mutants
Hoivever these preI iminq results indicate that some operators ni- be more effective a t reveding
faiilts than others. The operators appearing to be the rriost effective a t reveding faults arc tbc
çoverage rint1 action delete operators. Moreover these operators gencrace reIatively small niimbers
of mutants. The TCX operator. althotigh generating many mutants. also appears to bc goocl at
revealing faults.
P B S
3 66.7 6 --
r .S 1s 76.0 26
Traffic Light a 73.3 12 70.5 40 54.8 U 62.6
9.12 Procedure
TRD lii~tarits , 1::*3 1 4 4 TCA 1 75.0 1 60.2
27 7-1.0 16 103
Tt is difficult to say that in generaI one operator is better than another a t revealing faults. Ho%-ever
the coverage operators generate a srna11 number of mutants (linear in S and T) and are relatively
easy to kill; just ûnd a test scquence that traverses the state or transition. Test sequences usetl to
kil1 cwerage mutants tend to kiii large numbers of mutants generated by other operators such as
TCX. TAT, and TTG. These non-linear operatos can generate hundreds of mutants each for even
relatively srnail state machines. most of which are noise. By using the coverage test sequences to
Cniise
4 60.0 13 34.4 34
total average total mutants
161 70.0 36 337
Ce11
9
78.9 7 4 . 1 17
18 47.9
68.1
27 60.0
1944 i
62.0
RCC
3 72.2 S 83.3 16
11 85.0
136 1 70
61.4 104 67.0
13 32.2
S 61.0 2s 52.0 40 116
66.8 131 380
95.0
Pop
8 42.3 1s 70.3 45
60.3 1 I
S5.S 35 163
66.7 71.2 26 62.2
48 1 LU 1 16
310 70.3 101 602
69.6 I
68.9
.Iammer
4 48.4 12 1 . 7 23
62.5
48 54.1 121 243
Average
63.1
91.0
Total 5Imants
66.6
71.8
1 73 74.6 / 183
780
483
ki11 the majority of these noisy mutants, the tester c m focus on hilling the remaining non-1ine.v
mucants which seem to mode1 semantic&- smdl faults. This reduces the burden on the tester-
In the first 3 experiments a.ü O€ the mutants werc gnerated and nui agaiast ail of the test
secltiences bcfore any analvsis ivas periormed. During analpis live mucants were examinecl in ari
arbitra- order. new test çecpences were aclded. and ~ h e live mutants were rim against al1 of the
ncldeci test sequences. Adding a new test sequence CO Hl a particuIar mutant often haç the side
effect of killing other live muttmts. For this reason tn-ing to Ml di the rernaining nititants in ri single
iterntion n-as ncver atternptcd. ïnsccatl iisudy only a single nen- test sequcnce ainiccl at killing 2 or
3 live mutants rn added during cach iteration.
The proccss of riinning a11 of the mutants each tirne is inefficient. If a fniilt is foitncl r.hcn the
cnpsiile mtist he repurctl, the mutants regenerated. and the tests r e m . \$%en a fatilt is foiinrl earIy
in the analysis the hitlk of the results are not iised. W t h small capsides the nuniber uf mutants
genera~ctl was on the order of 100. In these cases thc inefficiency of this methocl is tolerable.
However when the number of mutants is Iarger a more efficient process is neecietl. Starting ai th the
4th experirnent the follo~\lng process was folloued:
L. Cienerate the roverage mutants (CVT. CSE- CSX) and add test sequences. (itcratively) imtil
LOO% coverage is obt'zined.
2. Generateaction mutants (XDT. ADE. XDS) and add test sequences to El1 al1 the nea mutants.
3. Generate the TRD mutants and add test sequences ta H l ai1 of the neiv mutants. Since TRD
is almmt the same as CVT the number of live mutcuits produced bu this operator is mai1
(tisualIy zero).
4. Generate the rernaining linear operator mutants (Sn. ITS. SCP.TCP. SDL. TGT) and acld
test sequences CO kir1 ail the new mutants.
.5. Generate che TAT mutants and add test sequences to kiii a11 ne%- mutants. TRD produces
manj- mutants, on che order of T*S.
6. Generate the TCX mutants and add test sequences to d nev mutants. This operator
produces man>- mutants on the order of T f A . where A is the number of action objects in the
capsde. In the fen- capsdes studied in this thesis TCh encourageci the creation of subtle but
legal sequences.
7. Generate the TTG mutants and add test sequences to Ml al1 new mut~uts . TTG protluces
many miitnnts. on ~ h e ortler of T*mumber 01 in signnls.
3Lny tirne a faiilt Li found the procetlure is stoppetl. the capsule repaired. and the mutation
proccctiire starter1 over. Ki th this procediire the noisy rniitants producecl hy siibseqiient operators
;ire likcly tri bc killed by the existing Lest sequenccs. Therefore racher chan facing a large nuniber of
mostly noisy live mutants. the tester only deals with a smallcr number of progressively hirrcler to kill
mut.urts nt e c h iteration. Further the total number of mutants tbac musc be çorripiled cliiritig t h t h
cntirc testing phase may be recliicerl, Ench tirne a Fairlt is foiintl ancl correctcri nrily the tniitants that
Iirn-t! n1re;irly bceri tested rieetl to be regeneratetl ancl cecompilecl. By killing cbe coverage niiitarirs
Hrst. che mniher of other. l e s intuitive. rniitants to be killec! is recliiced. Thiis both the effort of the
lester itiid t.he cunipiitatiorial biirden arc rediiced. It is intcresting thnt a siniilar strategy thas not
been proposetl elsewhere in the mutation literature.
9.13 Initial Test Sequences
Since ;il1 niiirants ;ire live prior to the running of the first test sequerice, initial test setpiences are
not tieçtissarity crentetl for the express piirpose of killing porticular livc mutants. Rathcr they may
reprt'sent espcctetl uses of the capsule. transition toius. or possibly are randoni. They riiay be
basecl on sperifications. Hontever most of the capsules testetl in this thesis ntere not accornpnnierl
by specifications. The initial test sequences used in this thesis were baser1 on trarisitiori totiI+".
~ i t h o u t regard for whether the resulting test sequence representecl a reasonable itse of the capside.
Trnnsition tours are usually easy CO create liowever complications a i s e tvlieri transitions have girards.
dien choice points are present. and if internai events (timers. intermpts) are possible. After rtuining
the mutants against the initiai test sequences other "mutant kiUingFtest sequences are addeti to !dl
an'. remaining l i ~ e mutants.
Test pIanning issues were not a significant factor in t h k thesis due to the smalI size of the capsules
tested. Bowever in larger projects test planning rnq- be important. Good test plCanning requires
some structure amongst the tests. Sets of unrelated test sequences generally are poor choices. since
there rvilI be a lot of diiorganized repetition in the sequences. Good test planning recognizes this
and es structure to the test sequences [Shepard OL1. The transition tours usetl to kill coverage
"A single transition tour is not dways possible. In this case a set of partiai transition tours is needed.
98
mutants are a gooti bais for test planning: their segments can becorne preambles for other test
scquences that need to be at some -pseiido-initial' state.
Tu ki11 a- nliltant requireç that the mutant be reached by the test sequence. Conçider the A D S
(Action Delete state exit) mutant. To observe a Jiaerence UL behaviour it is nccessary chat, a a
minimuni. the state chat has had ics exit action tleleteci hy this operator be exiterl. This reqiiires a
test seqirence whoçe prefix pirts the CSM in the mutated scate. Then the tester ndds more signais to
the -pre.arnble" test sequence in order to produce an absenable clifference in the mirtant's hehavioiir.
ctius killitig the ADS rnutant. Segnients of transition cour resr seqiiciir:es cari lit! I I S E ~ i f i clie prwrr~hle
tu piit the CSSI in the "psendo-initial" state.
9.14 Killing mutants
Crcating test secluences to MI niutants rva the rnost labour intensive part of perfornii~i: ~tiritation
testing during this thesis. [t reqrrims matching the lire mutant name to the stnte clingram eierncnt
ttiat h a been mutated and cletermining what the mitcation il;. For linpar operators the nnnie i iF thc
rriiitant tells what the mutation is. For non-linear nperators siich ns T.T the operator's log file musc
t)e rromiulteci. Log files correlate the rliucant nuniber with the details of the mutation. For esample
the T..U log file wili show the origind and aew t a rg t state [or a transition. After cletermining vhat
the mutation is the cester miist constmct a test sequence that will kill the miitanE. For operators
stich as the coverage operatos this is relatit-eIy E . However finduig test secluences EO HI niutants
produced bu operator such as TCA and TTG rcquires a more creative effort.
Better tools aoiild eliniinate che tediousness of tuatchi~g a live mutant to the iicttral stiitc rliugram
elcnients thrit have heen mutated. Hovever the general probIeni of test sequence generncion is
irndecideabte and must be done rnanualIy. Rhile in the process of creating a mutant kiIler one
nnswers -hard questions' such as T v e deleted thîs action object. but it doesn't rnatter. why is
that:'" Xnswerïng this question leads to a new test sequence that should Ul che rnucant or reveai
an issue wïch ~ h e originai capsde.
hocher decision to be made is whether to create a single minimai test sequence to WI each
mutant. or whether it is better CO creace larger test sequences designed to kiU severd live mutants
(Le. combine short sequences into longer ones). From an understandabilis point of rien. short
minimal test sequences are preierred. These are easier to understand and easier to cun through
an oracle. On the other hand longer sequences generate more interesting paths through the çtatc
machine that rnay kill other as yet untargeted Iive mutants.
On çewral occasions a test sequence did not Ml the targeted mutant. .hnlysis revcded that
propagation of the fault caused by the mutation required that the capsule be obsemed for somr
tirne after it had received it's 1 s t input. This was the case with action rlelete operators in which
an action cariceling a timer was tleleted. Since the .'deleteci' timer was still active the capsuic hatl
co he obsemecl for. in one case. 5 seconds. before the timer expired causing a tirneoirt event which
in ciirn caused the capsule to emit an observable response. In test sequences chat rlid not waic long
cnough the capsule was terminated before the timeout could occur with the result that the incorrect
behaviour (an unespected response and possible state change) m not obsemcd.
9.15 Equivalent Mutants
One of the tIisatlvantages of procedural mutation is the hurnan cost of identifying eqiiivdent mutants.
This tloes nct seem tu be as large a problem with capsules. In the capsules testet1 in this thesis it rvnç
round that in several cases two or more transitions had the same action object (i.e. the C+L code
bodies tvere functiotially equivalerit. These were often simple functioii calls to cancel a tirrier 1. Khen
these capsides are rnutateti with the TCA operator each pair af eqirit;rlent action objccts generatcs
2 eqiiivdent rnut~ants. A visual inspection quickly and reliably reveals die eqtiitxlence. This type of
equim1ence checkng could also be ,zutornatecl using a tool such as TSL that is nblc to comparp tn-O
tion-terminais ahile ignoring both white-space and comments. Other cquimlent mutancs loiinrl in
these capsulcs incluclcd:
0 a transition chat could target one of two target statcs a i th no chnnge in behnvioirr.
0 repeated ~ar iable assignments (Le. a variable assignment made in one segment of a transition
ctiain is added to another transition on the same transition c h a h (TCA operator).
9.16 Finding Faults
The mutation fiterature seldom discusses what to do when a fault is found in a program umler test.
Should ali of the mutants be regenerated based on the new version or oniy the ones that nwe stil1
live? in this thesis when a fault found in a capsule the fault was fked. d l of the mutants
were regenerated. recompiled. and d l of the mutants aere run against al1 of the test sequences.
Basicaily the mutation testing was re-started from the beginning, the only riifference being rhat the
test sequences created in the previous iteration Rere present a t the start of retesting. This is an
extremely time consurning step. On a slow cornputer such a s two of the three iisccl in this thesis.
reconipilation can take 10 to 20 seconds per mutant. To \didate the necessity of rerunning formcrly
dearl mutants the results of mnning the existing test sequences or1 the buggy version were cornparecl
n-ith the results of running the same test sequences on the -repaireci" niiitants. In most cases one
or niore of the forrnerly deari rriutmts w x a live mutant when briset1 on the correcterl capsiik.
9.17 Strong vs Firm Mutation
\Vith strong niutation only the outputs of the progarn uncler test are comparecl. internal r l a ~ a scates
;ire not examinecl. Witii both n e d i and firm mutation ttie internal data state is examinecl. [fat any
time the data state of the orignal ancl a mutant differ then the mutants is weakly (firmly) killccl. in
this thesis a form of firm mutation testing was performed. Minimal instriinientation was arlrietl cn
cadi mutant that allows the TestHarness to que. the ciment state of the CnpCT. and also co infonn
the TestHarness whenever the CapCT received an iinexpecteri message. An ttne.upectet1 messagr is
a message that does not match any of the t r igers of any ciurently enabled transitions. ivithorrr.
adcling the instnimentation, unelrpected messages are tiiscarded by the CapCT. tF the otisen-ability
option is in use then an unexpected" message is sent to stdout. However if obsen-abilit- is not
enabletl theri the uneqected message cannoc be observed. X typicd result tiIe for an instrunicntccl
mutant is s h o m in the Ieft column of TabIe 9.LO. .Ili of the testing performed in ttiis thesis tisecl
instrumentation.
in order to caIcuIate the strong mutation score. the results file sxs fiIterec1 to remove ar- Lies
generated by the instrumentation. The right coIumn of Table 9.10 shows the results of filtering the
left colurrin. in al1 seven experïrnents the test sequences that yielded 100% firm mutation yieIded
aIrriost 100% strong mutation. Xctuai resuits ranged Gom 97% to 100%.
Weak and firm mutation testing vare orighaiiy proposed as methods of improting the efficien- of
mutation testing. Efficiency was improved by terminating the executing mutant as soon as a mutant
was found to be weakly killed. in the mutation testing of passive capsuies the potentid swings
to be realized is less since the major computationai burden is compiling the mutants. Koaever
wdïing TESTCOSISISST-4TEREQCEST CapCT in state:S3 sencling TCCCO!dlKFIRE paiweTime :-'LOO0 Driver:Catcli?ill Ru CapCT mg: firc Driver:CacchAU RK CapLT m g : n-atchDogReset ?TXL msg, timwut: Xo rcsponsc GomCapCT scntling TESTCOSIIIISSTATEREQC'EST CapCT in scace:S3 sending WSiSCOM S1SSE'I'DRJJ)ARJIRRAY ?;ILL msg, timcoiitr Si1 response fromCapCT CnpCT in stat~:S3 scnrling ~V5ISCO.\ISISSE?iDR.~DARARR.4\* iinexpccterl Oitrcr rnsg: timeoiic S L Z L nisg. timeoiit: So response froniCapCT ~cricling TESTCOSIJ[SST.TEREQL'EçT CapKT iu state:S3 mirling ~VlISCOSf~ISSE?;DE~VSS.\RRRR4Y iinexpected Oiitcr m g : timeoiic XCLL mg, timmuc: ';O rcspowe fromCapCT sending TESTC03IMSST.-\TEREQLTST CapCT in state:S3
sending TCCC0MMSEiR.E paiweTimc :-?O00 Drivcr:Cat&411 Ru CapLT msg: h c
Drivcr:CatdiAiI RK CapUT mg: w~tchDogRcset 'UTLL m g , timmut: 30 rcçponse fromCapCT
scnding iVlISCOXiSISSESDR4DARW.iY ?iTT.L m g , tirneout: Xo rcsponsc FromCapCT
scntling iVlfSCOXIliSSE3?)RiDARR4RRRlY
X L L msg, tirneout: So response FrornCapCT
X Z L rnsg, tirncoi~c: So responsc frnniCapCT
Table 9.10: Firm and Strong mutacion restilts
~ w a k mutation testing of ciynamic capsules has A grenter potential for tirne savings as therc is the
possibility of nvoicling lang timeout cvents. in either case it a p p e m that test secpierices chat firnily
kill mutants are d s o sufficient to strongly kill the same mutants. Sirice the purpose of niutation
testing is to develop strong test cases the question of we~i / f i rm vctsiis strong mily h~ ninot in ehcl
contcsc of capside testing. On the other hancl the estra state information collected ditring chc firm
rriutation testing pefiormed in ttiis thesis can provicle information that tieIps the tester create niutant
killing test sequences. This may be its greatest value.
Al1 testing methods require an oracIe against nhich the result of each test case is checked. As
mentioncd in Chapter 2 oraciiig crut require extensive and expensive humm effort. Mutation testing
offers a way to reduce this cast. Diiring mutation testing test cases are adclerl to the test suite in
order to kir1 live mutants. T'picaüy during mutation testing the resuIt of each neu- test case must be
vecified. via the oracle to ensure that the original program is correct. Hofiever it is possible to add
new mutant kiiiing test cases n-ithout first going to the oracie. It is Iikety thac some of the nem test
cases may not rwrk as pIanned and not kiU m y Live mutants (This occurred several cimes riuriag
CI& thesis). The results of these useless test cases do noc nced ta be verified. rather the test c x e
simiil(1 be tliscarcled. Oniy when s test case has shom itself to he uselul (Le. it has killed mutants)
shouId it be checked with the orade.
Only one of the capsules tested in this thesis was accompanied by specifications. The correct
behaviour of the other capsules &-as decided by the tester. Correctness of each test n-as rietertnined
on the liasis of thc results file that wx the result of execiiting the original capsde againsr the test
squcnce. Thcsc files contain a Iot O€ information that is usecl in firm mutation cornparison but is
rioc needed to clecirie correcmess. Since correctness is juclgecl strictiy on the basis of the ubservabIe
input-output sequence the results files were filtered to remove the instrumeritation data. Corrcctncss
of ttie original çapsiile nas determinect basecl on the fikered files.
9.19.1 Number of Mutants
In general i~ appenrs that state machines rio not generate as uiany mutants ns source cotte mutation.
[Oîfiic+ 931 contains statistia indicating that Fortran progruns of 30 lines generate From 1000 to
3000 mutants. With the current set of operators che number of mutants generatetl is maini? iI
function of the number of states. transitions. actions. triggers. and Ievels of hierdiy. Capsules I I S P ~
iri this thesis generateti froni 100 ta over 3000 mutants. The rnost nriable Factors are the nuniber
of triggers and the nntmber of action objects. Wherens a HFSM with 10 states and 10 transitions
wiii d w a c generate 30 coverage mutants and 10 TRD mutants. the number of TTG. TCX. TAT
mutants karies with the number of action objects. triggers. and leveIs of hierarchy in the capsule.
9.19.2 CPU Time
Dependiig on the nature of the test sequences cornpihg mutants c m be the &est cornputarional
espense. LU years ago such eupense &-as prohibitive. however gii-en contempocar?; computers such as
the Pentium DI 733,CiFIz used in this thesis. 3400 mutants can be compiled in about 7 hours. ic-hich
is not prohibitive- Sharing the Ioad amongsc severai computers can further reduce the t h e . The
other computationa1 e-qense is esecutbq each mutant against the test sequences.
-ifter sending a message to the Capu'T, the test harness used in this thesis can be instmcted
to wait for a response before sending the next message. A clelay between sends is usually needeti if
the capsule has 'an interna1 timer and the test sequence requires that the timer be aiiowetl to expire.
However. if the capsde responds immediately then there is no appreciable clel- between test harness
sentls. If the CapCT does not responci then the t e s hamess waits until the tester selected time period
tins clapseti. This means that the lower bound execution tirne for a test seqiience c'an be e-xtremely
short. much less than a second for the passive capsules (capsules that do not use interna1 timers
o r iriternal everits) iised in this thesis. However some of the dyriarnic capsules uset1 in this thesis
required rriultiple waits of up to LO scconcls for a tinicr co expire. Orie of the capsules rcqiiirrd a
test setlucrice chat uscd 10, 10 second clelays for a test scqueoce ext'ctttion tinic of alniost 2 riiinutes
( tliis tvas rctliicctl).
Running huntlrctls or even a feu-- thoiisand passive mutants against a hantlful of test secluences
is feasible #ven eonternporary coniputers. In the exampie citctf above compilation of 3400 niiitants
took 7 hours. whereas running 4 test setpiences against each mutant requiretl only 34 miniitcs.
However consider a test seqiience that has a singIe timecf event of 10 seconcfs. The cime increase
woiilci be 34000 seconds or about 10 hours. E v e n time that a simiiar delay is acitled to a test
sequence 10 hours is added to the total testing tirne. Testing these types of capsules may reqiiire a
tester to motlify the capsule by reducing the time in t end . if modifications are not possible then it
rnay not be cost effective to use mutation testing for these capsules.
9.19.3 Faults
As mentionecl eulier when a fault is fountl in a capsule the capsde niiist be rep;iiretl. the rriiitiirits
regenerated. reconipileci. ancl al1 of the mutants execrited against al1 of the test sequerices iigiiitl,
For n capsule chat generates maq- mutants the repeated cost of compiling mutants can hecome
prohibitive. K i th the capsule that generated 3400 mutant each time a fault is found and repaired Ï
hours are required to recompile al1 of the mutants, This indicates that mutation testing is probably
better suited to testing capsdes that are of high qualitt. when they are @en to the tester. Capsuies
chat appear to be buggy or of Ion- quaiitj- shodd be rejected.
9.20 Preambles
Often several test secluences are required that start From a common non-initial state. Siich ueqirences
consist of two pieces. The first piece. caiied the preamble. is used to get the capsule into the common
scate. The reuiaincler of the test sequence performs the -new" testing chat is unique to each test
scquence. If preambles are short then simply repeating them in each test sequence is acceptable.
However long preambles are a problem and present the opportunity for eEciency improvements. X
possible way CO eliminate re-executing common preambles is to record the capsule's total statc once
che preamble Iias b e n execiitetl. Then each test sequerice that relies on the preiimble. cile r~cordeci
state can be restorecl ancl the test seqiience run from there.
IKth most of tlie capsules testecl in this thesis the (tinie) length of common pre-nmbles r a s noc
liirgt. t*noiigii tri n-arriint iisirig a state recortling fiinction. Ho\vcvcr thcre arc sorne c;ises where Iong
diirarion timeoiit events constitiite a large part of the riin time. Avoirling the repetition of tt~ese
timeouts coiiIcl be worthwhile. This has been left for future work.
9.21 Testing your own code
Convericiritial tvisdoni says chat clevelopers should riot Lest their own CO& (as disciissed in section
2.5). SIutatirin tcsting triay mitigate this argument. The exercise behinc1 mutation testing is to kiIl
mntants. the programmer-tcster knom that the nititant is a faiilty version of his o c n progarn (his
creacion tvhich he tvants to protecc). It behooves the tester to kilt che mutant in orrier CO stion. that
k s program (the progenitor of the mutants) is the correct versioti by finrii~ig a t e s case that kilb
ttic mutant. if he cannot kill the mutant then either the mutant is equimient (this is the weakn~ess
of the argument. the programnier-tester rnay riot be entireiy objective if objecti\-ity means that his
iwacion is faufty) or the programmer-tester must face up to the fact that his version is fadty-
9-22 Summary
A live mutant is a signpost to a trouble spot in the code. Possibly the utiIity of mutation testing is
that it points the tester to potential fauits in the code by asking hard questions of the forrn: -the
test data is t e h g me that it doesn't matter that 1 have changed this element of the program. Why
is that?" The ansver is that the mutant is equivalent. or the test data is not adequate enough to
müke the ciifference apparent. or the fault is rnaskecl by subsequent actions of the program, or ttiere
ia ;i fault in the original progam.
This chapter h'as presented the results of and rficussed the e'cperiments performeri to e d i i a t e
mutation testing of RoseRT capsules. It h s presented a prelirninary nnalysis of the miiracion
operators proposed in chapter 6. Fin& the chapter conclirded with n discussion of chc niain
observations made during the performance of the experimcnts.
Chapter 10
Conclusions
Tliis ttiesis t i u met al1 of the objectives ot~tlincti in Chapter L: hackgoiind information on software
testing. mutation testing. and finite state machine testing was presented: a set of niutntion riperators
ivas proposcd: testing toois were biiiIt: and experirnents were performed in order to e~diiare mucation
testing ciE RoseRT HFSMs.
10.1 Summary of Results
Ctiripter 9 riiscussetl the esperirrients carried out u d the obscnations made. The foIIowing section
presents a concise surnmary of the important obsemtions. Tt is impartant to note ehnt clrcst* ;inv
preliminan obsenations. and are based on a srnall number of eqwriments. These conclusions are
cticreforc noc well supported bu ernpiricai data and some of thern may in f x t be mong.
O Firiding a fault effeccively requires restarting the cesting process. This impiies that mutation
testing is not suitabte for testing capsules that are likely to have many fauits.
rn If we expect to tind o d y a few fadts then oniy a few of the hundreds or thousands of mutants
wiII actuaiiy Iead co the creation oia test sequence that reveals a FauIt. The remaining mutants
do not coutribute anl-thiog to the testing procesS. Tt is therefore usehi to kiII as rnaq of ~hese
niutrints as possible sith the ûrst test sequences. Initial test sequences based on transition
tours seem to be effecti~e at doing just th%.
Proceduce iç an important aspect in etficient mutation tesctng. Working ~ - i t h subsets of the
total mutant population rather than d g ail of the mutants every time reciiices the iiniount
of CPC time reqtiired and potentially rvasted. Further the tester is not beleaguered by the
large number of live mutants that may e-xist after the initial test scqiiences are executed. An
effective strategy appears to be LO add mutants in the following order and to kill d l of them
before adcling the next set of mutants:
- Coverage operators (CVT. CSE. CS'i)- Test sequences that kiU coverage mutants tend
to kiU many other mutants
- Action delete operators (.mS/E/X) - iiilling these operators seenis to lead to faiilt
revealing test sequences.
- TRD - llost TRD mutants are killed by the C3.T test sequences. Any live TRD mutants
indirate an issue tvith the capsule.
- Lon- volurrie operators: internal/external. choice point. state clelete. ancl trigger (TGT)
- Since the effectiveness of these operatos is ciirrently iinknonn thc only rcasnn tii use
thern at this stage is becatise they do not generate manu mitcants.
- T.\T - This operator and the ne- two produce mutants that are more than linear in the
numbcr of statps and transitions. However the TAT operator protluces fewr miitanls
t h r i eitticr TC.4 or TTG.
- TC'X - This opcrator proctttccs many mutants niosc of ntiich do not contribrite nny thin;:
to the testing process. By adding these mutants nt this point it is Iikely that many of them
ail1 be killed by the esisting test sequences. However the remainitig Live TCX trititarits
seerxi to be effective at revealing faults.
- TTG - This operator does not seem to produce semantically srnnll fatrlts. hon- ver i t
generntes rnany mutants. If it is going to be uscct at dl . it should be usecl last in the hopc
that niost of its triiitants n-il1 be MIed bu existing test sequences.
E~amining [ive mutants in order to deciare them equitalent or in order to create a nem test
sequence is the most labour intensive part of the mutation testing process. Improring the
evisting tooIs is possible and has the potentiaI to greatiy reduce the difncuity =and time reqtùred
to examine live mutants and to create nem test sequences.
The ~Lifference betn-een weak, hm. and scrong mutation does not seem to be as reletant with
RoseRT KFSNs as it is n-ïth source code mutation. The time and cost sat-ïngs realized bu
perforrning weak mutation tating on the capsules in this thesis is not significant. This m -
be due ta the frict that the mutated elements of a HFSM are qualitativeiy more significant
structures than many of the elements mutated in source code (Le. states and transitions are a
larger granularity than source code statements? they represent a higher IeveI of abstraction).
Executing a WSM mutant that results in a weakIy/Fimly obsenable difference alrnost aiways
leads to a strongly- observable dierence.
0 h1iitation cesting tloes not micigate the oracle probleni. A srnall gain c m be macle in that
ineffective test sequences (ix. ones chat do not !dl mutants) can be identifiecl arici ~liscnrcled
hcfore havirig to determine tvhecher the original capsule's behaviour is correct d i e n eseciiting
the test sequence.
0 Capsules Stace maclùnes ancl possibly KFSM's in generd do nut seem to nenerate ri?; rtiary
miitants as source codt? miitation. The lowcr nimber of rniitnnts retiiices the niairi disatl~.iint;ijic
of convenrional source code mutation testing which is the necessity of runuhg thousnticls of
miitants ngainst miiItipIe test cases.
0 The major cornpucational espense of testing passive capsules is the cime reqiiired to compile
al1 of the mutants: execittion time for the test seqiienccs is insignificant comparecl to the
compilation time.
For dynmic capsules the major computational expense m e be the r~xecircic~ri t i n i ~ of test
sequences. Internalty timed events of Iong duration (on the order of several secancls) iri;iy
niake mutation testing ECO inefficient to use. It is possibIc ro retiuce cimcoiit intenals. however
if a capsule uses niiiltiple interacting timers then coordination of the tinied events becomes a
problern.
0 Mutation testuig may aeaken the claim chat programmers shoiild noc test their own code.
S[utation testing encourages the construction of tests chat kiU mutants. It does not seem to
reIy on the abZty of the tester to assume a destructive attitude tomrd the program being
tested.
0 in some of the capsules tested in this thesis. mutation testing guided the construction of test
sequences that traverseci non-intuitive fauit revediig paths.
10.2 Future Work
Although the o b j e c t k s of this thesis were met there is much more ~ o r k that c m be done.
The main weakneçs of this thesis is the limited number of capsules that were used in experiments.
In order to strengthen the conclusions drawn frorn this thesis it is necessacy to perform mutation
testing on a much larger and more diverse population of capsules. SIodels developed For inclustrial
use by professional developers are needed. however this would require greater cooperation from
industry.
The set of operators proposed in this thesis is a subset of possible mutation operators. in
addition to a d d i g other behavioural operators it is possible to use mutation operators that mutate
the structure of one or more capsules. For evample capsule roles can have multiplicities peater
than 1. A possible mutation is to change the ma?cirnurn number of containecl capsules ttiat can
be instantiated by a containiig capsule. Also mutation of toolset elements cleaiing ivith message
prioritics ancl threads coiild be considerecl. Finally the RoseRT toolset supports inlieritance. Wlicn
a capsiile is siib-cliissetl the capsiile inherits the siiperclass's CSSI. The (levcloper is then ihle c»
atld states ancf transitions to the inherited CSSI. Future research could look at performing mutation
trsting of the entirc superclass-~ubclass CSSI or just thc siib-clnssecl CSLI.
Theoperators propased in this thcsis are di first-ciegree operators in that cach mutant is the rcsulr.
of niaking a sirigle syntactic change to the testual reprwencation of the state machine. In source
code mutation second and higher degee mutations are not used becaiisc they prorluce iiri espcitieiitiril
increase in the number of mutants generatecl %?thout an ensiiing improvement in thc fault revealing
powcr of the testing process [Offut+ 70001. This result may not lioltl in scat@ machine testing: aitii
a graph a tirst-degree mutation may be an nth degree mutation when \-ieverl clifferently. For exarriple
(leleting n state is a tirst-clegree mutation from the point of view of tieleting states. Iiowever it is
similar to an nLh degee mutation from the point of \<en- of deleting transitions [Dean 011. In uther
words when a state with nvo incoming and t a o outgoing transitions is cleletecl this is eqii ident to
deleting the four transitions. This is not exactly the same as a fourth clegree mutation since rather
than deleting four a r b i t r w transitions. the deIetecL transitions are related by their association n-ich
a common state- Another =ample is the second-degree mutation in which the target states of m o
transitions originating a t a common state are swapped. When a state has more than tn.0 outgoing
transitions this "paincise" operator could he appIied severai times. once for each unique pair of
transitions Ieaving the state.
hproving the tools used in this thesis would increase the efficïency with which mutation testing
could be performed. Some automated equivaiency checkhg could be integrated using TSL. integrat-
ing the tools such that they presented a çingie consistent user interface to the tester woiild reduce
some of the cediousness euperiencecl in using the current prototypical system. For esample the staric
charts produced bu RT-SuperDiff could be changed such that clicking on a live niutant on the chnrt
results in the presentation of al1 information concerning the mutant.
The cime required to compile each mutant from the generated code is sipiticam (10-'20 seconds).
Future work could look at applying mutation operators to the p ~ m e tree c l h g compilation rather
than CO the capsule text file.
It is possible CO automatically generate certain types of test sequences. .A TSL prognm coiilcl
be useci to trnnslate capsule state machine diagrams into a probleni description for a planner sirdi
as BlackBox [Kautz; 981[BIackboxI[FrohIich+ 001. The pl.anncr can then search for ancl prodocc
transition tours 'anci statc tours.
Finally nuitatiori of the dctail level C++ source code coiild be the subject of future research.
10.3 Contributions
This thesis ilas presentcd pre l iminq evidence indicating chat mutation testing of RoseRT FiFSlIs
is both feasible (n-ith tool support) ancl c m result in the creation of fault revealing test secluences.
The contributions of this thesis incliide the following:
i Critical anaiysis of the background literature on mutation testing,
O Proposed mutation operators for RoseRT capsules and a p re l iminq analysis of each operator.
O CFG for RoseRT capsule test files.
a Mutant creation tools based on TXL.
i Test sequence creation tool ( RT-Test CaseManager).
O Generic test harness capsule (RT-TestHamess).
a Test results analpis tools.
Development of an efficient procedure for performing mutation testing on capsules.
Analysis of the feasibility of mutation testing of RoseRT capsules.
a Identification of some areas of future work.
Ill
10.4 Concluding Rernarks
10.4.1 Design Mutation
h t a t i o n testing of source code is a verification activity: ;did we build the t h b g right" Mutation
tcsting of specifications is like ~.alidacion:"are we buiIding the right thing' However n i th CSSI
mutation testing, the design is being tested. X Rose Sfodel n-ithout detail level code is a specification.
SIutating this is similar to specification mutation. However it is not n-orth doing because al1 oritpi~ts
of the CSSI are the result of detail I e d code: there is nothing to observe. With the atltlition of
detail levcl code. mutation testing of CSMs is doser to. but is not the same as source code mutation
(the source code is not being tiirectly mucatcci). Uowcvcr. one coultl argue chat mutation testing of
CS!& is nctiially tl~sigri nuit ation.
10.4.2 Operator Rationale
The mutation litcrature often implies that miitation operators are tlesignerl to mimic small syntnctic
niistakes niade by programmers. The competent programmer hypothesis and coupling effect hy-
pothesis arc then citecl as being the theoretka1 foundation iipon which this tiecision is baseri. Thii
cloiirls the real piirpose of mutation testing rvhich is to encourage the creation of high -qirality-"tests.
Whether mutation operators are designed to mimic typical programmer errors (whaterer they are)
or not is irrelevant. h t a t i o n operators shouid be designed to mco~lrage the coristnrction of f i d t
reuealing tests (this is what is done in practice: Mothra has statement coverage operators. Proteiun
has transition coverage operators). On the other hand lion- does one design an operator that is es-
pected to-encourage the construction of a fault revealing testr This is where mimicking programmer
m o r s beconies usefiil. It is a heiiristic usecl to attack the ambiguous problem of tlesigting mutation
operators that will encourage the construccion of f a d t revealing tests.
Bibliography
[Acree+ 791 AT. Acree. T.=\. Butld. R.;\. DeSIillo. R.J. Lipton. F.G. Saynartl. Mutation AnaI- pis. Technical ReportGIT-[CS-79/08. School of Information and Cornputer Science. Gcorgia Institute of Technology. 1979.
[Acrce 801 A.T. Acree. On Mutation. PhD thesis. School of Information and Cornpucer Science. Beorgia Institute of Technology. Atlanta. GA. hugiist 1980.
[:lgrawal+ 891 H. Agawal et ai.. Design of Mutant Operators for the C Progamming Lan~qiage. SERC-TR-41-P. Software Engineering Research Centre. Pirrduc University. 1989
[.Ammarin 9-41 P.E. Ammann. S.S. Brilliant. .J.C. Knight. The EfFect of imperfecc Error Deteccion on Rcliability .Assessrnent vin Life Testing. IEEE Transactions i i t ~ Software Erigineering, Co1 '10-2. Feb 1994.
[Bernhard 941 P.J. Bernhard. X Reduced Test Suite For Protocol Conformance Testing. Comniuni- cations of the AC'II. 1994.
[BIackbosj http://cs.washington.edu/homes/kautz/blackbo+/incles.html
[Boch+ 941 G. Bochrnann. A Petrenko. Protocol Testing: Rei-iew of Methotls and Relevnnce for Software Testing, Proceedings of ISST.4 94, AC11 1994.
[Rucid SOI T.X. Butld. Mutation hnalysis OF Prograrn Test Data. PhD thesis. Yak Cnivcrsicy. Scn- Haven CT. 1980.
[Budti + $21 T..LBudd. D. Angluin. Two Sotions of Correctness and Their ReIation to T~sririg. Acta Inforniatica. LS(l). pp3145.Sov 1982.
[Butler - 931 R.W.Butler. G.B. Fineiii. The infeasibiiicy of ex~erimental quantification of Liie critical softnare reiiabiIity. IEEE Transactions on Software Engineering, Vol. 19. So. 1. 1993.
[Cai 981 Kai-fian Cait Software Defect and Operational Profile hIodehg, Klumr. Boston L99S.
[Chow 781 Tsun S. Chow. Testing SoFtware Design Modeled by Finite-State SIacitines. [EEE T r w actions on Softn-are Engineering, May 1978.
[Cordyt 001 .J.R. Cordh I.H. Carnichael. R. Haiiiday. The TSL Progamniing Lmguage \-ersion 10. http://~~-~t2d.ça/inrIe~.html
[Dean U l I Communication niith Tom Dean May 2001.
[Deharo+ 971 11. Delmaro et ai.. Integration Testing using interface LIutation. iEEE Transactions on Software Engineering, h l . 27. Yo. 3.4Iarch 2001.
[DeSIilIo+ ÏS] R. DeMillo. R. Lipton. F. Sayward, Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11. p 34-43, April 1978
[DeSIillo+ 791 R. DcMillo. R.J Lipton. F.G. Sa-n-ard. Program SIutation: a ne&- appronch to pro- gram testing. infotech State of the A r t Report. Software Testing and Edriation. Vol 2 Imqtecl Papers. Infotech Lnternational. 1979.
[DeSIilIo+ SS] R. DeMillo. D.S.Guindi. \V.;.41.;.4lcCracken. A..J.Offut. K.Y.King. An Estended Ovewiew of the SIothra Software Testing Environment. in Proceedmgs Second Workshop on Software Testing \-enfication and Analysis. IEEE July 1988.
[DeSIilIo 891 R. DeMillo. Test Xdeqiiacy and Program Miitation. ICSE S9, 1989.
[Fabbri+ 981 S.C.P.F Fabbri. S[.E.Delrnarro. J.C.~Ialdonado. P.C. SInsiero. Proteum/FS&i: A Tool to Support Finite State Machine Widation Basecl on Sfiitation Testing. Proceecling of chc 19th International Confercnce of the Chilean Compiirer Science Society. 1998.
[Fiibliri-k 9Sbl S.C.P.F Fahbri. .I.C.SIaldonado. SIutation Testing Applied to Validate Specifications Bascd on Stntecharts. Proceedings of the 10th [riternational Symposium on Software Reliabiiicy Engineering. 1995.
[Frankl+ 971 P.G Frankl. S.S. Weiss. C. Hu, .U-&es versus SIutation Testing: .ln Experinientd Coniparison of Effectiveness. .Journal of Systerns and Softwarc. Vol. 38. 1997.
[Frohlich+ 001 P. Frohlich. .J Link. .4utornatecl Test Case Generation Froni Dynnrnic JIoticIs. ECOOP ZOO0 LSCS 1SSO. pp.472491.2000.
[Garrliner 991 S. Gardiner (Ed.). Testing Snfet-Relatecl Software: A PracticaI Hancibook. Springm. London. 1999
[Gili 621 Introc11iction to the Theory of Finite State 5Iachines. .\IcC;rnw-Hill Book Compati- hic. 19G2.
[Good+ÏJI .I. Goodenough. S. Gerhart. Toward a Theory of Test Data Selection. CEEE Transactions on Software Engineering, .lune 1975.
[Gest+S'L] R. Geist. X..J.Offutt. and F. Harris. Estimation and enhancement of real-tirne softn-rire re1iabiIit.y through mutation analysis. E E E Tr.ansactiotis on Cornputers. vol. 41. pp. -550-558. 41- 1992. Speciai issue on FauIt-Tolerant Computing.
[HamIet S8i D. Harnlet Special Section Software Testing, Comrnunicatioris of the XCM. ikI 31. Xo 6. June 1988.
(Harnlet+ 901 D. Hamlet. R. Taylor. Partition Testing Does Yot inspire Confidence. KEE Trnnsac- tions on Software Engineering, 16(12): 1402-1411. Dec. 1990.
[HamIet 941 D. Hamlet. Software Quality. Software Process. and Software Testing.
[Hamiet 9431 D. Hamlet. Foundations of Software Testing: Dependability T h e o n SIGSOFT 94. ACSI. 1994.
fHarel 871 D. Harel. Statecharts: a visual formalism for complex systems. Science of Computer Programming S. pp231-274. 1987.
[Kowden 92) W.E. Howden, Weak Mutation Testing and Cornpleteness of Program Test Sets. EEE Transactions on Software Engineering. Vol. Se08. ?io.-l. .Jdy 1982.
[ET4961 Reference Slodel for OSI.
[Jalote 971 PanIsJij ..Talote. An integrated Approach to Software Engineering, Springer-\ërlag7 xew York. 1997.
[Jorg 9.51 P. Jorgençen, Software Testing: X Crdtsmans Approach. CRC Press. Boca Raton. 1993.
[I<niitz+ 981 H. Kaucz, B. Selman. BLACKBOS: A New Approach to che Application of Theorem Preting to Problem Solt-ing. Working notes of the Workshop on Planning as Combinatorid Search, held in conjunction with .OS-98. Pittsburgh, PA, 1998.
[Kini-i 991 S. Kim. J .b . Clark. .J..L IIcDerrnid. The Rigorous Generacion of .Java .CIirtation Oper- ators Using HAZOP. Technical Report. The University of kmk. 1999.
[Koo 961 1. Koo. Slutation Testing and Three Variations.
[Iinighti 851 J.C. Knight. P.E. Animann. An Experimental Evaluation of Simple SIethods For Seecling Progam Errors. Proceedings of the Eighth International Conference on S o f t ~ x r e En- gineering. pages 337-3-12. August 1985.
[Maci 911 A. SIathirr. R.DeMil10. On The Usc of Software Artifacts to Evaliiate Effectiveness of SIiitation Analusis for Detecting Errors in Production Software. Technical report. Software Engineering Research Center. Purclue University . SERC-TR-92-P. SIarch 1991.
[Liskov 011 B. Liskov with .J.Giittag. Progam Development in .rat;r. Xtlclison-Wesley. Boston. 2001.
Lyons 981 -4ndrew Lyons. CSIL for Real Time Overview.
[Mat 911 X.P. SIathur. Performance. Effectiveness. ancl Reliability issues in Softm-are Tcsting. LEE€ Proceecling of 15th Xnnriai international Cornputer Software ÿ; Application Conference. 1991.
[SIycrs 791 G. Lieuers. Thc Art of Software Testing, A \Vile-interscience Publication. 1979.
[Sior S4j L..I. SIorel. X Teon of Error-Based Testing, PhD thesis. Cniversicy of LIarylcantl. ColIege Park MD. 19%. Technical Report TR-1395.
[Sliillcrbtirg 951 M. SIiillerburg, Systematic Testirig: a Means for Valitlating Reactivt! Systonis. Eu- roST.4R 94 Special Issue of Software Testing, Verification and Reliability. \QI .?. S o 3. 199.5.
[Offuc 921 A . J . Offut. hvestigations of the Software testing coiipling effect. ACSI Trans:ictions on Softwarirc Engincering 5fcthodology. 1(1):3-15. Jaiiuary 1992.
[Offut+ 931 X.J. Offut. G. Rothermel. C. Zapf. An Lqeriniental Etduation of Seleetive SIiitation. 15th Annual Conference on Software Engineering. EEE. May 1993.
[Offutt 951 A. .J. Offutt. A Practical System for Slutation Testing: Help for the Common Progam- mer. I2eh International Conference on Testing Cornputer Software. p99-109. 199.5.
[ O f i t i 061 A.J. Offut. A. Lee, G. Rothermel, R.C'ntch, and C.Zapf. An Experirnental Detennina- tion of Sufiicient mutation operators. AC41 Transactions on Softmre Engineering Slethotlologu. :(2):99-)-tls. +ri1 1996.
[ O f i t f 96bj X.J. Ofhic. .J. Pan. K. T e w ~ T. Zhang, An Esperimental Ernluation of Data Flow ,and JCutation Testing, S o h a r e Practice and Experience. 26(2):1763-176. February- 1996.
[ORut+ 961 A.J. Offut. J . Hufffiman Hayes. A Semantic Mode1 of Progam Faults. ISSTA 96. L996.
[Offutt+00] A. .J. Ofhm. R.H- tintch. Mutation 2000: Gniting the Orthogonal. Mutation 2000: 1Iutation Testing in the Twentieth and Twenty First Centuries. p-15-55. 2000.
[Petredmt 961 A. Petrenko. G. Bochmann. On Fault Coverage of Tests for Finite Stace SpecSca- tions. h t t p : / / ~ ~ 7 ~ . i r o . u m o n t r e a I . c a / p u b / t e l e ~ 1996.
[Rapps+ 351 S. Rapps. E.J. Weyuker, Selectïng Software Test Data Using Data Flow information. iEEE Transactions on Software Engineering, SE-11. April 1935. 367-375.
[RoseRT 991 Rational Rose ReaiT'ie 6.1 Oniine HeIp: Jlodelùig Language Guide. 1999.
[RTCh 921 RTC.4. hc . SoEtmre Considerations in Airborne Systems and Equipment Certification. Document So. RTC.4/DO-liSB. Dec. 1992.
[Shepard+ 011 T. Shepard. SLLamb. D-Kelly, More Testing Should Be Taught. Cornrniinications of the ACM. (Vol. 44. ?JO. 6), dune 2001.
[Sheparrl 011 Communication with Te- Shepard Aug. 2001.
b?iet 931 .J-C. van Lliet. Software Engineering: Principies and Practice. Wiley. Great Bricain. 1993.
[\:os + 981 J.11. Voi~as. G. SIcCraw. Software Fault Injection: Inoculating Progrmis Againsc Errors. \Viley. Xew kbrk. 1997.
PVeyrktrr YS1 E..J. Kcyuker. .-ln Empiricai Stuciy of the Complexity of Daca Flow Testing, Proceed- ings of the Second Workshop on Software Testing, Verification. ancl Analysis, Banff. Canacla. .Jiily 1988.
[Wang+ 971 W. Eric Wong, J-C. hidonado. X E . Delrnaro. S.R.S. Souza. Cse of Proteurn to Accelerate Mutacion testing in C Programs. Proccedings 3rd ISST.4 International Conference on Reliability and Qudity in Design. h r c h 1997.
[Wootiwartl- 881 S[.R.Woodtvard. K. Hdemood. From tT;Cak to Strong, Dead or Alive:' An Anal- p i s of Some hlutation Testing Issues.Proceedings of the 2nd Korkhop on Software Testing, Qrificacion. and Analysis. pp. 152- 138, Jiily 1988.
[Zhu 371 Fi. Zhu. Softn-are Cnit Test Coverage and Adeqiiacy. .\Ch[ Computing Siineys. Der LBDÏ.
Appendix A
Test ing Glossary
error - An error is clefined as the human activity that results in software containing a fault. Errors
;irr iisiidIy niistakes niade by the programmer. Honcver not riIl m o r s leati t o failrircs.
failure - A hilure occiirs nhen a hiilt eseciites. X failiire is an es t~rnal . inr'rirr~c't 11oti;~vioiir of ;i
program (an incorrect output or run time failure)
fault - -4 fault is the result (manifestation) of an error. Also known <as clefects or brigs. The terni
faiilt can d s o refer to missing behavioitr. i.e. functionality that nns not irnplementeri (dso
called n fauh of omission).
test - The act of esercising software with test cases. A test can fincl a failure or clemonstrate
cornpliance.
test case - A single input vector to a program.
test adequacy criterion - Some measure or mle useci to establish a firm requirement for hoa
much testïng is necesan;.
test sequence - X t h e series of vectors input to a program.
Appendix B
Example Capsule Text File
This appenrlix shows a RoseRT capsule file. This capsule is From the criiise control esperinient. It Las
a structure. classattribt~tes. and statemachine section. Only the statemachine section nns stlk>jp(T
to mutation. The seccions describing the graphiral placemctit of the eleirients arc rioc iriclircled iri
this description.
B.1 Handling C++ Within Capsule Files
The [action-objectl non-terminal contains C f + code. Therefore capside files ciln be parsect by risirig
two separate grammars. one for the CSM schematic lever. and one For the C++ rletail level. How~t-er
each Lie of C+-. s h e n encoded nithin the capsule file. is prefked s i t h a vertical bar symbol. Thc
Folloning capsrile file Frahment shows an instance of an [action-object] non-terminal. Yotice the tn-O
Lines O € Ctt rode.
entryAction (object Action quid "385F19800276" body
I ControlCamms . l ightProblem0 . sendo ; I EngineComms . disengage0 .sendO ;
1
Because of the Ieatüng vertical bar SI-mbol it is oot possible to parse the C++ se,gnents wÏth a
standarci C++ garnmar. The presence of the veaicd bar symbol requires that the Cf+ gammar
be changed to ailox- a vertical bar s p b o l to be option&- present anpühere in a Ctt program
chat a tine break is le&. Rather than changing the C f + grammar it waç decidecl to eliminate
the C++ grammar and pre-process the original capsule file before parsing it nith T U . The pre-
processor encapsulated each C++ l i e nithin double quotes. Double quotes chat were present in the
original C i + line were escaped (prekred n-ith the backslash s-bol). The capsule grarnmar r a s
then changecl so chat action objects consist of [stringIitI non-terminais rather than C++ fragments.
After mutants are generated the doubIe quotes are rernowd bu a post-processor progam. These
pre-process/post-process steps are not discussed in Chapter T as they were not consiclerccl relevant
to the discussion OF the core mutation system.
The reniaiticlcr of this appendix shows the capsirle file for one of the capsdes testetl in this chrsis.
clic criiise cotitrol capsille.
B.2 Capsule File for Cruise Control
(object Capsule "CruiseControl" quid "3A6400CFOOD6" structure (ob j ect CapsuleCollaboration "Structure"
quid "3A6400CFOOE4" ports (list portsList
(object Port "ControlComms" classifier "Logical Viev::ControlU quidu '13A6411800149" isRelay FALSE)
(object Port "EngineCommsN classifier "Logical Viev::engiueM quidu "3A6412510149" isRelay FALSE)
(object Port "timer" classifier "Logical Vieu::RTClasses::Timing" quidu "37E9292COl8C" isRelay FiILSE isWired FALSE)
(ob ject Port "oatchDogTimer" classifier "Logicaï Vieu::RTClasses::Timing" quidu "3ïE9292CO 18C" isRelay FALSE iswired FALSE)
(ob ject Port "periodicTimer" classifier "Logical Vieu::RTClasses::Timing" quidu "37E9292COl8C" isRelay FALSE isWired FALSE)
(object Port "log" classifier "Logical Vieu::RTClasses::Log"
quidu 1137E92926019B11 isRelay FALSE isWired FALSE)
(ob ject Port "TestComms" c l a s s i f i e r "Logicd Vieu::TestProt" quidu "3A549694028D" isconjugated TRUE isRelay FALSE) ) )
class-at t r ibutes ( l i s t c lass -a t t r ibu te - l i s t (object ClassAttribute "setSpeedn quid "3A64124301F3"
type " int") (object ClassAttribute "DELTA" quid "3A6412AF036B"
i n i t v "5" type " int" changeable "Frozen")
(object ClassAttribute "actualSpeedtt quid "3A64136002BBB1 type " int")
(ob j e c t ClassAttribute " t imer ID" quid "3A64151B013Cn type "RTTimerId")
(object ClassAttribute " th ro t t l e" quid "3A641A5E0360m type " in t " )
(object ClassAttribute "~atchDogTimerID~~ quid "3A641BFDOlA4" type "RTTimerId" 1
(object ClassAttribute "periodicTimerIDM quid "3A70910100A2" type "RTTimerId" 11
sçatemachine (object State-Machine quid "3A6400CFOOEOv S t a t e (object S t a t e "TOP"
s t a t e s ( l i s t S t a t e s (object I n i t i a i P t " In i t i a i1T) (object S t a t e I1error"
entryAction (object Action quid "3B5F19800276" body
I ControlComms . l ightProblem0 .sendo ; 1 EngineComms .disengage (1 .sead() ;
1 s t a t e s ( l i s t S ta tes
(object f u i t i d P t " I n f t i a l * ) (object JuncPt "Junctionl"
isEntry TRüE emernal lyVisible TFLUE)
(ob j e c t JuncPt " Junctionl" external lgVisible TRUE) ) )
(ob j e c t JuncPt " Junction4" isEntry T U E )
(object S t a t e "on" entryAction (ob j e c t Action
quid "3AT096E103BC" body
I timerID = tiner. informEvery(RTTimespec(2~0, O) ) ; //ask f o r speed every 2 seconds
e x i t Action (object Action
quid "3A70970902SB" body
I i f (timerTD.isValid()) timer.caacelTimer(timerID); I i f (watchDogTimerID.isValid0) watchDogTimer . cancelTimer (vatchDogTimerID1;
1 s t a t e s ( l i s t S t a t e s
(object I n i t i a l P t " I n i t i a l " ) (object S t a t e "active"
entryAction (object Action quid "3A70911EOSOC" body
IControlComms .lightOnO + send0 ; I periodicTimerfD= periodicTimer- informEvery(RTTimespec(1 .O ,O) ) ; I
1 exitAction (object Action
quid "3A7090FDOOED" body
I ControlComms . l ight0f f (1 . send0 ; I EngineComms . disengage0 . sendo ; I 1 periodicTimer . cancelTimer(periodicTimer1D) ;
1 s t a t e s ( l i s t S t a t e s
(object I n i t i a i P t " I n i t i a l " ) (object JuncPt " Junctionl"
ex te rna l lyv i s ib le TRUE) ( o b j e a S t a t e " justRighttl
s t a t e s (list S t a t e s (object I n i t i a l P t " I n i t i a l t t ) (ob j e c t JuncPt " Jiuictionl"
isEntry TRüE ex te rna l lyv i s ib le TRUEI 1)
(object S t a t e "tooSlou" s t a t e s ( E s t S ta tes
( o b j e a Tni t i a iP t " I n i t i a l " ) (object JuncPt "Junctionl"
isEntry TRüE ex te rna l lyv i s ib le TFtüE) ) 1
(object S t a t e "tooFastn s t a t e s ( E s t S t a t e s
(object I n i t i a l P t " i n i t i a l " ] (object JuncPt "Junctionl"
isEntry TFlüE e a e r n a l l y V i s i b l e TRITE) 1)
(ob ject CholcePt "CP2" guard (object Guard
expression "return setSpeed>actualSpeed; "
1) (object ChoicePt trCP1"
guard (object Guard expression
[ i n t deltaSpeed = çetspeed-actualspeed;
(object JuncPt "Junction4" ishtry TRüE)
lob ject JuncPt "JunctionSV isEntry TRUE externallyvisible TRUE)
(object JuacPt "Junction6" isEntry TRUE externallyYisible TRUE)
(abject JuncPt "Junction3" isbtry TRüE externailyVisible ïFLiJE)
(ob ject JuncPt "Junct ion2'' i sEntry TRITE1
lob ject JuncPt " Junction7"l (object JuncPt "Juaaion8"
isEntry TI1iE) (object JuncPt "Junction9"))
transitioas (List transition,list (abject Tram "True"
sourceRegion 1 source " :TOP:on:active:CPl" target
" :TOP:on:active: justRigbt: Junctioal") (object Trans "False"
sourceRegion O source ":TOP:on:active:CPl" target " :TOP :on:active:CP2")
(object Trans "True" sourceRegion 1 source " :TOP :on:active:CP2" targe t
" :TOP:on:active: tooSlow: Junccionl" action (object Action
quid "3A708FT60004" body
"EngineComms .setThrottle(throttle+5) .sendo ;" 1)
(object Trans "False" sourceRegion O source " :TOP : on:active : CP2" target
" :TOP :on:active:tooFast : Junctionl" action [ob ject Action
quid "3A708Fïi3015F" body
"EngineComms -setThrottle(throttle-5) .send() ;" I I
( o b j e a Trans "timeout" source " :TOP : on:aaive : Junction4" target ":TOP :on:active:CPl" eventGuards (List event Guardslist
(ob ject EventCuard Event (ab j ect PortEvent
p o r t s ( l i s t p a n s L i s t ports
signals (l ist
(object Trans "accel" is In terna1 TRUE source ":TOP:on:active:Junction2" t a r g e t ":TOP:on:active: Junctioa7" a c t i o n [object Action
quid "3AfOA4090093" body "setSpeed+=DELTA; " 1
eventGuards ( l i s t eventGuardsList (obj e c t EventGuard
Event (ob jec t PortEvent p o r t s (list p o r t s L i s t
po r t s
s i g n a l s (list
s ignals
(ob jec t Trans "decel" is1nterna.i TRUE source " :TOP:on:active:function8" t a r g e t ":TOP:on:active:function9" ac t ion (ob j e c t Action
quid "3A70A40E032FM
eveatGuards ( l i s t eventGuardsList (obj e c t EventGuard
Event (object PortEvent ports ( L i s t portsList
ports " :Structure :Contro1CommsV)
signals (list signaLsList
s ignals " :base:deccel l ' ) ) ) ) ) ) )
( s b j e c t S t a t e "paused" s t a t e s (list S t a t e s
(objem I n i t i a l P t " I n i t i a l t 1 ) (ob ject JnncPt "Junctionl"
isEntry 'CRUE excernàl lyVis ib le TRüE)
lob j e c t JuacPt " Junction2" e x t e r n a l l y v i s i b l e TRüE)
l o b j e n JuncPt "Junction3" e r t e rna ï lyVis ib le TRUE)))
(objec t JrrncPt " J m c t i o n l " isEntry TRUE
externallyvisible TRUEl (abject JuncPt " Junction2"
extedlyvisible TRITE) (object JuncPt "Junction3"
isEntry TRUE) (obj ect JuncPt "Junction4") (ob j ect JuncPt "JurictionS"
isEatry TRIE) (ob j ect JuncPt " Junction6") (object JuncPt "Junction8"
isEntry TRITE) 1 transitions (list transition-list
(object Trans "disengage" source " :TOP:on:active: Juactionl" target " :TOP :on:paused: Junctionl" eventGuards (list aventGuardsList
(ob j ect EventCuard Event (ob ject PortEvent
ports (lis portsList ports
" :Stnicture:ControlCommsfl) signals (list signalstist
signals ":base:coastM signals ":base:brake"I3)1)
(ob ject Tram "resume" source " :TOP :on:paused: J ~ n a i o n 2 ~ ~ target 'q:TOP:on:active: Junction3" eventGuards (list eventCuardsList
(ob ject EventGuard Event (object PortEvent
ports (list portsList
ports '' :Structure : ControlCommsl')
signals (list signalsList signals ":base:resume")) 1 ) )
(ob j e n Trans "getSpeedm islnternal TRüE source " :TOP : on:Junctioa3" target ":TOP:on:Junction4" action (object Action
quid "3A70973900DD" body
IactuaiSpeed=(intl~rtdata; I I//reset the watchdog timer tif (watchDogTimerID.isVdid0) vatchDogTfmer . cancelTimer(vatchDogTimerID) ;
1 uatchDogTimerID = uat chilogTimer. inf ormIn(RTTimespec(5 - O, O) ) ; 1
eventGuards (list eventGuardsList (ob j ect EventGuard
Event (ob ject PortEvent ports (list portsList
ports
" :Structure : EagineComms "1 s igaa l s (list s igaa l sLis t
s ignals " :base: speed") ) ) 1 ) (object Trans "requestSpeed"
i s h t e r n a l TRUE source " :TOP : on: Junct ion5" t a r g e t ":TOP:on:Junction6" ac t ion (object Act ion
quid "3A70904700AE" body "EngineComs .getSpeedO .sendo ; " 1
eventGuaràs ( l i s t eventGuardsList (object EventGuard
Event lcsbject PartEveat ports ( l i s t por tsLis t
por ts " :Structure : timer") s igna l s ( l i s t s igna l sLis t
s igna l s " :base:tirneout"))))l
(ob ject Tram "setPushed" source ":TOP:on:JunctionS" ta rge t ":TOP:on:active:JunctionS" ac t ion (object Action
quid "3A7094CFO229" body "setspeed = actualspeed;
eventGuards ( l i s t eventCuardsList (ob j e c t EventGuard
Event (ob j e c t PortEvent por ts ( l i s t por t sLis t
por ts " :Stnctue:ControLComs")
s igna l s ( E s t signaZsList s igna l s " :base:set t i ) ) ) ) )
(object Trans "coastReleasedtt source " :TOP :on: paused: Junaion3" t a r g e t " :TOP :on: active: Junction6" ac t ion (object Action
quid "3A7094CB002Em body "setspeed = actudSpeed;
(ob j ec t EventGuard Event (object PortEvent
ports ( l i s t p o r t s l i s t por ts
s igna l s (list s igna l sLis t s igna l s
tt:base:coastReLeasedit) 1)) 1 )) Cobject S ta te
S t a t e s (list S t a t e s (ob j e c t InitialPt " I n i t i a i " ) (object JrrncPt tlJunctionl"
isEntry 'RüE
external lyl l is ible TRUE) (ob j e c t JuncPt " Junction2"
ex te rna l lyv i s ib le TRUE) (object JuncPt "Junction3"
isEntry TRUE ex te rna l lyv i s ib le TFlUE)
(ob j e c t JuncPt "Junction4" isEntry TRE exte rna l lyv i s ib le TRUE) ) )
(ob j e c t JuncPt " Junctionl" isEntry TRUE)
(object JuncPt "Junction2") (object JuncPt "Junction3"
isEntry TRUE) (object JuncPt " Juaction5") )
t r a n s i t i o n s ( l i s t t r a n s i t i o n - l i s t (object Trans "watchDog"
source ":TOP:Junction4" t a r g e t :TOP : e r r o r : Junct ionl" eventCuards ( l i s t eveatGuardsList
(ob j ec t EventGuard Event (ob j ect PortEvent
ports ( l i s t p o n s L i s t por ts :Structure : uatchDogTimer")
s igna l s ( l i s t s ignalsLis t s igna l s ":base:timeout")))))
(ob ject Trans " I n i t i a l " source ":TOP:1nitialw t a r g e t " :TOP :off : Junctionl")
(object Tram "turnOnM source " :TOP :off : Junction2" t a r g e t ":TOP:on:Juxctionl" eventGuards ( l i s t eventGuardsList
(ob j e c t EventCuard Event (object PortEvent
por ts ( l i s t por tsLis t pofts 'l :St~cture:ControlComms~)
s igna l s ( l i s t signalslist s igna l s " : base:onU) ) ) ) )
(oh j e c t Trans "turn(lff If source " :TOP :on : Junctian2" t a r g e t " :TOP :off : Junction3" eventGuards ( l i s t eventGuardsList
(ob ject EventGuard Event (ob ject PortEvent
por ts ( l i s t por tsLis t por ts ":Stnicture:ControlComms")
s igna l s [List s i g n a i s t i s t s igna l s ":base :offw) ) ) ))
(object Trans "tl" source " :TOP : error : Jrrnction2" t a r g e t " :TOP : of f : Junction4" eventGuards (Lis t eventGuardsList
(ob ject EventGuud Event Iobj e n PoaEven t
ports (list porcsList por ts " :St~cture:ControlComms")
signals (Iist signalsList signals ":base:offU))))))
Appendix C
TXL Grammar Files Used For
Parsing RoseRT Capsule Files
This appcnclk shows the T U gamrnar files usecl in the MTS-RT system. The cornplete grnriirriiir
is rtistribiitetl ncross chree files.
C. 1 Grammar file: MarkUpCapsule.grm
%File: MarkUpCapsule.gm %B.M. Graham, Queen's University, 15 May 2001,
include "CapGrammarVieu-p" include " protocol .grm8'
keys 'State 'abject 'states 'statemachine 'State-Machine 'saurce 'target 'Trans 'transitions 'action 'Action 'body 'eventCuards seventCuardsList 'Event 'PortEvent 'Capsule 'PetalRT 'language rcondition 'entryAction 'exitAction 'documentation 'ChoicePt 'sourceRegion
end keys
end compounds
def i n e program [repeat petal-section]
end def i n e
def ine left-mark l < Cid1
end def i n e
def ine right-mark ' >
end def i n e
def ine petal -sect ion [opt petal-header] I [capsule-ob j e c t l I lprotocol,obj ec t l
end def ine
def ine capsule-object [NL] ' ( 'object 'Capsule [ s t r i n g l i t ] [IN]
[NLl [unique-id1 [NL] Crepeat section]
'1 CEXI end def i n e
def i n e s e c t i o n [statemachine] I [ s tmct t l re] I [language-specif i e r ] i [stmcture-diagram] I Cstate,diagraml I [c lass-at t r ibutes] I [col laborat ions- l is t ] I ~ p r o t o c o l ~ r o l e s l I [class,operations] I [uses-relationship-list] I [documentation-blockl I [associat ion- l is t ]
end def i n e
def ine assoc ia t ion- l i s t [NL] 'assocsOwned '( ' list 'assocsOunedList[IN]
[repeat association-ob ject] ' 1 CEXI
end def i n e
def ine association-object CNLI l C'object 'Assoc btringlitl [LUI
CNLI [unique-id1 [NL] [ends-List]
' > CEXI end def ine
def ine uses-relationship-List [NL] 'used-nodes ' ( ' l i s t 7uses-re la t ionship- l is t [IN]
[repeat re la t ionships] ' ) EEXI
end def i n e
def ine re la t ionsh ips CNLI ' ( 'ob j e c t 'Uses-Relationship CINl
Crepeat option1 ' 1 CEXI
end def i n e
dei ine p o r t s - l i s t [NL] ' por t s ' ( ' list *por t sL is t [IN]
Crepeat port-ob ject] 1 CEXI end def ine
define port-object [ml ( lobject 'Por t [ s t r i n g l i t ] [NL] [IN]
[id] [ s t r i n g l i t ] CNL] runique- id1 [repeat option]
' 1 [EX] end def iae
\ textbf{to here u i t h tab)
def ine statemachine INLl 'statemachine ' ( ' ob j e c t ' State-Machine CNL1 CINI
[unique-id] 'S ta te [repeat pseudo-state-obj ec t ] [NL)
% there is an lg one but repeat helps vri t i r ig n i l e s ' 1 CExl
end def ine
de£ i n e state-ob j e c t [Ni.] ' ( ' ab jec t 'S ta te [ s t r i n g l i t ] [IN]
[opt documentat ion-block] [repeat action-ab jea;] [opt s t a t e s - l i s t ] Copt t r a n s - l i s t l
' 1 CEXI end d e i ine
def ine documentation-block CNLI do cumentat ion Cs t r ing l i t l
end def i n e
def ine s t a t e s - l i s t ENLl s t a t e s ' ( ' l i s t 'S ta tes CINI
Crepeat pseudo-state-ob j ec t l ' 1 CExl
end def ine
define object Cpseudo-state-ob ject]
end def ine
define pseudo-state-object [old-pseudo-state-abject] I (Nt] [left-mark] [old-pseudo-state-ob ject] [right-mark]
end define
define old-pseudo-state-object Estate-ob jectl I ~ j u c t i o n ~ o b j e c t l I Cinit id -po in t -abject 1 I [choice-point-ob j ec t l
end def ine
def ine choice-point-object [KL] ( 'abject 'ChoicePt [ s t r i ng l i t ] [IN]
Copt guardl '1 Cwrl
end define
def ine guard [NL] 'guard ' ( lob j ec t 'Guard (IN] CNLl 'expression Cexprsssion-body] CNL] '1 CEXI
end def ine
define expression-body [s t r ingl i t ]
end def ine
def ine junction-ob ject CNL'I ' ( 'object ' JuacPt [stringLit] [IN]
[repeat option] ' 1 CEXI
end def ine
def ine initiai-point-ob jec t [Nt] ' ( 'abject ' In i t i a lP t Cstringlitl [IN] ' 1 CEXI
end define
define t rans- l i s t [NL] ' t ransi t ions ' ( 'list ' t rans i t ion- l i s t [IN]
[repeat trans-ob j ect] '1 cm
end def ine
def ine trans-ob ject [old-trans-ob jectl I Cm1 Clef t - m a r k 3 [o:ald-trans-ob ject] Cright-mark]
end define
define old-trans-abject [NLI ' ( 'object 'Trans Cs t r ing l i t l [ I N ]
[opt source-region] [opt predicate] CNLI 'source Cs t r ing l i t l CNLl ' t a r g e t Cs t r ing l i t l [opt action-ob ject] [opt event-guards-listl
' 1 CEXI end def ine
def ine source-region [NL] ' sourceRegion [number]
end def ine
def ine event-guards-list [NL] ' eventGuards ' ( ' l ist ' eventGuardsList [IN]
Crepeat event-guard-ob j ec t l '1 CEXI
end def ine
def ine event-guard-abject [NL] ' ( ' object ' EventGuard (1 NI
[opt code-condition] [repeat event]
'1 CEXI end def ine
def ine code-condition [ml ' condition Cs t r ing l i t l
end def ine
def i n e event [NL] 'Event ' (job j e c t 'PortEvent [IN]
[repeat ports-and-s ignêls l ' 1 CEXI
end def ine
def ine ports,and,signals Cportsl I Csignalsl
end def ine
define por ts [ml ' por t s ' ( 'list ' por t sLis t CINI
[ml Crepeat single-port1 ' 1 CEXl
end def ine
de£ ine single-port [NL] 'por ts [ s t r i n g l i t ]
end def i n e
define c lass -a t t r ibu tes CNLI ' c iass -a t t r ibu tes ' ( ' l i s t ' c l ass -a t t r ibu te - l i s t CINI
[repeat class-attribute-object J ' 1 CEXI
end def ine
def ine class-attribute-object [NL] ( ' ob j e c t 'ClassAttribute [ s t r i n g l i t ] [IN]
Copt unique-id1 Copt documentation1 Crepeat opt ion1
' 1 CEXI end def ine
def ine class-operations [NL] 'operations ' ( ' l ist ' Operations [ IN]
Crepeat operation-ob ject] '1 CEXI
end def ine
def ine operation-object [NLI ' ( 'object 'Operation [ s t r i n g l i t ] [ IN]
Copt unique-id1 [code-section] [repeat operation-option]
' 1 [EX] end def ine
def ine operation-option [NLl ' v i r t u a l Cbooleanl 1 [NL] ' r e s u l t Cstr ingl i t l 1 CNLI ' v i s i b i l i t y Cs t r ing l i t l I [NL] [parameters-list]
end def ine
define parameters-list [ml 'parameters ' ( 'list 'Parameters [IN]
Crepeat parameter] ' 1 CEXI
end def i n e
def ine parameter [NLl ' ( lob j e c t ' Parameter Cs t r ing l i t l [IN]
[ml ' type Cstr ingl i t l ' 1 CEXI
end def ine
def ine documentation [NLl documentation [ s t r ing l i t ]
end def ine
def ine signals [NI.] ' s igna l s '( 'list 'sigaïsList [IN]
Crepeat signal] ' 1 CEXI
end def ine
def ine s i g n a i [NL] 'sipals [ s t r i n g l i t ]
end def i n e
def ine action-type ' ac t ion I ' entryAct ion I 'exitAct ion
end def i n e
def ine action-ob jec t [NL] [action-type] ' ( 'ob j ec t ' Action [IN]
CNW [unique,idl Copt code-section]
INLI ' 1 CEXI end def i n e
define code-section [ N t ] ' body [ s t r i n g l i t l I CNLl 'body [repeat barJinel
end def ine
def i n e bar-line [NL] [TAB-11 ' I [ s t r i n g l i t ]
end def i n e
def ine doc-bits [ s t r i n g l i t l l Cid1 I J:
1 ' . end def i n e
C.2 Grammar file CapGrammarView.grm
%File: CapCrammarView.grm %B .M. Graham, queenJ s University, April 2001.
def ine petal-header ' ( l o b j e a 'PetaïRT [NLI CINI ' version [number] CNL] ' -u r i t t en Estr ingl i t ] [NI.] 'charSet [number] ' 1 CEXl
end def i n e
def ine language-specifier [NL] 'language [ s t r i n g l i t ]
end def ine
def i n e structure-diagram [NL] ' structureDiagram ' C J ob j e c t Cid1 [ s t r i n g l i t ] [ IN]
[NL] [unique-id] [i tems,list l [opt f i l t e r - f l ags l
'1 CEXI end def i n e
def ine f i l t e r - f l a g s [NL] 'f i l t e r F l a g s [number]
end def ine
def ine i t ems- l i s t ' items ' ( ' l ist Cidl CNLI CINI
[repeat view-ob j e c t l '1 CEXI
end def i n e
def ine view,object [opt id] ' ( 'object Cid1 [opt s t r i n g l i t ] [opt s t r i n g l i t l [unique-id] [NL] [IN]
[repeat viewable-objeml [Nt] 1 [mi CEXI
end def i n e
def ine viewable-object [position] 1 [option] 1 [unique-id1 1 [bound-icon-list] 1 [label-ob jec t l 1 [state-diagram] 1 [vieu-ob j e c t l 1 Cview-tgt-tagl 1 [ver t i ces - l i s t l 1 [id] [coordinatesl
end def ine
def ine v e r t i c e s - l i s t ' ve r t i ces ' ( ' l ist [id] [IN]
[repeat coordinates] ' 1 CEXI
end def ine
def ine view-tgt-tag [NLl Cid1 B Cnumberl
end def ine
def ine label-object
[repeat vieuable-ob j ect] [NL] ' 1 Ex1
end def ine
define bouud-icon-list ' bouadIcons ( ' Es t [id] [NL] [IN]
[repeat viev-ob ject] ' 1 CExl
end def ine
def ine s tructure ENLl ' structure ' ( ' ob jert * CapsuleCollaboration
[ s t r i ng l i t l CNL] [IN] [unique-id1 [opt classifier,roles] [opt ports-List] %mutation change [opt connectors-list]
' 1 CExl end def ine
define connectors-list [NLl ' connectors ' ( ' l is t ' c o ~ e c t o r s L i s t [IN]
[repeat connectorj ' 1 CEXl
end def ine
def ine connector [NL] ' ( lob ject ' Connecter [ s t r ingl i t ] [IN]
[fi] ' connectorCardinality ' constVaZue [str ingl i t ] [ends-list]
' 1 CEXI end def ine
def ine ends-l is t [ml ' ends ' ( ' L i s t 'endsList [ IN]
Cm1 l ends Cst rmgl i t l ENLI ' ends [s t r ingl i t ]
' 1 [EX] end def ine
define classif ier-roles [NL] c lassif ierRoles ( 'List 'classif ierRolesList [IN]
Crepeat capsale,rolel '1 CEXI
end define
define capsule-role [NL] * ( 'ob j ec t ' CapsuleRole [s t r ingl i t ] [ IN]
[NL] ' c l a s s i f i e r [ s t r ingl i t ] CNLl [unique-id1 Crepeat option]
' 1 CExl end def ine
define option Cpredicatel I [labelled-thing] I [set t ingl I [a t t r ibute-set] I [value]
end def ine
def i n e value [NL] 'value ' ( [ s t r i n g l i t ] [numberl ' )
end def ine
def ine a t t r ibu te - se t [NL] ' a t t r i b u t e s ' ( ' l is t ' Attribute-Set [IN]
[repeat a t t r i b u t e ] 1 CEXI
end def ine
def ine a t t r i b u t e [NL] ' ( 'abject ' Attr ibute [ I N ]
[repeat option] ' 1 CEXI
end def ine
def ine labelled-thing [NL] [id] [ s t r i n g l i t ] l [NL] [id] [number] ! [HL] Cid] [id] [ s t r i n g l i t ]
end def ine
def ine s e t t i n g [NLI Cnumberl
end def ine
def ine unique-id 'quid Cs t r ing l i t l I 'quidu [ s t r i n g l i t ] I ' O [number]
end def i n e
def ine coordinates ' ( [number] , [number] )
end def i n e
def i n e pointer-to-id Cidl [unique-id]
end def uie
def ine option-setting Cidl [nnmber]
end define
def ine language-id ' language Cs t r ing l i t l
end def ine
def i n e boolean 'TRUE I IFALSE
end def ine
def i n e posi t ion
POS ( [number] ' , [numberl ' ) end d e i ine
def ine predicate [NL] [id] [boolean]
end d e i ine
def ine named-string [NL] [id] [ s t r i n g l i t ]
end def ine
def ine state-diagram [NL] lstatediagram ' ( 'object 'State-Diagram [ s t r i n g l i t ] [IN]
Diagram" [unique-id1 [i tems-listl [opt f i l t e r - f lagsl
'1 CEXI end de£ ine
C.3 Grarnmar file: protocol.grm
%File: p r o t o c o 1 . p % B.M. Graham, Queen's University, 15 May 2001,
de f ine protocol-abject [ N t ] ' ( 'object ' Protocol [ s t r i n g l i t ] CINI
CNLl [unique-id] [Nt] [repeat section]
' 1 CEXI end def ine
def ine col laborat ions- l is t [NLllcollaborations ' ( ' l i s t lcol laborat ionsLis t
[IN] ( ob j e c t 'Collaboration [ s t r i n g l i t ] [IN] [tinique-id] '1 ' 1 [EX] CEXI
end def ine
def ine protocol-roles [NL] 'protocolRoles I ( 'list JprotocolRolesList
[Nt] [IN] Crepeat protocol-role] '1 CEXI
end def ine
de£ ine protocol-role [NL] ' ( 'object 'ProtocolRole Cs t r ing l i t l
[Nt] CINI Crepeat prot-role-section] ' 1 CEXI
end def i n e
def ine prot-role-section [in-signalsl 1 [out-signalsl
end def i n e
def i n e in-signais CNLl ' i n S i p a l s ' ( ' list ' inSignalsList [ IN]
Crepeat protocol-signal] CNLI ' 1 CEXI
end def ine
def ine out-signals [NL] 'outsignals ' ( list loutSignaisList [IN]
Crepeat protocol,signal] [HL1 ' 1 CEXI
end def ine
def ine protocol,signal [NL] ' ( 'object ' Signai [ s t r i n g l i t ] [IN]
Copt data-type1 CNLI ' 1 [EX3
end def i n e
def ine data-type 'dataType Cstr ingl i t l
end def i n e
Appendix D
RoseRT Capsule Statemachine
Behaviour
This appenciiu contains dctailed <lescriptions of capsule staternachine behaviour as implementeci by thc RoseRT toolset. It is extrncced and adapted from [RoseRT 991.
D.l Top state
Every capsule state machine has a unique cap Ievcl State ttiat is the root of the stnte machine hierarchy. The top state cannot have exit and entry actions. or outer self- transitions.
D.2 Actions
Actions are the things the behavior does ahen a transition is taken. They represent esecutabIe atomic computations that are witten as statements in a det'aii-Ievel prograrnming Iangriage and incorporated into a state machine. They are simiIar to a procedure cd1 in a procedural language- Actions are atomic. in the sense that the- cannot be intempted bu the a m d of a higher priority event. An action therefore runs to cornpietion.
Actions may be composed of nny number of operation caiis. creation or ~Iestruction of otber objects. or the sending of messages to other objects- An action may be attachecl to the foiiowing parts of a state machine:
1. A transition (including a transition to an initiai state)
2. A state. as an eut- action
3. 1 state. as an exit action
D .3 Transitions
,k transition is a relationship betaeen two states: a source state and a clestination state. It specifies thnt wtien an object in the source state receives a specified event ancl certain conditions are met. the behavior will move from the source state CO the destination state.
D.3.1 Kinds of transitions
O Yormal transitions -4 transitions that originates anci terminates on different states.
O Three kincls of self-transitions chat are characterizcd by originating ancl terminating on the same state. having no continuing segments. and not encling on a continuing jiinction point.
1. inner self transitions Where the exit and entry code is not esecutetl for the state on which it originates ancl terminates.
2. Inner interna1 self transitions \Thece the transition esecutes withouc exiting or rcentcring the state in which it is defined. And in addition the exit <and entry actions of iitl states which where esited aricl re-entered are not esecuted. These kincls of transitions are similar to having global operations defincd on a state machine: shen t;iken do iiot chiinge the Stace of the system.
3. Extemal self transitions Where the esit and entry code is esecutctl for the statc on wtiich it origmates and tcnninatcs.
D.3.2 Initial point and initial transition
An initial point is a specid point which e~plicitly shows the beginning of the state triachitie. YCN connect the initial point to a start state. Where the start state ~ i11 be the tirst active state in che nhjects state machine. The transition from the initial point to the start state. initial transition. is
rlL 1 StiLtC ctie Brst transition taken before a y other transition. Only one initial state is allotvctl in P. 4 tliagrani.
:Vote: The trrlnsition jrom fin initial pnint to the sturt state clin have rm wtirm: tiu.u~-
tuer. the other trm.sition /eutirres. inchrrling R pmnl condition rrnd triqger event. rire nnot ulloweh
D.3.3 Transitions to and from the initial state
Only one outgoing transition can be placecl korn the initial point. There can be several incoming transitions to the initiai state. in this case the initiai state acts like a junction point n-hicti forces the behavïor back through the initiai transition. If the initiai transition is usetl to completeiy initialize an object, then any incoming transition to the initiai state d i effectively reset the behavior of an object without haiing to destroy then re-create it-
D.3.4 No initial transition
A state machine does not require an initiai transition. In Bgure D.1. mhen the state machine is created we are in the Tcp (parent) state until the triggeriag event for tL is received. :Vote: The initial state icon is a srnaIl fiiled n'rcle.
l
Figure DA: -\O initial Transition.
D.3.5 Group transitions
Group transitions are transitions from hierarchical states that arc cornmon to al1 the stibstates aitliin that state. Thus conimon behaviour that is normally representecl by equivalent trcansitions tiom every state. can be reprcsented by a single transition originating frorn the containing hierarchical state.
D.4 Choice points
Choice points allow a single transition to be split into two outgoing transition segments. each of whicti c m terminate on a different state. The decision of mhich branch to take is made after thc transition is taken.
Choice points are motiVatcd mainly by practical considerations: it often happeris that the clecisioti on which state to tetminate a transition can only be made following certain calcii1ntions. Each choice point has an associated boolean predicate that is etaiuated after the incoming transition action is esecuterl. Depending on the truth d u c of this predicate. one or the other branch is taken.
D.4.1 Choice Point Example
A very cornmon use of choice points is to count events. That is when the decision of transferring €rom one state to the next depends on the number of events that have occurred. For example. in figure D.?, if a player can on& draw car& once he has received 3 car& from a cIeder. ?ou could mode1 this bchavior the foiiowing way:
The pIayer would keep track of how man? car& he has received and evety time a new carcl is received would test if he has enough cards.
D.5 History hierarchical state machines
The history of a state is dehed as the substate that was the Iast current substate the Iast time the state nias active. In the case of simple states, they are alnays the Iast active state.
Initial .
Draw
AIICards
I
Figure D.2: Choice Point
History is iiseful u-hen dealing Fvith situations where an event cakes control away frorn the ctirrent Stace iuid initiates ir separate behavior sequenee for handling the new event. The netv seqiience can involve nex States and transitions. Hoivever, once cornpleted. ne often want to resunie froni the point hefore the interruption occurred.
D.5.1 Continuation Ends shallow history, deep history and default
iVhen a transition terminates on shallow history, the active substate becornes the niost rec:encly active substate prior to this e n t . Whereas. deep hiscor?; implies retttrning CO Iiistory at al1 state liierarchy levels.
D.5.2 History Example
X comnion use of histor?; is shown in the state machine in figure D.3. In this case transition ce is a self-transition that has a trigger for an event that none of the substates can handle. When thar event occurs the self-transition %<II 6re then go to history. meaning that it uiii revert to the Iast active substate. The effect is to perform event handling n-ithout chan,& the state of the system.
in figure D.3 if the curent active state is 52. then when ee is triggered, the exit action for S2 tt-ill be taken, then the actions for ee d l e-ecute. and ha l ly the entry action for S2 d l he eseciited.
D .6 Transition segments
Transitions that span muitiple hierarchies. thuç cross state boundaries. change conte\% on the frorn the source to the destination state. Therefore they must be partitioned into different segments. Each transition segment has a distinct name' and only the originating segment lias a trigger deherl.
The sum of al1 transition segments is called the transition chain. Transition chains are excciitecl in one single riin-to-completion step.
D.6.1 Junction points
Junction points provide a way of correlating differcnt segments of a transition that sp,m multiple hierarchical state machines. Jiinction points are locatetl on the boundary of statcs. and reprsent cither the source or the destination of a transition segrlent. .Jttnction points are split into those ttiat terminate on the state boundary. history junctions. and those mhich are notations t h continiie rvithin the state. SimiIarIy transitions outgoing from a hierarchicdiy nesteci state are rlivirled into those that terminate on the enclosing state and those that continue Gom the state b o u n d a ~ - to ;L t a r g t state. Every transition, whetlier composecl of multiple segments or of a single se,ment. cventually terminates to a junction point.
D.6.2 Joining transition segments
'in-O or more transition segments can converge on to a single jtinction point. This allon-s multiple trxisitiotis to be definecl. whkh perform the same action in response to an event. and share the s m e destination state. Iu figure D.4 t4 and t9 both terminate on the sanie junctioti point. OnIy the originating transition segments can have cnggers defined. For example. dthough t6 can have actions. it camot have a trigger.
D.6.3 Continuation Kind Property
When a transition terminates on a composite stat e (there are no other transition s continuing from the junction point). the behalor of the state machine a t this point is determineci bu the Continuation kind propecty of the terminating junction point.
This selection specifies the semantics for how the state &tory will be used when there is no continuing transition. There are three options:
O Defaultspecifies that the default (initiai) tr'msition shodtf be taken.
O Historyspecifies rhat the state shouid return to shdow history.
Dcep Historyspccifics that the state should return to tleep history.
menning that ai1 substates also return to histor. This is the behavior for al1 capsule Stace machines. so ic is autornaticalIy selected. The defauit for capsule statc machines is to always go [O deep h i s to r~ so deep history niII bc automatically selected for capsule States. and the selections !vil1 Lie grqerl ririt.
.Junction points are clispla~cd differentlj- depending ahether a scate is shown from an abstracr view. as a substate uE anotlier containing state. or whether stiotvn frotn a cletailet1 vica. the insicte of n composite state. Junction points viewetl from the nbstrnct state view are almays shown .~s a ~aliti dot. Hon*cver. from a detail view üke figure DA. different junctions are shonn with gnphicd clues. Ti~bles D.1 ;uid D.2 describe in cletail the cypes of junctioris shown in figure D.4.
Figure DA: Junction Types
D .7 Transition Selection Rules
Wùen an event is ready to be processed by a state machine. a search for a candidate transition cakes place to determine which one wilI be taken. A transition is said to be euabIed if its trigger is satisfied bu the current event. meaning that the transition has the same event and incerface specified as the current event . and the guard condition evaluates to crue.
The searcli nrder is definecl hy the folloshg algorithrn:
Type of Junction internal
external
external in- corning
external oiit- going
Table D.1: Continuing .Jtinctions.
Example t8 originates fiom an internai junction. The junction is noc visible fiom the abstract x-iem of this state. tT originates from an extemal junction. This junction is visible from the extemai view of the state s'o that it c m eventually be connected to. Cntil an extemai junction hiis been connectecl it behaves like an internal junction. t6 originates From an estemal incoming junc- tion. The label of the transition segment that is attached to che junction is shonm as e6. t6 cannot have n trigger. t.5 teminates on an extemal outgoing junc- tion. The next transition segment is shosn as CS. t5 has ii trigger.
1. The search begins in the innerrnost ciment active state.
Shown as... a small circle on the state bound- a-y
a solid circle connected s i t h a solid line to the container state's b o u n d a .
a solicl circle connected u-irh ;i solicl line to ttic container statc's bniinclary with an a r ro r nt ch^ circle. a solid circle connectecl nich n solicl line to the container state's baiindan with an arrow ac the container state boiinciary.
2. n'ithin the scope of the innermost current active çtate. transitions are cnluatect seqiientiall- if a transition is enabled. the search terminates and the corresponding transition is tnken.
3. If rio transition is enablecl in the current scope. the search in step 2 is repeated for the next tiigher scope. one level iip in the state hierarchy.
4. If the top-Ievel state has heen reachert and no transitions are enablecl. then the current ewnt is ciiscarcled and the state of the behaviour remains unchangeci.
Wiieii a transition is enabletl the algorithni continues with the follonring:
1. If thc enablecl transition is not an internal transition. theii eseciite tlit! esit iictiotis (if al1 substates starting with the deepest history up to the cricrerit scope.
2. If the enabled transition is an internat transition, then none of the substates exit and entry actions are executed.
3. Euecittc the enablecl transition (this includes ail the transition segments) actions. This chairi ultimateiy terminaces on a simple state. Yote that =ecuting the transition may inclutle esecuting other state entcy and exit actions as meii.
4. The termiriating simple state becomes the current active state.
D .7.1 Transit ion Select ion Rules Example
in figure D.5 tL is a group transition outside of SI. IFS11 is the current active state vhen the currenc event enabled transition tl. then the foilowing actions would be evecuted as part of the transition c h a h
L. Exit action for SI1
external r t8 terminates on an internai tenninating junc- tioa. This junction is not visible kom the eu- temal rie%- of chc state.
t7 terminates on an extenid tenninating junc- tion. This junction is visible from the externat tien: O€ the state. If this junction is ever osed to cotlnect another transition segment. it n5ll become a contiming junction. since it Nil1 no longer terminace a transition. e3 teminates on an es~ernai ccrminating junc- cion. This junction is nsiblc From the emcrnal view of the state.
S h o w as... a drclc rc-ith the lctter 'H' (for &tory) connected to the con- tainer states boiinclary n-ith n dotted iine. a circlc a i th the letter 'H' (€or history) connected to the con- tainer states boiindary wicli ii
solid line.
a circle with the letter 'H' (for history) connected to the con- tainer states bauntlary with ;L
SOM line with a r r w irt the circle.
TabIe D.2: Tenninating .Iiinctians (co history).
1. Exit action For S 1
3. Action code For c l
4. Entry action for S L
5. E n t l action For SI1
SI1 rernains the crirrent activc state after the transition is cornplete,
Figure D.5: Transition SeIection Rdes
Appendix E
Capsule Statemachine Diagrams
This appendis contnins stnte diagarns for the seven capsules testetl in this thesis.
E . l Experiment 1: Ce11 Phone Component
1 disconnected
Figure E.1: Ceii f hone TOP state.
active
I
Figure E.2: CeII Phone ConnectetI.
E.2 Experiment 2: Cell from Game of Life
I
Figure €2: Ceii From Game of Life TOP state.
E.3 Experiment 3: TkafEic Light Controller
I I I - ..- , tirneout
1
Figure E.4: Trafic Ligiits TOP Stace.
1
Figure E.6: Trd!ic Lights Flxhing;\mbcr.
timeout
transitioning
I
Figure E.C Trafic Lights Working.
f - amber
*
greenTimeout
Figure E.3: Traffic Lights Transitioning.
# + tirneout2
True greenTheout
I ' #
Figure E.9: Traffic Lights Green.
E.4 Experiment 4: Cruise Control
Figure E.10: Criiise Controt TOP itate.
watchDog
off f
error
t l
resume
-
Figure EU: Cruise Control On.
Figure E.12: Cruise Control Active.
E.5 Experiment 5: Radar Control Unit
I 1
\ init f initiaiiic ' scan ' rwning
l 1 w '. .J
init scan
stopScan
9
Figure E.13: Radar Control Cuit TOP State.
E.6 Experiment 6: Pop Machine
Figure E.14: Pop 4Iachine TOP.
Figure EX: Pop Machine initialize.
Figure E.16: Pop Machine Operating.
E. 7 Experiment 7: Elect ronic Warfare Jammer
Initial {ZU ~2, reset
Figure E.lT: EWS Jammer TOP State.
L5S
Figure E.18: EWS .ranimer \Varking.