mutation testing
DESCRIPTION
TRANSCRIPT
![Page 1: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/1.jpg)
An Analysis and Survey of the Development of Mutation Testing[JH09]
About 30 minutes Tao [email protected]
Software Engineering LaboratoryDepartment of Computer Science, Sun Yat-Sen University
With thanks to Yue Jia and Mark HarmanThese slides are mainly extracted from Jia and Harman’s survey work
May 2011Sun Yat-Sen University, Guangzhou, China
1/371/37
[JH09] Yue Jia, Mark Harman (September 2009). "An Analysis and Survey of the Development of Mutation Testing" (PDF). CREST Centre, King's College London, Technical Report TR-09-06.
![Page 2: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/2.jpg)
Outline
Objectives Scope Classification of Research Fundamental Hypotheses Related Concepts Future Trend Tools
![Page 3: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/3.jpg)
Objectives of Mutation Testing
Provide “mutation adequacy score” Measure the effectiveness of a test set in
terms of its ability to detect faults
3/20
![Page 4: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/4.jpg)
Scope of Mutation Testing
The unit level The integration level The specification level As a white-box unit test technique
![Page 5: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/5.jpg)
Theoretical work on Mutation Testing
Hypotheses supporting Mutation Testing Optimization techniques
techniques for reducing computational cost techniques for the detection of equivalent mutants
![Page 6: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/6.jpg)
Practical work on Mutation Testing
Applications of Mutation Testing Development work on Mutation Testing tools Empirical work
![Page 7: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/7.jpg)
Fundamental Hypotheses
![Page 8: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/8.jpg)
Competent Programmer Hypothesis (CPH)
Programs can be corrected by a few small syntactical changes.
![Page 9: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/9.jpg)
Coupling Effect
Test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults A simple fault is represented by a simple mutant
which is created by making a single syntactical change
A complex fault is represented as a complex mutant which is created by making more than one change
![Page 10: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/10.jpg)
A Mutation Operator
A transformation rule that generates a mutant from the original program
Generate ‘realistic faults’
![Page 11: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/11.jpg)
Behaviors of Mutation Operators
Mutation Objects Variables Expressions
Mutation Operation Replacement Insertion Deletion
![Page 12: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/12.jpg)
Problems of Mutation Analysis
Executing the enormous number of mutants Human effort
The human oracle problem The equivalent mutant problem
![Page 13: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/13.jpg)
Cost Reduction Techniques
![Page 14: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/14.jpg)
Classification of Cost Reduction Techniques
Reduction of the generated mutants (may be taken into consideration in our approach for Fault Localization)
Reduction of the execution cost
![Page 15: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/15.jpg)
Mutants Reduction Techniques
Mutant Sampling Random Based on the Bayesian sequential probability ratio
test (SPRT)
Mutant Clustering Based on killable test cases
Selective Mutation By reducing the number of mutation operators applied. Ignore operators ASR and SVR - redundant generation Only using ABS and ROR
![Page 16: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/16.jpg)
Evaluation of Mutants Reduction
A mean mutation score Reduction in the number of mutants Number of equivalent mutants
![Page 17: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/17.jpg)
Execution Cost Reduction Techniques
Strong Mutation Weak Mutation Firm Mutation
by providing a continuum of intermediate possibilities - compare state, which lies between execution (Weak Mutation) and the final output (Strong Mutation).
![Page 18: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/18.jpg)
Equivalent Mutant Detection Techniques
10% to 40% of mutants which are equivalent
![Page 19: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/19.jpg)
Empirical Study
Compare mutation criteria with data flow criteria such as 'all-use'
Compare mutants with real faults
![Page 20: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/20.jpg)
Future Trend
A need for high quality higher order mutants A need to reduce the equivalent mutants A preference for semantics over syntax
mutation An interest in achieving a better balance
between cost and value A pressing need to generate test cases to kill
mutants
![Page 21: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/21.jpg)
Life circle of Mutation Testing
Generate mutants based on specified mutation operations
Reduce mutants Run mutants against a test suite
![Page 22: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/22.jpg)
Mutation Testing Tools
![Page 23: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/23.jpg)
Goals of Mutation to Enhance Fault Localization
Not aim to simulate real faults Analyze the impact of mutation on the
suspiciousness of the mutated statement
![Page 24: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/24.jpg)
Published C Mutation Testing Tools
Name Application Year Character Available Suitable
MILU C 2008Higher Order Mutation, Search-based technique, Test harness embedding
Yes No
MUFORMAT C 2008 Format String Bugs No No
ESPT C/C++ 2008 Tabular No No
CSAW C 2007Variable type optimization
Yes No
ExMAn C, Java 2006 TXL No No
SESAME C, Lustre, Pascal 2006 Assembler Injection No No
Certitude C/C++ 2006 General (Commercial) Commercially No
Plextest C/C++ 2005 General (Commercial) Commercially No
Proteum/IM 2.0 C 2001Interface Mutation,
Finite State MachinesYes Yes
Insure++ C/C++ 1998Source Code
Instrumentation (Commercial)
Commercially No
TUMS C 1995Mutant Schemata
GenerationNo No
Proteum 1.4 C 1993Interface Mutation,
Finite State MachinesNo No
![Page 25: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/25.jpg)
Proteum
Environment Variable: PROTEUMIMHOME li -P pre-filename [-D directory] source-filename LI-filename
Call gcc to preprocess the source code file source-filename.c and produce a file pre-filename.c. Then parse the file pre-filename.c and generate various info files
opmuta [-<operator> n m] [-all n m] source-filename Li-filename Apply mutant operators to the source code and LI files. As ouput,
opmuta produces a description file in a format that muta is able to read and include in the mutant database.
![Page 26: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/26.jpg)
Proteum – An Example: Space
space.c
li ~/smart_debugger/toolkit/proteum/li -P pre-space space li-space
li-space.nli li-space.fun li-space.cgr li-spaec.gfc
source codestrutt.h include
statement info
pre-space.c
function info call graph info def-use pair info
opmuta ~/smart_debugger/toolkit/proteum/opmuta -O Operators pre-one_statement li-one_statement > mutants.txt
mutants.txt
all mutants info
GenetrateMutants.py
Proteum.py
include
Mutants function name by directory
line number by mutant.dsc
mutation operator by mutant.dsc
![Page 27: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/27.jpg)
Thoughts of Self-Made Mutants Generator
I once consider to implement a self-made mutants generator. Only mutate one statement Not aim to simulate real faults
IPO Input
A piece of source code Line number to mutate Mutation operators
Process Preprocess -> Scan -> Parse -> Mutate
Output Mutants
![Page 28: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/28.jpg)
Collection of Compiler Front-end Tools
GCC-XML …failed E.g.
source code: http://www.gccxml.org/HTML/example1in.html
Parse-tree: http://www.gccxml.org/HTML/example1out.html
LLVM … not try, good for Objective-C? http://llvm.org/
![Page 29: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/29.jpg)
Collection of Compiler Front-end Tools
GCC … failed, maybe no parse-tree for GCC -fdump-tree-fixup cfg-lineno
http://stackoverflow.com/questions/697817/how-to-make-gcc-spit-out-a-mapping-from-flow-graphs-to-source-code-line-numbers
-fdump-tree-all -fdump-rtl-all http://stackoverflow.com/questions/1496497/how-can-i-see-parse-tree-intermediate-code-optimization-code-and-assembly-code
-fprofile-arcs -ftest-coverage gcno gcda: not readable for human beings gcov: offer the line number, separated statements
![Page 30: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/30.jpg)
Useful links
http://en.wikipedia.org/wiki/Mutation_testing http://cs.gmu.edu/~offutt/rsrch/mut.html http://www.dcs.kcl.ac.uk/pg/jiayue/repository/
![Page 31: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/31.jpg)
Q & A
31/37
![Page 33: Mutation testing](https://reader033.vdocuments.us/reader033/viewer/2022061202/546b2507af795902048b5436/html5/thumbnails/33.jpg)
Abstract
33/20
Fault localization is a technique that aims to pinpoint faults by analyzing program execution spectrum, while mutation is a testing technique used by generating faulty programs called mutants. Most of existing research in fault localization focuses on the spectrum of the given programs. Yet there are few attempts to introduce mutation’s impact to enhance fault localization techniques. In this paper, we propose a strategy that automatically introduces mutation into fault localization techniques, and present variations of a heuristic method to compute suspiciousness of each statement by considering the impact of mutation. To validate our method, experiments is conducted on benchmark programs, namely Siemens, grep, gzip, sed, space, flex, make, vim, and bash. Results indicate that the method can help programmers locate faults more effectively than methods without mutation.