yue jia, mark harman king’s college london crest constructing subtle faults using higher order...
Post on 13-Jan-2016
219 Views
Preview:
TRANSCRIPT
Yue Jia, Mark HarmanKing’s College London
CREST
Constructing Subtle Faults Using Higher Order Mutation Testing
Higher Order Mutation Testing
Agenda
BackgroundProblemsSolutionEmpirical StudySummary
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Which test case is better?
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
First Order MutantOriginal Program
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
Higher Order MutantFirst Order MutantOriginal Program
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
Higher Order MutantFirst Order MutantOriginal Program
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
-----------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
First Order Mutant Higher Order Mutant
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
Problem 1970s
Computational cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study SummaryProblems
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution 1980s
Simple Complex cost
If a test set kills simple faults, it also kills the complex faults
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution 1990s
Selective mutation / Mutation sampling
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Selected First Order Mutants
+ -
> <
a b
Trivial Fault
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
90% of the real faults are complex faults (Purushothaman and Perry)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------HOMs -> Subtle faults
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
Solution Programmer
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
Solution Programmer
1
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
Solution Programmer
2
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
Solution Programmer
3
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Subtle Fault
4
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution Programmer
Solution 1990s
Selective mutation / Mutation sampling
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
cost
Problem 1970s
Computational cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution
Higher order mutation testing
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Solution
Higher order mutation testing
…Search Based Optimization
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutant
Most common case
Test set T
Ta
Tb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutant
Uncommon case
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutant
Subsuming higher order mutant
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutant
Subsuming higher order mutant
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutant
Strongly Subsuming higher order mutant
TaTb
Tc
Tabc
If a test case kills HOM Mabc, it also kills the FOM Fa, Fb and Fc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Test cases Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Test cases Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
Results
Mutants LoC FOM sHOM
Triangle 50 584 47Tcas 150 679 98Schedule2 350 1014 78Totinfo 500 2570 320Printtokens 750 866 67Space 6000 7570 522
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary
Algorithms
Greedy Hill ClimbingGenetic Algorithm
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
Results
RND GA GR HC0
10
20
30
40
50
60
70
80
90
Subsuming HOMs
Valuable HOMs
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
Triangle Program
Classify triangle by the lengths of the sides
Equilateral Isosceles Scalene
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
Strongly Subsuming HOM
if(trian == 1 && a + b > c)
if(trian > 1 && a + b > c)
if(trian == 1 && a + b <= c)
if(trian == 1 && a + b <= c)
a == c && a + b > c && a + c <= b
b == c && a + b > a && b + c <= a
a == b && a + b > c
a == b && a + b <= c
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
Tool
MiLuDeer
Horse
Cow
Donkey
rare valuable
Strongly Subsuming HOM
(Père David's Deer)
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
www.dcs.kcl.ac.uk/pg/jiayue/milu
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
Summary
Higher order mutation testingQuality of mutantComputational Cost
QuestionAre real faults higher order mutants?
Thanks
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary
top related