![Page 1: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/1.jpg)
Yue Jia, Mark HarmanKing’s College London
CREST
Constructing Subtle Faults Using Higher Order Mutation Testing
Higher Order Mutation Testing
![Page 2: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/2.jpg)
Agenda
BackgroundProblemsSolutionEmpirical StudySummary
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary
![Page 3: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/3.jpg)
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 4: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/4.jpg)
Mutation Testing
Which test case is better?
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 5: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/5.jpg)
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 6: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/6.jpg)
Mutation Testing
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 7: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/7.jpg)
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
First Order MutantOriginal Program
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 8: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/8.jpg)
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
Higher Order MutantFirst Order MutantOriginal Program
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 9: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/9.jpg)
Mutation Testing
Generate Mutants
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
Higher Order MutantFirst Order MutantOriginal Program
------------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
-----------------------------------------------------------------------------------------------------
+------------------------------------------------------------------------------------------------------
>
First Order Mutant Higher Order Mutant
Mark Harman and Yue Jia King’s College London, CREST Centre
Problems Solution Empirical Study SummaryBackground
![Page 10: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/10.jpg)
Problem 1970s
Computational cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 11: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/11.jpg)
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 12: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/12.jpg)
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 13: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/13.jpg)
Problem 1970s
Computational cost
e.g. Triangle LoC 50
FOM 500 HOM 2500
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study SummaryProblems
![Page 14: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/14.jpg)
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 15: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/15.jpg)
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 16: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/16.jpg)
Solution 1980s
Simple Complex cost
If a test set kills simple faults, it also kills the complex faults
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 17: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/17.jpg)
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 18: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/18.jpg)
Solution 1990s
Selective mutation / Mutation sampling
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 19: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/19.jpg)
Selected First Order Mutants
+ -
> <
a b
Trivial Fault
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 20: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/20.jpg)
Subtle Fault
90% of the real faults are complex faults (Purushothaman and Perry)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------HOMs -> Subtle faults
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 21: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/21.jpg)
Subtle Fault
Solution Programmer
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 22: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/22.jpg)
Subtle Fault
Solution Programmer
1
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 23: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/23.jpg)
Subtle Fault
Solution Programmer
2
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 24: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/24.jpg)
Subtle Fault
Solution Programmer
3
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 25: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/25.jpg)
Subtle Fault
4
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
Solution Programmer
![Page 26: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/26.jpg)
Solution 1990s
Selective mutation / Mutation sampling
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 27: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/27.jpg)
Solution 1980s
Coupling effect hypothesis
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 28: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/28.jpg)
cost
Problem 1970s
Computational cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Solution Empirical Study SummaryProblems
![Page 29: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/29.jpg)
Solution
Higher order mutation testing
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 30: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/30.jpg)
Solution
Higher order mutation testing
…Search Based Optimization
cost
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 31: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/31.jpg)
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 32: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/32.jpg)
Higher Order Mutant
Most common case
Test set T
Ta
Tb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 33: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/33.jpg)
Higher Order Mutant
Uncommon case
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 34: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/34.jpg)
Higher Order Mutant
Subsuming higher order mutant
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 35: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/35.jpg)
Higher Order Mutant
Subsuming higher order mutant
TaTb
Tc
Tabc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 36: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/36.jpg)
Higher Order Mutant
Strongly Subsuming higher order mutant
TaTb
Tc
Tabc
If a test case kills HOM Mabc, it also kills the FOM Fa, Fb and Fc
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 37: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/37.jpg)
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effortReduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 38: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/38.jpg)
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 39: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/39.jpg)
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Test cases Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 40: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/40.jpg)
Higher Order Mutation Testing
Increase the subtletyReduce the Oracle effort
Mutants Test cases Reduced number of equivalent mutants
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Empirical Study SummarySolution
![Page 41: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/41.jpg)
Results
Mutants LoC FOM sHOM
Triangle 50 584 47Tcas 150 679 98Schedule2 350 1014 78Totinfo 500 2570 320Printtokens 750 866 67Space 6000 7570 522
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary
![Page 42: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/42.jpg)
Algorithms
Greedy Hill ClimbingGenetic Algorithm
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 43: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/43.jpg)
Results
RND GA GR HC0
10
20
30
40
50
60
70
80
90
Subsuming HOMs
Valuable HOMs
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 44: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/44.jpg)
Triangle Program
Classify triangle by the lengths of the sides
Equilateral Isosceles Scalene
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 45: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/45.jpg)
Strongly Subsuming HOM
if(trian == 1 && a + b > c)
if(trian > 1 && a + b > c)
if(trian == 1 && a + b <= c)
if(trian == 1 && a + b <= c)
a == c && a + b > c && a + c <= b
b == c && a + b > a && b + c <= a
a == b && a + b > c
a == b && a + b <= c
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 46: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/46.jpg)
Tool
MiLuDeer
Horse
Cow
Donkey
rare valuable
Strongly Subsuming HOM
(Père David's Deer)
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 47: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/47.jpg)
www.dcs.kcl.ac.uk/pg/jiayue/milu
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution SummaryEmpirical Study
![Page 48: Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing](https://reader033.vdocuments.us/reader033/viewer/2022051316/56649efc5503460f94c0f7ce/html5/thumbnails/48.jpg)
Summary
Higher order mutation testingQuality of mutantComputational Cost
QuestionAre real faults higher order mutants?
Thanks
Mark Harman and Yue Jia King’s College London, CREST Centre
Background Problems Solution Empirical Study Summary