mutation analysis & testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 mutation...
TRANSCRIPT
![Page 1: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/1.jpg)
Mutation Analysis & Testing
CMPT 473Software Quality Assurance
Nick SumnerWith material from Ammann & Offutt, Patrick Lam, Gordon Fraser
![Page 2: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/2.jpg)
2
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
![Page 3: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/3.jpg)
3
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
![Page 4: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/4.jpg)
4
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
Requirements
![Page 5: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/5.jpg)
5
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
Requirements Tests
![Page 6: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/6.jpg)
6
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
A
G
BC
EDF
Requirements Tests
![Page 7: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/7.jpg)
7
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
A
G
BC
EDF
ABCDGACDGABCEGACEGACEFFEGEFEFEF
Requirements Tests
![Page 8: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/8.jpg)
8
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.
C1 C2
A
G
BC
EDF
ABCDGACDGABCEGACEGACEFFEGEFEFEF Tests
Requirements Tests
![Page 9: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/9.jpg)
9
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.– But they still have difficulties finding bugs!
![Page 10: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/10.jpg)
10
How Else Can We Judge Adequacy?
● Input & graph based techniques provide requirements that measure quality.– But they still have difficulties finding bugs!– Can we try to measure that directly?
How might you go about this?
![Page 11: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/11.jpg)
11
Fault Seeding
● Insert or seed representative/typical faults
![Page 12: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/12.jpg)
12
Fault Seeding
● Insert or seed representative/typical faults● Measure how many are found or killed by the test
suite
![Page 13: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/13.jpg)
13
Fault Seeding
● Insert or seed representative/typical faults● Measure how many are found or killed by the test
suite– Effectiveness = # killed / # seeded
![Page 14: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/14.jpg)
14
Fault Seeding
● Insert or seed representative/typical faults● Measure how many are found or killed by the test
suite– Effectiveness = # killed / # seeded– Directly measures bug finding ability
![Page 15: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/15.jpg)
15
Fault Seeding
● Insert or seed representative/typical faults● Measure how many are found or killed by the test
suite– Effectiveness = # killed / # seeded– Directly measures bug finding ability
● Why might this fail?
![Page 16: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/16.jpg)
16
Fault Seeding
● Insert or seed representative/typical faults● Measure how many are found or killed by the test
suite– Effectiveness = # killed / # seeded– Directly measures bug finding ability
● Why might this fail?– What are representative faults?– Are there enough faults to be meaningful?– Did you forget to remove faults afterward?
![Page 17: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/17.jpg)
17
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original
![Page 18: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/18.jpg)
18
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs
a = b + c a = b * c
![Page 19: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/19.jpg)
19
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs– A test t kills a mutant m if t produces a different outcome
on m than the original program
![Page 20: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/20.jpg)
20
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs– A test t kills a mutant m if t produces a different outcome
on m than the original program
What does this mean?
![Page 21: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/21.jpg)
21
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs– A test t kills a mutant m if t produces a different outcome
on m than the original program● Systematically generate mutants separately from
original program
![Page 22: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/22.jpg)
22
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs– A test t kills a mutant m if t produces a different outcome
on m than the original program● Systematically generate mutants separately from
original program● The goal is to:
– Mutation Analysis – Measure bug finding ability
![Page 23: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/23.jpg)
23
Mutation Analysis & Testing
● Mutant– A valid program that behaves differently than the
original– Consider small, local changes to programs– A test t kills a mutant m if t produces a different outcome
on m than the original program● Systematically generate mutants separately from
original program● The goal is to:
– Mutation Analysis – Measure bug finding ability– Mutation Testing – create a test suite that kills a
representative set of mutants
![Page 24: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/24.jpg)
24
Mutation
● What are possible mutants?int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 25: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/25.jpg)
25
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 26: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/26.jpg)
26
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
● Some are not generally useful:
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
Why might they not be useful?
![Page 27: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/27.jpg)
27
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
● Some are not generally useful:– (Still Born) Not compilable
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 28: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/28.jpg)
28
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
● Some are not generally useful:– (Still Born) Not compilable– (Trivial) Killed by most test cases
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 29: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/29.jpg)
29
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
● Some are not generally useful:– (Still Born) Not compilable– (Trivial) Killed by most test cases– (Equivalent) Indistinguishable from original program
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 30: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/30.jpg)
30
Mutation
● What are possible mutants?
● Once we have a test case that kills a mutant, the mutant itself is no longer useful.
● Some are not generally useful:– (Still Born) Not compilable– (Trivial) Killed by most test cases– (Equivalent) Indistinguishable from original program– (Redundant) Indistinguishable from other mutants
int foo(int x, int y) { if (x > 5) {return x + y;} else {return x;}}
![Page 31: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/31.jpg)
31
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
● Mimic mistakes● Encode knowledge from other techniques
![Page 32: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/32.jpg)
32
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a;
if (b < a) { minVal = b; } return minVal;}● Mimic mistakes
● Encode knowledge from other techniques
![Page 33: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/33.jpg)
33
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { minVal = b; } return minVal;}
Mutant 1: minVal = b;
● Mimic mistakes● Encode knowledge from other techniques
![Page 34: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/34.jpg)
34
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { minVal = b; } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {
● Mimic mistakes● Encode knowledge from other techniques
![Page 35: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/35.jpg)
35
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
● Mimic mistakes● Encode knowledge from other techniques
![Page 36: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/36.jpg)
36
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; BOMB(); } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
Mutant 4: BOMB();
● Mimic mistakes● Encode knowledge from other techniques
![Page 37: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/37.jpg)
37
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; BOMB(); minVal = a; } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
Mutant 4: BOMB();Mutant 5: minVal = a;
● Mimic mistakes● Encode knowledge from other techniques
![Page 38: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/38.jpg)
38
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; BOMB(); minVal = a; minVal = failOnZero(b); } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
Mutant 4: BOMB();Mutant 5: minVal = a;Mutant 6: minVal = failOnZero(b);
● Mimic mistakes● Encode knowledge from other techniques
![Page 39: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/39.jpg)
39
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; BOMB(); minVal = a; minVal = failOnZero(b); } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
Mutant 4: BOMB();Mutant 5: minVal = a;Mutant 6: minVal = failOnZero(b);
What mimicsstatement coverage?
● Mimic mistakes● Encode knowledge from other techniques
![Page 40: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/40.jpg)
40
Mutationint min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; minVal = b; if (b < a) { if (b > a) { if (b < minVal) { minVal = b; BOMB(); minVal = a; minVal = failOnZero(b); } return minVal;}
Mutant 1: minVal = b;
Mutant 2: if (b > a) {Mutant 3: if (b < minVal) {
Mutant 4: BOMB();Mutant 5: minVal = a;Mutant 6: minVal = failOnZero(b);
What mimicsinput classes?
● Mimic mistakes● Encode knowledge from other techniques
![Page 41: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/41.jpg)
41
Mutation Analysis
MutantsMutant 1Mutant 2Mutant 3Mutant 4Mutant 5Mutant 6
![Page 42: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/42.jpg)
42
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3Mutant 4Mutant 5Mutant 6
min(1,2) 1min(2,1) 1
![Page 43: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/43.jpg)
43
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3Mutant 4Mutant 5Mutant 6
min(1,2) 1min(2,1) 1
Try every mutant on test 1.
![Page 44: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/44.jpg)
44
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3Mutant 4Mutant 5Mutant 6
min(1,2) 1min(2,1) 1Ki
lled
![Page 45: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/45.jpg)
45
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3Mutant 4Mutant 5Mutant 6
min(1,2) 1min(2,1) 1Ki
lled
Try every live mutant on test 2.
![Page 46: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/46.jpg)
46
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3
min(1,2) 1Ki
lled
Mutant 4Mutant 5Mutant 6
min(2,1) 1
Kille
d
![Page 47: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/47.jpg)
47
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3
min(1,2) 1Ki
lled
Mutant 4Mutant 5Mutant 6
min(2,1) 1
Kille
d
So the mutation score is...
![Page 48: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/48.jpg)
48
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3
min(1,2) 1Ki
lled
Mutant 4Mutant 5Mutant 6
min(2,1) 1
Kille
d
So the mutation score is... 4/5. Why?
![Page 49: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/49.jpg)
49
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3
min(1,2) 1Ki
lled
Mutant 4Mutant 5Mutant 6
min(2,1) 1
Kille
d
min3(int a, int b): int minVal; minVal = a; if (b < minVal) minVal = b; return minVal;
min6(int a, int b): int minVal; minVal = a; if (b < a) minVal = failOnZero(b); return minVal;
So the mutation score is... 4/5. Why?
![Page 50: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/50.jpg)
50
Mutation Analysis
Mutants Test SuiteMutant 1Mutant 2Mutant 3
min(1,2) 1Ki
lled
Mutant 4Mutant 5Mutant 6
min(2,1) 1
Kille
d
min3(int a, int b): int minVal; minVal = a; if (b < minVal) minVal = b; return minVal;
min3(int a, int b): int minVal; minVal = a; if (b < a) minVal = failOnZero(b); return minVal;
Equivalent to the original!There is no injected bug.
So the mutation score is... 4/5. Why?
![Page 51: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/51.jpg)
51
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
![Page 52: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/52.jpg)
52
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
# Killed#Mutants−#Equivalent
![Page 53: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/53.jpg)
53
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
# Killed#Mutants−#Equivalent
![Page 54: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/54.jpg)
54
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation
# Killed#Mutants−#Equivalent
![Page 55: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/55.jpg)
55
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation
# Killed#Mutants−#Equivalent
![Page 56: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/56.jpg)
56
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation
# Killed#Mutants−#Equivalent
![Page 57: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/57.jpg)
57
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation
# Killed#Mutants−#Equivalent
![Page 58: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/58.jpg)
58
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation?
# Killed#Mutants−#Equivalent
![Page 59: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/59.jpg)
59
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation?
# Killed#Mutants−#Equivalent
![Page 60: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/60.jpg)
60
Equivalent Mutants
● Equivalent mutants are not bugs and should not be counted
● New Mutation Score:
● Detecting equivalent mutants is undecidable in general
● So why are they equivalent?
Reachability Infection Propagation? ?
More on this later....
# Killed#Mutants−#Equivalent
![Page 61: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/61.jpg)
61
Equivalent Mutants
● Identifying equivalent mutants is one of the most expensive / burdensome aspects of mutation analysis.
![Page 62: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/62.jpg)
62
Equivalent Mutants
● Identifying equivalent mutants is one of the most expensive / burdensome aspects of mutation analysis.
min3(int a, int b): int minVal; minVal = a; if (b < minVal) minVal = b; return minVal;
Requires reasoning about whythe result was the same.
![Page 63: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/63.jpg)
63
Mutation Testing
● Given an unkilled mutant, how can we improve the test suite?
![Page 64: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/64.jpg)
64
Mutation Testing
● Given an unkilled mutant, how can we improve the test suite?
min3(int a, int b): int minVal; minVal = a; if (b < a) minVal = failOnZero(b); return minVal;
![Page 65: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/65.jpg)
65
Mutation Testing
● Given an unkilled mutant, how can we improve the test suite?
min3(int a, int b): int minVal; minVal = a; if (b < a) minVal = failOnZero(b); return minVal;
min(2,0) 0New Test:New Score: 5/5
![Page 66: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/66.jpg)
66
Mutation Operators
● The mutants should guide the tester toward an effective test suite
![Page 67: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/67.jpg)
67
Mutation Operators
● The mutants should guide the tester toward an effective test suite– Need a 'representative' pool of mutants
idea: “If there is a fault, there is a mutant to match it”
![Page 68: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/68.jpg)
68
Mutation Operators
● The mutants should guide the tester toward an effective test suite– Need a 'representative' pool of mutants
idea: “If there is a fault, there is a mutant to match it”– Need a rigorous way of creating mutants
![Page 69: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/69.jpg)
69
Mutation Operators
● The mutants should guide the tester toward an effective test suite– Need a 'representative' pool of mutants
idea: “If there is a fault, there is a mutant to match it”– Need a rigorous way of creating mutants
● Mutation Operators– Systematic changes that may be applied to produce
mutants
![Page 70: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/70.jpg)
70
Mutation Operators
● The mutants should guide the tester toward an effective test suite– Need a 'representative' pool of mutants
idea: “If there is a fault, there is a mutant to match it”– Need a rigorous way of creating mutants
● Mutation Operators– Systematic changes that may be applied to produce
mutants– Language dependent, but often similar
![Page 71: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/71.jpg)
71
Mutation Operators
● The mutants should guide the tester toward an effective test suite– Need a 'representative' pool of mutants
idea: “If there is a fault, there is a mutant to match it”– Need a rigorous way of creating mutants
● Mutation Operators– Systematic changes that may be applied to produce
mutants– Language dependent, but often similar
Why might they be language dependent?
![Page 72: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/72.jpg)
72
Some Mutation Operators – in Java
● Absolute Value Insertion– Each arithmetic (sub)expression is wrapped with abs(),
-abs(), and failOnZero()
w = x + y + z
Just for abs()?
![Page 73: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/73.jpg)
73
Some Mutation Operators – in Java
● Absolute Value Insertion– Each arithmetic (sub)expression is wrapped with abs(),
-abs(), and failOnZero()
w = x + y + z
Just for abs()?
w = abs(x) + y + z
w = x + abs(y) + z
w = x + y + abs(z)
w = abs(x + y) + z
w = x + abs(y + z)
w = abs(x + y + z)
Just for abs()!
![Page 74: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/74.jpg)
74
Some Mutation Operators – in Java
● Absolute Value Insertion– Each arithmetic (sub)expression is wrapped with abs(),
-abs(), and failOnZero()● Arithmetic Operator Replacement
– Each operator (+,-,*,/,%,...) is replaced with each other operator and LEFTOP and RIGHTOP (returning the named operand).
w = x + y + z
![Page 75: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/75.jpg)
75
Some Mutation Operators – in Java
● Absolute Value Insertion– Each arithmetic (sub)expression is wrapped with abs(),
-abs(), and failOnZero()● Arithmetic Operator Replacement
– Each operator (+,-,*,/,%,...) is replaced with each other operator and LEFTOP and RIGHTOP(returning the named operand).
w = x + y + z
w = x + y * z w = x + y ...
![Page 76: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/76.jpg)
76
Some Mutation Operators – in Java
● Absolute Value Insertion– Each arithmetic (sub)expression is wrapped with abs(),
-abs(), and failOnZero()● Arithmetic Operator Replacement
– Each operator (+,-,*,/,%,...) is replaced with each other operator and LEFTOP and RIGHTOP(returning the named operand).
● Relational Operator Replacement– Each operator (=,!=,<,<=,>,>=) is replaced with each
other and TRUEOP and FALSEOP
![Page 77: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/77.jpg)
77
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP
![Page 78: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/78.jpg)
78
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP
Could these be used to mimic edge coverage?
![Page 79: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/79.jpg)
79
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP● The operator replacement pattern continues...
– Assignment, Unary Insertion, Unary Deletion
![Page 80: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/80.jpg)
80
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP● The operator replacement pattern continues...
– Assignment, Unary Insertion, Unary Deletion● Scalar Variable Replacement
– Replace each variable use with another compatible variable in scope
What does compatible mean? Is it necessary?
![Page 81: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/81.jpg)
81
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP● The operator replacement pattern continues...
– Assignment, Unary Insertion, Unary Deletion● Scalar Variable Replacement
– Replace each variable use with another compatible variable in scope
● Bomb Statement Replacement– Replace a statement with BOMB()
![Page 82: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/82.jpg)
82
Some Mutation Operators – in Java
● Conditional Operator Replacement– Replace operators (&&, ||, &, |, ^) with each other and
LEFTOP, RIGHTOP, TRUEOP, FALSEOP● The operator replacement pattern continues...
– Assignment, Unary Insertion, Unary Deletion● Scalar Variable Replacement
– Replace each variable use with another compatible variable in scope
● Bomb Statement Replacement– Replace a statement with BOMB()
How does the BOMB() operatormimic statement coverage?
![Page 83: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/83.jpg)
83
Some Mutation Operators – in Java
● These are all intraprocedural (within one method)● What might interprocedural operators be?
![Page 84: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/84.jpg)
84
Some Mutation Operators – in Java
● These are all intraprocedural (within one method)● What might interprocedural operators be?
– Changing parameter values– Changing the call target– Changing incoming dependencies– ...
![Page 85: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/85.jpg)
85
Some Mutation Operators – in Java
● These are all intraprocedural (within one method)● What might interprocedural operators be?
– Changing parameter values– Changing the call target– Changing incoming dependencies– …
● And more...– Interface Mutation, Object Oriented Mutation, …
![Page 86: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/86.jpg)
86
Some Mutation Operators – in Java
● These are all intraprocedural (within one method)● What might interprocedural operators be?
– Changing parameter values– Changing the call target– Changing incoming dependencies– …
● And more...● Often just the simplest are used
![Page 87: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/87.jpg)
87
Mutation Operators
● Are the mutants representative of all bugs?● Do we expect the mutation score to be meaningful?
Ideas? Why? Why not?
![Page 88: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/88.jpg)
88
Mutation Operators
● Are the mutants representative of all bugs?● Do we expect the mutation score to be meaningful?
2 Key ideas are missing....
Ideas? Why? Why not?
![Page 89: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/89.jpg)
89
Competent Programmer Hypothesis
Programmers tend to write code that is almost correct
![Page 90: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/90.jpg)
90
Competent Programmer Hypothesis
Programmers tend to write code that is almost correct– So most of the time simple mutations should reflect the
real bugs.
![Page 91: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/91.jpg)
91
Coupling Effect
Tests that cover so much behavior that even simple errors are detected should also be sensitive enough to detect more complex errors
![Page 92: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/92.jpg)
92
Coupling Effect
Tests that cover so much behavior that even simple errors are detected should also be sensitive enough to detect more complex errors
– By casting a fine enough net, we'll catch the big fish, too (sorry dolphins)
![Page 93: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/93.jpg)
93
Higher Order Mutants?
Suppose traditional mutations are too simple● How could mutants be made that are more
realistic?
![Page 94: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/94.jpg)
94
Higher Order Mutants?
Suppose traditional mutations are too simple● How could mutants be made that are more
realistic?● Combine apply multiple mutation operators...
What will this do?
![Page 95: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/95.jpg)
95
Higher Order Mutants?
Suppose traditional mutations are too simple● How could mutants be made that are more
realistic?● Combine apply multiple mutation operators...● Carefully. Want to catch subtle interactions.
![Page 96: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/96.jpg)
96
Higher Order Mutants?
Suppose traditional mutations are too simple● How could mutants be made that are more
realistic?● Combine apply multiple mutation operators...● Carefully. Want to catch subtle interactions.● Still an emerging area.
![Page 97: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/97.jpg)
97
What Problems Remain?
● Scale (there are a lot of tests)
![Page 98: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/98.jpg)
98
What Problems Remain?
● Scale (there are a lot of tests)● Equivalence
![Page 99: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/99.jpg)
99
What Problems Remain?
● Scale (there are a lot of tests)● Equivalence
● Scale may be attacked in many ways
Ideas?
![Page 100: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/100.jpg)
100
What Problems Remain?
● Scale (there are a lot of tests)● Equivalence
● Scale may be attacked in many ways– Coverage filters– Short circuiting tests– Testing mutants simultaneously
![Page 101: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/101.jpg)
101
What Problems Remain?
● Scale (there are a lot of tests)● Equivalence
● Scale may be attacked in many ways– Coverage filters– Short circuiting tests– Testing mutants simultaneously
● Can also modify mutation criteria to help with both...
![Page 102: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/102.jpg)
102
Mutation Criteria
● Recall: If a test can detect a mutant, that mutant is killed by the test.
![Page 103: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/103.jpg)
103
Mutation Criteria
● Recall: If a test can detect a mutant, that mutant is killed by the test.
What does it mean if a mutant was killed?
![Page 104: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/104.jpg)
104
Mutation Criteria
● Recall: If a test can detect a mutant, that mutant is killed by the test.
What does it mean if a mutant was killed?
What does it mean if a mutant was not killed?
![Page 105: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/105.jpg)
105
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)
![Page 106: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/106.jpg)
106
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)
Reachability Infection Propagation
![Page 107: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/107.jpg)
107
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
![Page 108: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/108.jpg)
108
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
Reachability Infection Propagation?
![Page 109: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/109.jpg)
109
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but might not propagate.
How might this happen?
![Page 110: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/110.jpg)
110
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but might not propagate.
Mutation Criteria
How might this happen?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
![Page 111: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/111.jpg)
111
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How might this happen?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 10, b = 5
![Page 112: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/112.jpg)
112
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How might this happen?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 10, b = 5
minVal = 5
![Page 113: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/113.jpg)
113
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How might this happen?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 10, b = 5
minVal = 5
minVal = 5
![Page 114: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/114.jpg)
114
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How might this happen?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 10, b = 5
minVal = 5
minVal = 5
return 5
![Page 115: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/115.jpg)
115
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How can we strongly kill the mutant instead?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
![Page 116: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/116.jpg)
116
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How can we strongly kill the mutant instead?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 5, b = 10
![Page 117: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/117.jpg)
117
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How can we strongly kill the mutant instead?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 5, b = 10
minVal = 10
![Page 118: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/118.jpg)
118
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
How can we strongly kill the mutant instead?
int min(int a, int b) { int minVal; minVal = b; // was a if (b < a) { minVal = b; } return minVal;}
a = 5, b = 10
minVal = 10
return 10
![Page 119: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/119.jpg)
119
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but might not propagate.
What might an equivalent mutant look like?
int min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
![Page 120: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/120.jpg)
120
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
What might an equivalent mutant look like?
int min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; if (b < minVal) { minVal = b; } return minVal;}
![Page 121: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/121.jpg)
121
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but doesn't propagate.
They always behave the same way!
int min(int a, int b) { int minVal; minVal = a; if (b < a) { minVal = b; } return minVal;}
int min(int a, int b) { int minVal; minVal = a; if (b < minVal) { minVal = b; } return minVal;}
![Page 122: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/122.jpg)
122
Mutation Criteria
● Strongly Killed– A test strongly kills a mutant m if m(t) produces different
output than p(t)● Weakly Killed
– A test weakly kills a mutant m if m(t) produces different internal state than p(t)
– Reachable, infects, but might not propagate.Leading to...
![Page 123: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/123.jpg)
123
Mutation Criteria
● Strong Mutation Coverage– For each mutant, the test suite contains a test that
strongly kills the mutant
![Page 124: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/124.jpg)
124
Mutation Criteria
● Strong Mutation Coverage– For each mutant, the test suite contains a test that
strongly kills the mutant● Weak Mutation Coverage
– For each mutant, the test suite contains a test that weakly kills the mutant
![Page 125: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/125.jpg)
125
Mutation Criteria
● Strong Mutation Coverage– For each mutant, the test suite contains a test that
strongly kills the mutant● Weak Mutation Coverage
– For each mutant, the test suite contains a test that weakly kills the mutant
How might weak coverage help with equivalence?
![Page 126: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/126.jpg)
126
Mutation Criteria
● Strong Mutation Coverage– For each mutant, the test suite contains a test that
strongly kills the mutant● Weak Mutation Coverage
– For each mutant, the test suite contains a test that weakly kills the mutant
How might weak coverage help with equivalence?
How might weak coverage help with scalability?
![Page 127: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/127.jpg)
127
Mutation Criteria
● Strong Mutation Coverage– For each mutant, the test suite contains a test that
strongly kills the mutant● Weak Mutation Coverage
– For each mutant, the test suite contains a test that weakly kills the mutant
How might weak coverage help with equivalence?
How might weak coverage help with scalability?
Is there any reason to prefer strong coverage?
![Page 128: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/128.jpg)
128
Mutation Testing
● Considered one of the strongest criteria
Why?
![Page 129: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/129.jpg)
129
Mutation Testing
● Considered one of the strongest criteria– Mimics some input specifications– Mimics some graph coverage (node, edge, …)
![Page 130: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/130.jpg)
130
Mutation Testing
● Considered one of the strongest criteria– Mimics some input specifications– Mimics some graph coverage (node, edge, …)
● Massive number of criteria.
Why?
![Page 131: Mutation Analysis & Testingwsumner/teaching/473/07-mutation.pdf · 2016. 2. 18. · 23 Mutation Analysis & Testing Mutant – A valid program that behaves differently than the original](https://reader033.vdocuments.us/reader033/viewer/2022060921/60ace7d12cde3a4aee376d8c/html5/thumbnails/131.jpg)
131
Mutation Testing
● Considered one of the strongest criteria– Mimics some input specifications– Mimics some graph coverage (node, edge, …)
● Massive number of criteria.● Still not always the most tests.
Why?