IMPROVE YOUR TEST QUALITY WITH MUTATION TESTINGNICOLAS FRANKEL@nicolas_frankel
@nicolas_frankel
ME, MYSELF AND IDeveloper & Architect•As Consultant
Teacher/trainerBook AuthorBlogger
@nicolas_frankel #mutationtesting 3
SHAMELESS SELF-PROMOTION
MANY KINDS OF TESTINGUnit TestingIntegration TestingEnd-to-end TestingPerformance TestingPenetration TestingExploratory Testingetc.
@nicolas_frankel #mutationtesting 4
THEIR ONLY SINGLE GOALEnsure the Quality of the production code
@nicolas_frankel #mutationtesting 5
THE PROBLEMHow to check the Quality of the testing code?
@nicolas_frankel #mutationtesting 6
CODE COVERAGE“Code coverage is a measure used to describe the degree to which the source code of a program is tested”
--Wikipediahttp://en.wikipedia.org/wiki/
Code_coverage
@nicolas_frankel #mutationtesting 7
MEASURING CODE COVERAGECheck whether a source code line is executed during a test•Or Branch Coverage
@nicolas_frankel #mutationtesting 8
@nicolas_frankel #mutationtesting 9
COMPUTING CODE COVERAGECC: Code Coverage(in percent)Lexecuted: Number of executed lines of codeLtotal: Number of total lines of code
JAVA TOOLS FOR CODE COVERAGEJaCoCoCloverCoberturaetc.
@nicolas_frankel #mutationtesting 10
100% CODE COVERAGE?“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebobmarti
n/status/55966620509667328
@nicolas_frankel #mutationtesting 11
@nicolas_frankel #mutationtesting 12
ASSERT-LESS TESTING@Testpublic void add_should_add() {new Math().add(1, 1);
}
But, where is the assert?
As long as the Code Coverage is OK…
CODE COVERAGE AS A MEASURE OF TEST QUALITYAny metric can be gamed!Code coverage is a metric…
⇒ Code coverage can be gamed• On purpose• Or by accident
@nicolas_frankel #mutationtesting 13
CODE COVERAGE AS A MEASURE OF TEST QUALITYCode Coverage lulls you into a false sense of security…
@nicolas_frankel #mutationtesting 14
THE PROBLEM STILL STANDSCode coverage cannot ensure test quality• Is there another way?
Mutation Testing to the rescue!
@nicolas_frankel #mutationtesting 15
@nicolas_frankel #mutationtesting 16
THE CAST
William StrykerOriginal Source Code
Jason StrykerModified Source Code
a.k.a “The Mutant”
@nicolas_frankel #mutationtesting 17
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@nicolas_frankel #mutationtesting 18
STANDARD TESTING
✔Execute Test
@nicolas_frankel #mutationtesting 19
MUTATION TESTING
?Execute SAME Test
MUTATION
@nicolas_frankel #mutationtesting 20
MUTATION TESTING
✗
✔Execute SAME Test
Execute SAME Test
Mutant Killed
Mutant Survived
KILLED OR SURVIVING?Surviving means changing the source code did not change the test result• It’s bad!
Killed means changing the source code changed the test result• It’s good
@nicolas_frankel #mutationtesting 21
@nicolas_frankel #mutationtesting 22
TEST THE CODEpublic class Math {public int add(int i1, int i2) { return i1 + i2;}
}
@Testpublic void add_should_add() {new Math().add(1, 1);
}
✔Execute Test
@nicolas_frankel #mutationtesting 23
SURVIVING MUTANTpublic class Math {public int add(int i1, int i2) { return i1 - i2;}
}
@Testpublic void add_should_add() {new Math().add(1, 1);
}
✔Execute SAME Test
@nicolas_frankel #mutationtesting 24
TEST THE CODEpublic class Math {public int add(int i1, int i2) { return i1 + i2;}
}
@Testpublic void add_should_add() {int sum = new Math().add(1, 1);Assert.assertEquals(sum, 2);
}
✔Execute Test
@nicolas_frankel #mutationtesting 25
KILLED MUTANTpublic class Math {public int add(int i1, int i2) { return i1 - i2;}
}
@Testpublic void add_should_add() {int sum = new Math().add(1, 1);Assert.assertEquals(sum, 2);
}
✗Execute SAME Test
MUTATION TESTING IN JAVAPIT is a tool for Mutation testingAvailable as• Command-line tool•Ant target•Maven plugin
@nicolas_frankel #mutationtesting 26
MUTATORSMutators are patterns applied to source code to produce mutations
@nicolas_frankel #mutationtesting 27
@nicolas_frankel #mutationtesting 28
PIT MUTATORS SAMPLEName Example source ResultConditionals Boundary > >=Negate Conditionals == !=Remove Conditionals foo == bar trueMath + -Increments foo++ foo--Invert Negatives -foo fooInline Constant static final FOO= 42 static final FOO = 43Return Values return true return falseVoid Method Call System.out.println("foo")Non Void Method Call long t = System.currentTimeMillis() long t = 0Constructor Call Date d = new Date() Date d = null;
IMPORTANT MUTATORSConditionals Boundary• Probably a potential serious
bug smell if (foo > bar)
@nicolas_frankel #mutationtesting 29
IMPORTANT MUTATORSVoid Method Call Assert.checkNotNull()connection.close()
@nicolas_frankel #mutationtesting 30
REMEMBER It’s not because the IDE
generates code safely that it will never change• equals()• hashCode()
@nicolas_frankel #mutationtesting 31
FALSE POSITIVESMutation Testing is not 100% bulletproofMight return false positivesBe cautious!
@nicolas_frankel #mutationtesting 32
@nicolas_frankel #mutationtesting 33
ENOUGH TALK…
Time for DEMO
DRAWBACKSSlowSluggishCrawlingSulkyLethargicetc.
@nicolas_frankel #mutationtesting 38
METRICS (KIND OF)On joda-moneymvn clean test-compilemvn surefire:test• Total time: 2.181 s
mvn pit-test...• Total time: 48.634 s
@nicolas_frankel #mutationtesting 39
WHY SO SLOW?Analyze test codeFor each class under test• For each mutator
• Create mutation• For each mutation
• Run test• Analyze result• Aggregate results
@nicolas_frankel #mutationtesting 40
WORKAROUNDSThis is not acceptable in a normal test runBut there are workarounds
@nicolas_frankel #mutationtesting 41
@nicolas_frankel #mutationtesting 42
SET MUTATORS<configuration> <mutators> <mutator> CONSTRUCTOR_CALLS </mutator> <mutator> NON_VOID_METHOD_CALLS </mutator> </mutators></configuration>
@nicolas_frankel #mutationtesting 43
SET TARGET CLASSES<configuration> <targetClasses> <param>ch.frankel.pit*</param> </targetClasses></configuration>
@nicolas_frankel #mutationtesting 44
SET TARGET TESTS<configuration> <targetTests> <param>ch.frankel.pit*</param> </targetTests></configuration>
@nicolas_frankel #mutationtesting 45
DEPENDENCY DISTANCE
1 2
@nicolas_frankel #mutationtesting 46
LIMIT DEPENDENCY DISTANCE<configuration> <maxDependencyDistance> 4 </maxDependencyDistance></configuration>
@nicolas_frankel #mutationtesting 47
LIMIT NUMBER OF MUTATIONS<configuration> <maxMutationsPerClass> 10 </maxMutationsPerClass></configuration>
USE MUTATIONFILTERFrom extension points
@nicolas_frankel #mutationtesting 48
PIT EXTENSION POINTSMust be packaged in JARHave Implementation-Vendor and Implementation-Title in MANIFEST.MF that match PIT’sSet on the classpathUse Java’s Service Provider feature
@nicolas_frankel #mutationtesting 49
SERVICE PROVIDERInversion of control• Since Java 1.3!
Text file located in META-INF/servicesInterface• Name of the file
Implementation class• Content of the file
@nicolas_frankel #mutationtesting 50
@nicolas_frankel #mutationtesting 51
SAMPLE STRUCTURE
JARch.frankel.lts.Sample
@nicolas_frankel #mutationtesting 52
SERVICE PROVIDER API
@nicolas_frankel #mutationtesting 53
SERVICE PROVIDER SAMPLEServiceLoader<ISample> loaders = ServiceLoader.load(ISample.class);for (ISample sample: loaders) {
// Use the sample}
OUTPUT FORMATSOut-of-the-box•HTML• XML• CSV
@nicolas_frankel #mutationtesting 54
@nicolas_frankel #mutationtesting 55
OUTPUT FORMATS<configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats></configuration>
@nicolas_frankel #mutationtesting 56
MUTATION FILTERRemove mutations from the list of available mutations for a class
@nicolas_frankel #mutationtesting 57
@nicolas_frankel #mutationtesting 58
@nicolas_frankel #mutationtesting 59
Time for DEMO
@nicolas_frankel #mutationtesting 60
DON’T BIND TO TEST PHASE!<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <executions> <execution> <goals> <goal>mutationCoverage</goal> </goals> <phase>test</phase> </execution> </executions></plugin>
@nicolas_frankel #mutationtesting 61
USE SCMMUTATIONCOVERAGEmvn \org.pitest:pitest-maven:scmMutationCoverage \-DtimestampedReports=false
maven-scm-
plugin
must be configured!
@nicolas_frankel #mutationtesting 62
DO USE ON CONTINUOUS INTEGRATION SERVERSmvn \org.pitest:pitest-maven:mutationCoverage \-DtimestampedReports=false
IS MUTATION TESTING THE SILVER BULLET?Sorry, no!It only• Checks the relevance of your
unit tests• Points out potential bugs
@nicolas_frankel #mutationtesting 63
WHAT IT DOESN’T DOValidate the assembled application• Integration Testing
Check the performance• Performance Testing
Look out for display bugs• End-to-end testing
Etc.
@nicolas_frankel #mutationtesting 64
TESTING IS ABOUT ROIDon’t test to achieve 100% coverageTest because it saves money in the long runPrioritize:• Business-critical code• Complex code
@nicolas_frankel #mutationtesting 65
@nicolas_frankel
Q&A@nicolas_frankelhttp://blog.frankel.ch/https://leanpub.com/integrationtest/