MUTATIONTESTINGIN PHP WITH HUMBUG

@MARKREDEMANSTUDENT APPLIED MATHEMATICSFREELANCE (WEB) DEVELOPER

Introduction to mutation testingHumbug, a mutation testing framework for PHP.Analyzing code coverage of open source projectsImproving your workflow with mutation testing

What's this talk about?

A tool to analyze stability of a piece of codeSimilar to code coverage, but betterCan find missing tests

WHAT IS MUTATION TESTING?

A quick exampleclass Customer private $orders = 0;

public function __construct($orders) $this­>orders = $orders;

public function isGoldCustomer() return $this­>orders > 10;

function testIsGoldCustomer() $this­>assertFalse((new Customer(0))­>isGoldCustomer()); // 100% line coverage :D $this­>assertTrue((new Customer(11))­>isGoldCustomer());

public function isGoldCustomer() return $this­>orders >= 10;

class Customer private $orders = 0;

public function __construct($orders) $this­>orders = $orders;

public function isGoldCustomer() return $this­>orders > 10;

function testIsGoldCustomer() $this­>assertFalse((new Customer(0))­>isGoldCustomer()); // 100% line coverage :D $this­>assertFalse((new Customer(10))­>isGoldCustomer()); $this­>assertTrue((new Customer(11))­>isGoldCustomer());

Generated mutants aresimilar to real faults

- Andrew, Briand, Labiche, ICSE 2005

Some Definitions

a piece of code that has been changed (mutated) by a mutator

killed if at least 1 test failsescaped if at all test passequivalent if there does not exist a test case which can

distinguish the mutant from the original codeuncovered mutant is not covered by a testfatal mutant produces a fatal errortimout unit tests exceed allowed timeout

Mutation

Killed Mutantclass Customer private $orders = 0;

public function __construct($orders) $this­>orders = $orders;

public function isGoldCustomer() return $this­>orders < 10; // mutated ">" to "<"

function testIsGoldCustomer() $this­>assertFalse((new Customer(0))­>isGoldCustomer()); $this­>assertTrue((new Customer(11))­>isGoldCustomer());

Escaped Mutantclass Customer private $orders = 0;

public function __construct($orders) $this­>orders = $orders;

public function isGoldCustomer() return $this­>orders >= 10; // mutated ">" to ">="

function testIsGoldCustomer() $this­>assertFalse((new Customer(0))­>isGoldCustomer()); $this­>assertTrue((new Customer(11))­>isGoldCustomer());

Equivalent Mutantfunction sum_is_zero(array $values) $sum = 0;

foreach ($values as $value) $sum += $value;

return $sum === 0;

function sum_is_zero(array $values) $sum = 0;

foreach ($values as $value) $sum ­= $value; // mutated "+" to "­"

return $sum === 0;

Increments MutatorDecrements MutatorInvert Negatives MutatorReturn Values Mutator

Math MutatorNegate MutatorConditionals BoundaryMutatorRemove ConditionalsValues Mutator

MUTATOROperator that changes (mutates) a piece of code

Original Mutated+ ­­ +* // *% *

MATH MUTATORS

Original Mutated== !=!= ==<= >>= << >=> <=

CONDITIONAL MUTATORS

::

Mutator descriptionNegate Conditionals replace condition by ! (condition)

Remove Conditionals substitute conditional with true or falseIncrements crement numeric values by 1Decrements crement numeric values by 1

Invert Negatives multiply numeric values by ­1Return Values remove return statements

MetricsMutation Score Indicator (MSI):

percentage of mutants covered & killed by testsMutation Code Coverage:

percentage of mutants covered by testsCovered Code MSI:

percentage of killed mutants that were coverd by tests$vanquishedTotal = $killedCount + $timeoutCount + $errorCount;$measurableTotal = $totalCount ­ $uncoveredCount; // = $vanquishedTotal + $escapedCount

$msi = round(100 * ($vanquishedTotal / $totalCount));$coveredRate = round(100 * ($measurableTotal / $totalCount));$cc_msi = round(100 * ($vanquishedTotal / $measurableTotal));

Metrics ExampleTests: 361Line Coverage: 64.86%Mutation Score Indicator (MSI): 47%Mutation Code Coverage: 67%Covered Code MSI 70%

653 Mutants were generated284 mutants were killed218 mutants were not covered by tests131 covered mutants were not detected17 fatal errors were encountered3 time outs were encountered

47% of all mutations were detected versus 65% line coverage.

HUMBUGA Mutation Testing framework for PHPMeasures the real effectiveness of your test suites and assist in their improvement.

It eats Code Coverage for breakfast

Git:git clone https://github.com/padraic/humbug.gitcd humbug/path/to/composer.phar installbin/humbug

Phar:wget https://padraic.github.io/humbug/downloads/humbug.pharwget https://padraic.github.io/humbug/downloads/humbug.phar.pubkeychmod +x humbug.phar

Composer:composer global require 'humbug/humbug=~1.0@dev'# And add ~/.composer/vendor/bin to your path to make it easily executableexport PATH=~/.composer/vendor/bin:$PATH

Installation

Configuration$> git clone git@github.com:yourname/yourproject.git# Check if all tests are green$> phpunit$> humbug configure

_ _ _| || |_ _ _ __ | |__ _ _ __ _| __ | || | ' \| '_ \ || / _ ||_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/Humbug version 1.0­dev

Humbug configuration tool.It will guide you through Humbug configuration in few seconds.

When choosing directories, you may enter each directory and press return.To exit directory selection, please leave the next answer blank and press return.

What source directories do you want to include? : srcWhat source directories do you want to include? :Any directories to exclude from within your source directories? :Single test suite timeout in seconds [10] :Where do you want to store the text log? [humbuglog.txt] :Where do you want to store the json log (if you need it)? :Generate "humbug.json.dist"? [Y]:Configuration file "humbug.json.dist" was created.

Running Humbug$> humbug

_ _ _| || |_ _ _ __ | |__ _ _ __ _| __ | || | ' \| '_ \ || / _ ||_||_|\_,_|_|_|_|_.__/\_,_\__, | |___/Humbug version 1.0­dev

Humbug running test suite to generate logs and code coverage data...

97 [==========================================================] 2 secs

Humbug has completed the initial test run successfully.Tests: 97 Line Coverage: 98.27%

Humbug is analysing source files...

Mutation Testing is commencing on 19 files...(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)

E......M.M.......MMTT.....MMS..........................M.M.. | 60 ( 9/19)M..................M....E.......MMMMS

97 mutations were generated: 77 mutants were killed 2 mutants were not covered by tests 14 covered mutants were not detected 2 fatal errors were encountered 2 time outs were encountered

Metrics: Mutation Score Indicator (MSI): 84% Mutation Code Coverage: 98% Covered Code MSI: 85%

Remember that some mutants will inevitably be harmless (i.e. false positives).

Time: 31.39 seconds Memory: 8.75MBHumbug results are being logged as TEXT to: humbuglog.txt

Analyzing Humbughumbug.log.txt

2) \Humbug\Mutator\ConditionalBoundary\GreaterThanDiff on \Carbon\CarbonInterval::__construct() in /Carbon/src/Carbon/CarbonInterval.php:­­­ Original+++ New@@ @@ $specDays += $weeks > 0 ? $weeks * Carbon::DAYS_PER_WEEK : 0;­ $specDays += $days > 0 ? $days : 0;+ $specDays += $days >= 0 ? $days : 0;

$spec .= ($specDays > 0) ? $specDays.static::PERIOD_DAYS : '';

if ($hours > 0 || $minutes > 0 || $seconds > 0) $spec .= static::PERIOD_TIME_PREFIX; $spec .= $hours > 0 ? $hours.static::PERIOD_HOURS : '';

humbug.log.json

"summary": "total": 26, "kills": 23, "escapes": 1, "errors": 0, "timeouts": 0, "notests": 2, "covered_score": 96, "combined_score": 88, "mutation_coverage": 92,"escaped": [ "file": "src\/BroadwayDemo\/Basket\/Basket.php", "mutator": "\\Humbug\\Mutator\\ConditionalBoundary\\GreaterThan", "class": "\\BroadwayDemo\\Basket\\Basket", "method": "productIsInBasket", "line": 101, "diff": "­­­ Original\n+++ New\n@@ @@\n \n­ return isset($this­>productCountById[ "tests": [ "BroadwayDemo\\Basket\\AddProductToBasketTest::it_adds_a_product_to_a_basket", "BroadwayDemo\\Basket\\AddProductToBasketTest::multiple_products_can_be_added_to_a_basket" "BroadwayDemo\\Basket\\AddProductToBasketTest::a_product_can_be_added_to_a_basket_multiple_times" "BroadwayDemo\\Basket\\CheckoutTest::it_checks_out_a_basket", "BroadwayDemo\\Basket\\CheckoutTest::it_cannot_checkout_a_basket_that_has_been_emptied" "BroadwayDemo\\Basket\\CheckoutTest::nothing_happens_when_checking_out_a_basket_for_a_second_time" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_removes_a_product_that_was_added" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_does_nothing_when_removing_a_product_that_is_not_in_a_basket" "BroadwayDemo\\Basket\\RemoveProductFromBasketTest::it_only_removes_one_instance_of_a_product" ], "stderr": "", "stdout": "TAP version 13"],

OptionsTimeout:

humbug ­­timeout=10Restricting files:

humbug ­­file=PrimeFactor.phphumbug ­­file=*Driver.php

Incremental analysis:humbug ­­incrementalCan off significant performance boosts by caching previous results in/home/padraic/.humbug..

MutatorsBinary Arithmetic

Boolean SubstitutionConditional BoundariesNegated Conditionals

IncrementsReturn Values

Literal NumbersIf Statements

HUMBUG TEST RESULTS

Package LC MSI MCC CCM Executiontime

carbon/carbon 61% 95% 100% 95% 2.25msymfony/event-dispatcher

85% 54% 69% 78% 20s

hylianshield/validator 100% 75% 89% 85% 1.35mmathiasverraes/money 96% 92% 100% 92% 16.9sthephpleague/fractal 98% 84% 98% 85% 31sbroadway/broadway-demo

96% 88% 92% 86% 5s

broadway/broadway 57% 49% 56% 87% 46s

Line Coverage (LC), Mutation Score Indicator (MSI), Mutation Code Coverage (MCC), Covered Code MSI (CCM)Timout option set to 2 seconds.

TDD workflow with mutation testing

Read the book

Other toolsRuby: MutantJava: PitestC#: NinjaturtlesPython: MutPyMatlab: MatMute

Concluding remarksMutation testing will improve the quality of your testsIs becoming more mainstream over the last yearsWrite small (fast) tests

I have not failed. I've justfound 10,000 ways thatwon't work

- Thomas Edison

Find these slides atmutation.markredeman.nl