mutation testing - aurelijus banelis · why what how why we need mutation testing what it is and...
TRANSCRIPT
![Page 3: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/3.jpg)
Mutation testingwith PHP
![Page 4: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/4.jpg)
Inspired by Vaidas’ presentation
![Page 5: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/5.jpg)
WHY
WHAT
HOW
Why we need mutation testing
What it is and how it works
How it can be used in practice
![Page 6: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/6.jpg)
WHY
WHAT
HOW
Why we need mutation testing
What it is and how it works
How it can be used in practice
![Page 7: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/7.jpg)
It is my codeIt should be good
![Page 8: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/8.jpg)
Small mistakesBut huge impact
Illiustration: https://www.youtube.com/watch?v=_BgJEXQkjNQ
![Page 9: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/9.jpg)
Let’s test everything
![Page 10: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/10.jpg)
Who tests the tester?
![Page 11: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/11.jpg)
Automation and metrics
Iliustration of https://travis-ci.org/aurelijusb/symfony/builds/188337032
![Page 12: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/12.jpg)
WHY
WHAT
HOW
To be confident about our code and tests
What it is and how it works
How it can be used in practice
![Page 13: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/13.jpg)
WHY
WHAT
HOW
Why we need mutation testing
What it is and how it works
How it can be used in practice
![Page 14: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/14.jpg)
HambugOne of PHP mutation testing tools
![Page 15: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/15.jpg)
Getting started{ "source": { "directories": [ "." ], "excludes": [ "Tests", "vendor" ] }, "timeout": 3, "logs": { "text": "../../../../build/bumbug/humbuglog.txt", "json": "../../../../build/bumbug/humbuglog.json" }}
"require-dev": { "humbug/humbug": "~1.0@dev"},
![Page 16: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/16.jpg)
Let’s test on symfony
https://github.com/aurelijusb/symfony/tree/mutation-testing
![Page 17: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/17.jpg)
Mutation statistics
https://github.com/padraic/humbug https://travis-ci.org/aurelijusb/symfony/builds/188651125
● Killed Mutation (.): A mutation that caused unit tests to fail which is a positive outcome.
● Escaped Mutation (M): A mutation where the unit tests still passed which is not what we want! Our unit tests should detect any behaviour changes.
● Uncovered Mutation (S): A mutation which occurs on a line not covered by any unit test.
● Fatal Error (E): A mutation created a fatal error. ● Timeout (T): This is where unit tests exceed the
allowed timeout
![Page 18: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/18.jpg)
Mutation Details
![Page 19: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/19.jpg)
Under the hood
exec /usr/bin/php7.0 /vendor/.../phpunit --configuration=/tmp/humbug/phpunit.humbug.xml --stop-on-failure --tap
<logging><log type="coverage-php"
target="/tmp/humbug/coverage.humbug.php"/><log type="coverage-text"
target="/tmp/humbug/coverage.humbug.txt"/><log type="junit" target="/tmp/humbug/junit.humbug.xml"/>
</logging>
![Page 20: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/20.jpg)
Drawbacks
- humbug/humbug 1.0.x-dev requires phpunit/phpunit ^4.5|^5.0 - phpunit/phpunit 5.7.5 requires php ^5.6 || ^7.0 -> your PHP version (5.5.9) does not satisfy that requirement.
![Page 21: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/21.jpg)
WHY
WHAT
HOW
Why we need mutation testing
Testing framework, change code, run PHPUnit
How it can be used in practice
![Page 22: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/22.jpg)
WHY
WHAT
HOW
Why we need mutation testing
What it is and how it works
How it can be used in practice
![Page 23: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/23.jpg)
Metrics for better release planning
![Page 24: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/24.jpg)
Edge cases for you
![Page 25: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/25.jpg)
Quality for regression testing
![Page 26: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/26.jpg)
What this line should do?
![Page 27: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/27.jpg)
./phpunit --coverage-xml=build/xml
![Page 28: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/28.jpg)
100% mutation proof?
https://github.com/steos/php-quickcheck
Generate random input and test code against predicate
![Page 29: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/29.jpg)
WHY
WHAT
HOW
Why we need mutation testing
What it is and how it works
Planning, test cases, documentation by tests
![Page 30: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/30.jpg)
References ● http://www.slideshare.net/AgileLietuva/vaidas-pilkauskas-and-tadas-erbinskas-can-you-trust-your-tests ● http://mnapoli.fr/code-coverage-reports-with-github-travis-and/ ● https://coveralls.io ● https://phpunit.de/manual/current/en/code-coverage-analysis.html#code-coverage-analysis.edge-cases ● https://akrabat.com/global-installation-of-php-tools-with-composer/ ● https://travis-ci.org/aurelijusb/symfony/branches ● https://github.com/steos/php-quickcheck
![Page 31: Mutation testing - Aurelijus Banelis · WHY WHAT HOW Why we need mutation testing What it is and how it works How it can be used in practice](https://reader036.vdocuments.us/reader036/viewer/2022070919/5fb8cce7ad1b5c61380235ed/html5/thumbnails/31.jpg)
Mutation testingwith PHP
Questions?
Aurelijus Banelis VilniusPHP 0x322017-01-05