first steps in testing analytics: does test code quality matter?
DESCRIPTION
Does investing in high quality developer test code really matter? And does it help a team to become more productive?TRANSCRIPT
![Page 1: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/1.jpg)
First Steps in Testing Analytics: Does Test Code Quality Matter?
Andy ZaidmanDelft University of Technology
Nederlandse Testdag17 november 2014, Amersfoort
![Page 2: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/2.jpg)
Joint work with Moritz Beller & Georgios Gousios(TU Delft)
Dimitrios Athanasiou, Ariadi Nugroho & Joost Visser(Software Improvement Group)
![Page 3: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/3.jpg)
Who am I?
Software engineering teacher by day
Testing evangelist
Software engineering researcher by night
![Page 4: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/4.jpg)
What would you like to know?What analytics?
• ???
![Page 5: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/5.jpg)
![Page 6: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/6.jpg)
WHATEVER HAPPENS…
![Page 7: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/7.jpg)
@azaidman
![Page 8: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/8.jpg)
TODAY
2 test analytics stories
![Page 9: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/9.jpg)
STORY 1
Test Code Quality Matters
![Page 10: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/10.jpg)
Andy teaching
First year programming course @ TU Delft
In week 3 everyone is unit testing
“I don’t have time for testing”
“I don’t like testing, my code is good anyway”
“How much should I test?”
“How good should my test code be?”
![Page 11: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/11.jpg)
@Test
public void testConstructor() {
assertTrue(true)
}
Solution 1
![Page 12: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/12.jpg)
![Page 13: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/13.jpg)
Solution 2
@Test
public void testConstructor() {
House h = new House(“Street”, “9”);
assertEquals(“Street 9”, h.toString());
}
![Page 14: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/14.jpg)
Solution 3
@Test
public void testConstructor() {
House h = new House(“Street”, “9”);
assertEquals(“Street”, h.getStreet());
assertEquals(9, h.getNumber());
}
![Page 15: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/15.jpg)
Solution 4
@Testpublic void testConstructorStreet() {
House h = new House(“Street”, “9”);assertEquals(“Street”, h.getStreet());
}
@Testpublic void testConstructorNumber() {
House h = new House(“Street”, “9”);assertEquals(9, h.getNumber());
}
![Page 16: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/16.jpg)
My first reaction?
![Page 17: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/17.jpg)
SHOOT FIRST
![Page 18: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/18.jpg)
BRINGS US
NOWHERE
![Page 19: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/19.jpg)
Analytics?
![Page 20: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/20.jpg)
Testing Analytics!
![Page 21: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/21.jpg)
![Page 22: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/22.jpg)
![Page 23: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/23.jpg)
![Page 24: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/24.jpg)
![Page 25: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/25.jpg)
Thou shalt not have less than 70% coverage
![Page 26: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/26.jpg)
![Page 27: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/27.jpg)
What makes a good
unit test?
What makes a good
integration test?
![Page 28: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/28.jpg)
What makes a good
developer test?
![Page 29: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/29.jpg)
What makes a good
developer test?
Complete
Effective
Maintainable
![Page 30: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/30.jpg)
Complete
• How much of the code is covered by the tests?
• How many of the decision points in the code are tested?
Code coverage
#assertions/cyclomatic complexity
![Page 31: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/31.jpg)
Effective
• How able is the test code to detect defects?
• How able is the test code to locate the cause?
Assertion density #assertions / LOCtest
Directness unit versus integration testing
![Page 32: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/32.jpg)
Maintainable
• How maintainable is the system’s test code?
Software Improvement Group quality model
![Page 33: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/33.jpg)
Test Code Quality Model*
Complete
+ Effective
+ Maintainable
* Calibrated using 86 systems
![Page 34: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/34.jpg)
SO
WHAT?
![Page 35: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/35.jpg)
![Page 36: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/36.jpg)
Bettertests
Easier development
?
![Page 37: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/37.jpg)
High quality tests do not lead to quicker/easier bug fixing
Why?
The low-hanging fruit is already solved if you have good tests… the difficult bugs remain
Observation 1*
* While studying 75 years of combined development of 18 software systems
![Page 38: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/38.jpg)
High quality tests do lead to higher productivity
• More issues fixed per month per developer
• More issues fixed per month per KLOC
Why?
Implementing new features does seem to benefit from having a safety net for existing
functionality.
Observation 2
![Page 39: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/39.jpg)
High quality tests do make your team more productive!
![Page 40: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/40.jpg)
Action(able)
![Page 41: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/41.jpg)
The single most important thing for test
code quality
#assertionsMcCabe cyclomatic complexity
≥ 1
![Page 42: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/42.jpg)
…
![Page 43: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/43.jpg)
STORY 2
Testing takes 50% of your time
![Page 44: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/44.jpg)
Says this guy
• Study from 1975
• Questionable research approach
![Page 45: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/45.jpg)
Differences?
• Waterfall
• Developer testing
• Programming language
• Tools (e.g. IDE)
![Page 46: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/46.jpg)
New research
![Page 47: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/47.jpg)
![Page 48: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/48.jpg)
+ +
![Page 49: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/49.jpg)
Recording
1. Time spent on developing production code
2. Time spent on developing test code
3. How frequently are Junit tests executed?
4. How frequently do these test runs fail?
![Page 50: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/50.jpg)
124 developers participating
![Page 51: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/51.jpg)
![Page 52: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/52.jpg)
You might win
![Page 53: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/53.jpg)
http://testroots.org/testroots_watchdog.html
![Page 54: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/54.jpg)
…
![Page 55: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/55.jpg)
My ultimate dream?
Continuous
background
testing
![Page 56: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/56.jpg)
![Page 57: First steps in testing analytics: Does test code quality matter?](https://reader030.vdocuments.us/reader030/viewer/2022020307/559ef7cc1a28abd1768b467c/html5/thumbnails/57.jpg)
Take home
• Test code quality matters for the productivity of your team
• #assertions/cyclomatic complexity ≥ 1
• Help us out with our research!
http://azaidman.wordpress.com
httpp://www.st.ewi.tudelft.nl/~zaidman
@azaidman