gtac 2014: what lurks in test suites?
DESCRIPTION
We all want "better" test suites. But what makes for a good test suite? Certainly, test suites ought to aim for good coverage, at least at the statement coverage level. To be useful, test suites should run quickly enough to provide timely feedback. This talk will investigate a number of other dimensions on which to evaluate test suites. The talk claims that better test suites are more maintainable, more usable (for instance, because they run faster, or use fewer resources), and have fewer unjustified failures. In this talk, I'll present and synthesize facts about 10 open-source test suites (from 8,000 to 246,000 lines of code) and evaluate how they are doing.TRANSCRIPT
![Page 1: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/1.jpg)
Beyond Coverage:What Lurks in Test Suites?
Patrick Lam, @uWaterlooSE(and Felix Fang)
University of Waterloo
![Page 2: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/2.jpg)
Test Suites: Myths vs Realities.
![Page 3: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/3.jpg)
Subjects: Open-Source Test Suites
![Page 4: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/4.jpg)
Basic Test Suite Properties
Benchmark sizes: 30 kLOC (google-visualization) to 495 kLOC (weka)
% of system represented by tests: 5.3% (weka) to 50.4% (joda-time)
![Page 5: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/5.jpg)
Static Test Suite Properties
![Page 6: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/6.jpg)
Test suite versus benchmark size
m = 0.3002
m = 0.03514
![Page 7: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/7.jpg)
# test cases versus # test methods
![Page 8: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/8.jpg)
apache-commons-collection tests
Consider map.TestFlat3Map:contains 14 test methodsyet, 156 test cases
superclass tests: 42 tests+ 4 Apache Commons Collections “bulk tests”
![Page 9: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/9.jpg)
Run-time Test Suite Properties
![Page 10: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/10.jpg)
Test suites run quicklyjoda-time 4.9s
jdom 5.0s
google-vis 5.1s
jgrapht 16.9s
weka 28.9s
apache-cc 34.0s
poi 36.5s
jmeter 53.0s
jfreechart 241.0s
![Page 11: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/11.jpg)
Failing tests
76/384
0n/a 0
1
03/1109
00
0
![Page 12: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/12.jpg)
Continuous Integration: Daily Builds
![Page 13: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/13.jpg)
Continuous Integration: Daily Tests
(via SonarQube, Travis CI, Surefire)
![Page 14: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/14.jpg)
Myth #1:
Coverage is a key property of test suites.
![Page 15: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/15.jpg)
Coverage is central in textbooks
Ammann and Offutt, Introduction to Software Testing
![Page 16: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/16.jpg)
Coverage metrics from EclEmma
![Page 17: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/17.jpg)
Coverage metrics
![Page 18: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/18.jpg)
Reality #1
Coverage sometimes important, but tools only give limited data.
![Page 19: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/19.jpg)
Guideline #1
Consider metrics beyond reported coverage results:
- weka uses peer review for QA- not measured by tools:
input space coverage
![Page 20: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/20.jpg)
Myth #2
Tests are simple.- test complexity- test dependencies
![Page 21: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/21.jpg)
Static Code Complexity
![Page 22: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/22.jpg)
Test methods with at least 5 asserts
e.g. from Joda-Time:
public void testEquality() {
assertSame(getInstance(TOKYO), getInstance(TOKYO));
assertSame(getInstance(LONDON), getInstance(LONDON));
assertSame(getInstance(PARIS), getInstance(PARIS));
assertSame(getInstanceUTC(), getInstanceUTC());
assertSame(getInstance(), getInstance(LONDON));
}
![Page 23: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/23.jpg)
% Test methods with ≥ 5 asserts
![Page 24: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/24.jpg)
Test Methods with Branchesif (isAllowNullKey() == false) { try {
assertEquals(null, o.nextKey(null));
} catch (NullPointerException ex) {}
} else { assertEquals(null, o.nextKey(null));
}
// from apache-cc
![Page 25: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/25.jpg)
Test Methods with Loops counter = 0;
while (this.complexPerm.hasNext()) { this.complexPerm.getNext();
counter++;
} assertEquals(maxPermNum, counter);
// from jgrapht
![Page 26: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/26.jpg)
% Test Methods with Control-Flow
![Page 27: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/27.jpg)
Tests Which Use the Filesystem
![Page 28: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/28.jpg)
Filesystem Usage Details
new File(tempDir, "tzdata");
verifies vs canonical forms of serialized collections on disk
![Page 29: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/29.jpg)
More Filesystem Usage Details
resources, serialization
creates charts, tests their existencesome comparisons vs test data
![Page 30: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/30.jpg)
Tests Which Use the Network
*
![Page 31: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/31.jpg)
Network Usage Details
connects to http://sc.openoffice.org
tests HTTP mirror server at localhost
![Page 32: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/32.jpg)
flip side: Mocks and Stubs
True mocks only in Google Visualization.
![Page 33: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/33.jpg)
flip side: Mocks and Stubs
True mocks only in Google Visualization.
Found stubs/fakes in 4 other suites.
![Page 34: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/34.jpg)
Reality #2
Test cases are mostly simple.few asserts, little branchingsome filesystem/net usage
![Page 35: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/35.jpg)
Consequence #2
Many tests don’t need high expertise to write,
but some do!
![Page 36: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/36.jpg)
Myth #3
Test cases are written by hand.
![Page 37: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/37.jpg)
Types of reuse (standard Java)
1. test class setUp()/tearDown()
2. inheritance: e.g. in apache-cc,TestFastHashMap extends AbstractTestMap
3. composition: e.g. in jfreechart, helper class RendererChangeDetector
![Page 38: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/38.jpg)
JUnit setup/tearDown usage
![Page 39: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/39.jpg)
Inheritance is heavily used
(> 50% test classes inherit functionality)
![Page 40: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/40.jpg)
Test Classes with Custom Superclasses
![Page 41: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/41.jpg)
Helper Classes Example
from poi:
/** Test utility class to get Records * out of HSSF objects. */public final class RecordInspector {
public static Record[] getRecords(...) {}}
![Page 42: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/42.jpg)
Helper Class Countweka 1
google-vis 3
jdom 6
joda-time 7
jfreechart 7
jmeter 12
jgrapht 15
apache-cc 22
hsqldb 31
poi 54
![Page 43: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/43.jpg)
public void testNominalFiltering() {
m_Filter = getFilter(Attribute.NOMINAL);
Instances r = useFilter();
for (int i = 0; i < r.numAttributes(); i++)
assertTrue(r.attribute(i).type() != Attribute.NOMINAL);}
public void testStringFiltering() {
m_Filter = getFilter(Attribute.STRING);
Instances r = useFilter();
for (int i = 0; i < r.numAttributes(); i++)
assertTrue(r.attribute(i).type() != Attribute.STRING);}
Test Clone Example
![Page 44: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/44.jpg)
Assertion Fingerprints
detect clones by identifyingsimilar tests
![Page 45: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/45.jpg)
Incidence of cloning
![Page 46: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/46.jpg)
How to Refactor?
● setUp/tearDown/subclassing● JUnit 4:
Parametrized Unit Tests● Test Theories
![Page 47: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/47.jpg)
apache-cc: Bulk testspublic BulkTest bulkTestKeySet() { return new TestSet(makeFullMap().keySet());}
● runs all tests in the TestSet class with the object returned from makeFullMap().keySet()
![Page 48: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/48.jpg)
jdom: Generated Test Case Stubs
class ClassGenerator makes e.g.: class TestDocument {void test_TCC__List();void test_TCM__int_hashCode();
}
Developer still needs to populate tests.
![Page 49: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/49.jpg)
Automated Testing Technology
In our test suites, the principal automation technology was cut-and-paste.
![Page 50: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/50.jpg)
Reality #3
Automated test generationis uncommon in our test suites.
![Page 51: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/51.jpg)
Guideline
Maximize reuse:
whatever works for you!
setUp/tearDown,inheritance,parametrized tests,
![Page 52: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/52.jpg)
Suggestion
Use automated test generation tools!Some examples:
● Korat (structurally complex tests)● Randoop (random testing)● CERT Basic Fuzzing Framework
http://mit.bme.hu/~micskeiz/pages/code_based_test_generation.html
![Page 53: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/53.jpg)
Summary
Myths:1. Coverage is a key property
of test suites. ≈2. Tests are simple. ✓3. Tests are written by hand. ✓
![Page 54: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/54.jpg)
![Page 55: GTAC 2014: What lurks in test suites?](https://reader033.vdocuments.us/reader033/viewer/2022052316/559c19891a28ab22598b469e/html5/thumbnails/55.jpg)
Datahttps://docs.google.com/spreadsheets/d/1xAsdk35tJAOM4WGbGloliS4ovDJ8_MDn6_Gzk0DXEZQ