state of automated testing - libreoffice · unit tests c++ classes components via uno api...
TRANSCRIPT
1LibreOffice Aarhus 2015: State of Automated Testing
State of Automated Testing
Michael Stahl, Red Hat, Inc.2015-09-23
2LibreOffice Aarhus 2015: State of Automated Testing
Overview
1. Introduction
2. Regression Tests Developers Run Every Day
3. Even More Tests
3LibreOffice Aarhus 2015: State of Automated Testing
Introduction
Every release: ~10.000 commits, ~1 million LOC changedRegressions?Automated testingGoal: developers find bugs before they push them to master
“Your developers—or worst case, test organization—produce tests.”
— Michael Klepikov
4LibreOffice Aarhus 2015: State of Automated Testing
Requirements
Goal: developers find bugs before they push them to masterRequirements
Standard unit test librariesMust be run as part of standard build – make check
Reliable – avoid false positivesFastGood defect localizationDebuggable
5LibreOffice Aarhus 2015: State of Automated Testing
CppUnit Tests
C++Standard CppUnit library (Thanks to Markus Mohrhard)In-process (except smoketest)
Kinds:Unit tests
C++ classesComponents via UNO API
Integration testsFilter crash-tests (CVE-tests)Filter testsEtc.
System tests: smoketest
6LibreOffice Aarhus 2015: State of Automated Testing
Filter Tests
Filter test:Import fileCheck some properties were imported correctlyExport file, import againCheck properties were round-tripped correctly
--with-export-validation validates every exported file
ODF ValidatorOOXML Validator
7LibreOffice Aarhus 2015: State of Automated Testing
CppUnit Test Growth
3.3 3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
2000
4000
6000
8000
10000
12000
14000
16000
18000
0
50
100
150
200
250
CppUnit Tests
CPPUNIT_TESTCPPUNIT_ASSERTCppunitTest*.mk
LO version
8LibreOffice Aarhus 2015: State of Automated Testing
JUnit Tests
JavaStandard JUnit libraryKinds:
Unit tests – in-processURE Java binding
“Complex” tests – out-of-process – Remote UNOUnit tests of componentsIntegration tests
9LibreOffice Aarhus 2015: State of Automated Testing
JUnit Test Growth
3.3 3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
500
1000
1500
2000
2500
3000
3500
4000
0
5
10
15
20
25
30
35
JUnit Tests
JUnit @TestassertJunitTest*.mk
LO version
10LibreOffice Aarhus 2015: State of Automated Testing
qadevOOo “unoapi” Tests
JavaCustom test frameworkOut-of-process – Remote UNOObscure test code“black-box”, overly generic testsKinds
Unit tests – components via UNO API
11LibreOffice Aarhus 2015: State of Automated Testing
qadevOOo Test Non-Growth
3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
200
400
600
800
1000
1200
1400
1600
1800
0
5
10
15
20
25
30
35
qadevOOo UnoApi Tests
unoapi componentsunoapi interfacesunoapi*.mk
LO version
12LibreOffice Aarhus 2015: State of Automated Testing
Python Tests
PythonStandard unittest libraryIn-processThanks to David Ostrovsky Kinds:
Unit tests – PyUNO bindingUnit tests – components via UNO API
13LibreOffice Aarhus 2015: State of Automated Testing
Python Test Growth
3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
50
100
150
200
250
300
350
0
1
2
3
4
5
6
7
8
9
Python unittest Tests
Python test*assertPythonTest*.mk
LO version
14LibreOffice Aarhus 2015: State of Automated Testing
Tests Run by make check
3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
500
1000
1500
2000
2500
3000
3500
4000
"make check"
CPPUNIT_TESTJUnit @Testunoapi interfacesPython test*
LO version
15LibreOffice Aarhus 2015: State of Automated Testing
Requirements Checklist
Std. libs make check
Reliable Fast Defect localization
Debug-able
AutoTest / testtool ✘ ✘ ✘ ✘ ✘ ?
CppUnit„unit-test“ ✔ ✔ ✔ ✔ ✔ ✔
CppUnit „filter-test” ✔ ✔ (-) ✔ ✘ ✔
JUnit „complex“ ✔ ✔ (-) ✔ ✔ ✘
qadevOOo „unoapi“ ✘ ✔ (-) / ✘ (-) / ✔ ✘ ✘
Pythonunittest ✔ ✔ ✔ / ? ✔ ✔ (-)
16LibreOffice Aarhus 2015: State of Automated Testing
Better Testing With Assertions
Use assert() liberally in the product code to detect invalid statesAssertion failure → abort and test failureBut not necessarily good defect localization
17LibreOffice Aarhus 2015: State of Automated Testing
assert() Growth
3.4 3.5 3.6 4.0 4.1 4.2 4.3 4.4 5.0 (5.1)0
1000
2000
3000
4000
5000
6000
assert()
assert
LO version
18LibreOffice Aarhus 2015: State of Automated Testing
Code Coverage
GCC -fprofile-arcs -ftest-coverage
LCOVhttp://lcov.libreoffice.org/Updated dailyThanks to Maarten Hoes
19LibreOffice Aarhus 2015: State of Automated Testing
LCOV Code Coverage
01.04.2014 01.09.2014 01.02.2015 01.07.20150
500000
1000000
1500000
2000000
2500000
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
LCOV Code Coverage
Lines HitLines TotalLines Coverage
20LibreOffice Aarhus 2015: State of Automated Testing
More Tests
“Crash Testing”Import and export 80k documentsSee talk by Caolán McNamara
Performance testCallgrind profilingmake perfcheck
http://perf.libreoffice.org/Additional out-of-tree tests in test-files.git repoThanks to Matúš Kukan, Laurent Godard, Norbert Thiebaud
21LibreOffice Aarhus 2015: State of Automated Testing
All text and image content in this document is licensed under the Creative Commons Attribution-Share Alike 3.0 License (unless otherwise specified). "LibreOffice" and "The Document Foundation" are registered trademarks. Their respective logos and icons are subject to international copyright laws. The use of these therefore is subject to the trademark policy.
Now … go forth and write more tests!
Thanks for listening.