![Page 1: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/1.jpg)
Test Coverage with SOTAMichael Hildebrandt
HUMBOLDT UNIVERSITY BERLINDEPARTMENT OF COMPUTER SCIENCECHAIR OF SOFTWARE ENGINEERING
![Page 2: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/2.jpg)
DAAD Workshop Neum, 2009 2
motivation◦ testing is expensive
purposes of SOTA◦static program analysis◦structure oriented program test
approach◦source code parsing◦ instrumenting
workflow◦source code preparation◦ test execution◦evaluation
agenda
![Page 3: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/3.jpg)
DAAD Workshop Neum, 2009 3
given: function overlaps◦parameter:
(time) interval A (startA, endA) (time) interval B (startB, endB)
◦ result: true if A and B overlap, otherwise false
How many test cases exist?
motivationSoftware Engineering II, Prof. Dr. Holger Schlingloff
![Page 4: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/4.jpg)
DAAD Workshop Neum, 2009 4
1. test cases for overlapping intervals2. test cases for non-overlapping intervals3. test cases for A is before B and vice versa4. test cases for A B and B A5. test cases for the same starting/ending point
of A and B6. test case for A = B
motivation (cont.)
![Page 5: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/5.jpg)
DAAD Workshop Neum, 2009 5
7. test cases for intervals consisting of one point
8. test cases for identical intervals consisting of point
9. test cases for intervals containing integer / real numbers
10. test cases for invalid intervals11. test cases for intervals with values at the
borders of number range (maxint, maxreal)
motivation (cont.)
![Page 6: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/6.jpg)
DAAD Workshop Neum, 2009 6
about 20 test cases for this simple example imagine a project with 1.5 MLOC
conclusion:testing every possible input is expensive or even impossible
problem:Which subset of the possible test cases has the highest probability to detect the most possible failures?
possible solution:structure oriented tests (using SOTA)
motivation (cont.)
![Page 7: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/7.jpg)
DAAD Workshop Neum, 2009 7
Structure Oriented Test and Analysis
static program analysis◦metrics ◦control flow graph (CFG)
structure oriented tests◦evaluation of test cases by code coverage◦development of additional test cases
SOTA
author: Ronny Treyße(diploma thesis)
![Page 8: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/8.jpg)
DAAD Workshop Neum, 2009 8
calculated by parsing the source code provides several metrics◦lines of code◦number of statements, branches, atoms, conditions, …◦cyclomatic complexity◦essential complexity
static program analysis: metrics
![Page 9: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/9.jpg)
DAAD Workshop Neum, 2009 9
static program analysis: CFG
public static int fac(int n) { int result = 1;
if (n == 0 | n == 1) {
return result;
} else {
for (int i = n; i >= 2; i--) { result *= i; } return result; }}
![Page 10: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/10.jpg)
DAAD Workshop Neum, 2009 10
calculated by instrumenting and test execution
structure oriented tests
FEECC0
C1
function entries and exits coveragestatement coveragebranch coverage
C2
MMCCMCDCC3
simple condition coverageminimal multiple condition coveragemodified condition decision coveragemultiple condition coverage
MBIBI
modified boundary interior coverageboundary interior coverage
![Page 11: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/11.jpg)
DAAD Workshop Neum, 2009 12
instrumentation◦enrichment of the original source code by adding logging statements breaking up conditions, transforming loops, …
◦source code grows bigger (up to ten times)
logging information during test execution◦values of condition, atoms◦used function entries / exits and branches of control
structures (reconstruction of the control flow)
approach
![Page 12: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/12.jpg)
DAAD Workshop Neum, 2009 14
level 0◦ no instrumentation◦ to mark structures for no instrumentation
level 1◦ all entries and exists of a method, branching control structures logged◦ all coverage measures based on the control flow computable
level 2◦ additional to level 1 the assignment of every atom is logged◦ all coverage measures computable
level 3◦ complete instrumentation: additional to level 2 the execution of every
statement is logged◦ allows control flow analysis of abnormal terminating programs
instrumentation: levels
![Page 13: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/13.jpg)
DAAD Workshop Neum, 2009 15
instrumentation (cont.)if (n == 0 | n == 1) { return result;}
![Page 14: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/14.jpg)
DAAD Workshop Neum, 2009 16
instrumentation (cont.)
for (int i = n; i >= 2; i--) { result *= i;}
![Page 15: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/15.jpg)
DAAD Workshop Neum, 2009 17
3 modes of usage◦manual program testing
pre procession: SOTA test execution: by hand post procession: SOTA
◦ testing with an external test system (e.g. ATOSj) pre procession: SOTA test execution: external test system post procession: SOTA
◦ testing with an automatic test system usage of non-gui functionality by SOTA library accessible via ASTManager object or command line
modes of usage
![Page 16: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/16.jpg)
DAAD Workshop Neum, 2009 18
phase 1: preparation◦ read in and parsing the source code◦configure the instrumentation◦ instrumentation of source code
phase 2: test execution◦compiling of the instrumented source code◦ test execution including the logging of information
phase 3: test evaluation◦ restoration of the original code◦ read In the log files◦calculation of the coverage measure◦visualization of the results◦creation of report
workflow
![Page 17: Michael Hildebrandt H UMBOLDT U NIVERSITY B ERLIN D EPARTMENT O F C OMPUTER S CIENCE C HAIR O F S OFTWARE E NGINEERING](https://reader035.vdocuments.us/reader035/viewer/2022062417/5518c433550346881f8b5747/html5/thumbnails/17.jpg)
DAAD Workshop Neum, 2009 19
Questions ? Comments?
Thank you.