search-based testing of procedural programs:iterative single-target or multi-target approach?
TRANSCRIPT
![Page 1: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/1.jpg)
Search-based Testing of Procedural Programs:
Iterative Single-Target or Multi-Target Approach?
Andrea De LuciaRocco Oliveto
Dario Di NucciSimone Scalabrino Giovanni Grano
![Page 2: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/2.jpg)
“The overall cost of testing has been estimated at being at least half of the entire development cost, if not more.
Boris Beizer
Beizer. Software testing techniques. 2003. Dreamtech Press.
![Page 3: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/3.jpg)
“Software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.
Beller et al. When, how, and why developers (do not) test in their IDEs. ESEC/FSE 2015
Beller et al.
![Page 4: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/4.jpg)
Test case generation
![Page 5: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/5.jpg)
Test case generation
SBST Methodologies
![Page 6: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/6.jpg)
Test case generation
SBST Tools
AUSTIN
eToc - evolutionary Testing of classes
![Page 7: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/7.jpg)
OCELOT
Optimal Coverage sEarch-based tooL for sOftware Testing
![Page 8: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/8.jpg)
Test Case Generation for C
Fully Implemented in Java and C (through JNI)
Based on JMetal Framework
Structs and Pointers Handling
Check Unit Testing Framework
OCELOT
Features
![Page 9: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/9.jpg)
OCELOT
Why C?
![Page 10: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/10.jpg)
Code Execution
CE
SS
CI
TS
PB
Makefile Generation
Program Building
Target Selection
Solution Search
Code Instrumentation
OCELOT
Process Overview
MG
![Page 11: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/11.jpg)
MOSAMulti-Objective Sorting Algorithm*
Panichella et al.
LIPSLinearly Independent Path based Search
Scalabrino et al.
OCELOT
Target Selection Algorithms
![Page 12: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/12.jpg)
Many-ObjectiveSorting Algorithm
Fitness function reformulationFind a set of test cases that optimizes the branch coverage of each branch
Dominance and Pareto optimalityEach solution is evaluated in terms of Pareto dominance and optimality,
Preference SortingA new ranking algorithm for sorting the solutions
![Page 13: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/13.jpg)
Linearly IndependentPath based Search
Inspired by McCabe baseline method and Dynamic Symbolic Execution
● Starts with random test data (t0)● Selects a new target at each iteration
a > 1
a++ a--
End
Start
Single target selection algorithm
Collateral coverage
Considers the targets serendipitously achieved
Search budget optimization
Allocates SBi/ni evaluations for each target
Seeding
Final population of the previous iteration is reused
Independent of the search algorithm
Current implementation based on GA
![Page 14: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/14.jpg)
EMPIRICAL EVALUATION
![Page 15: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/15.jpg)
Research Questions
RQ1Effectiveness
BranchCoverage
RQ3Oracle Coast
Test SuiteSize
RQ2Efficiency
ExecutionTime
MOSA LIPS
![Page 16: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/16.jpg)
Design
Experiment Details
Settings
ContextPopulation Size 100
Crossover Rate 0.90
Mutation Rate 1/#variables
Search Budget 200.000
35 C functions605 branches
Gimp: Gnu Image Manipulation ProgramGSL: Gnu Scientific LibrarySGLIB: a generic library for Cspice: analogue circuit simulator
30 runsAverage ValuesWilcoxon’s Test (p-value 0.05)Vargha-Delaney Test
![Page 17: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/17.jpg)
Results RQ1:
Effectiveness
MOSA LIPS
84,73% 86.29%Overall Branch
Coverage
Cases in which is better
21 102
1 1 case with large effect size28 cases with medium/large effect size
![Page 18: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/18.jpg)
MOSA LIPS
14.80s 5.03sAverage
Execution Time
Cases in which is better
0 351
Results RQ2:
Efficiency
1 with large effect size
![Page 19: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/19.jpg)
MOSA LIPS
14.80s 5.03sAverage
Execution Time
Cases in which is better
0 351
Results RQ2:
Efficiency
1 with large effect size
Too much time for ranking the
Pareto Fronts!
![Page 20: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/20.jpg)
Results RQ3:
Oracle Cost
MOSA LIPS
4.4 6.1Average
# Test Cases
Cases in which is better
351 0
1 33 cases with large effect size
![Page 21: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/21.jpg)
Results RQ3:
Oracle Cost
MOSA LIPS
4.4 6.1Average
# Test Cases
Cases in which is better
351 0
1 33 cases with large effect size
LIPS does not directly handle the oracle
problem!
![Page 22: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/22.jpg)
LIPS*no collateral coverage
MOSA LIPS*
4.4 4.25Average
# Test Cases
Cases in which is better
6 7
1 Although better than MOSA
Worse than LIPS in terms of branch coverage Worse than LIPS in terms of execution time1
![Page 23: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/23.jpg)
LIPS + minimization
MOSA1 LIPS
3.61 3.66Average
# Test Cases
Cases in which is better
62 23
1 For a fair comparison the minimization was applied also to MOSA suites2 4 cases with medium/large effect size3 1 case with large effect size
![Page 24: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/24.jpg)
LIPS + minimization
1 For a fair comparison the minimization was applied also to MOSA suites2 4 cases with medium/large effect size3 1 case with large effect size
Minimizationexecution time < 1s
MOSA1 LIPS
3.61 3.66Average
# Test Cases
Cases in which is better
62 23
![Page 25: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/25.jpg)
Conclusions
![Page 26: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/26.jpg)
Conclusions
![Page 27: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/27.jpg)
Conclusions
![Page 28: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/28.jpg)
Conclusions
![Page 29: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/29.jpg)
Conclusions
![Page 30: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/30.jpg)
Conclusions
![Page 31: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/31.jpg)
Conclusions
![Page 32: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/32.jpg)
Conclusions
![Page 33: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/33.jpg)
Conclusions
![Page 34: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/34.jpg)
Future works
![Page 35: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/35.jpg)
Future works
Replicate on larger dataset
![Page 36: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/36.jpg)
Future works
Replicate on larger dataset
Study C landscapes
![Page 37: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/37.jpg)
Future works
Replicate on larger dataset
Study C landscapes
Add LIPS to Evosuite
![Page 38: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/38.jpg)
Future works
Replicate on larger dataset
Study C landscapes
Add LIPS to Evosuite
Go beyondGenetic Algorithm
![Page 39: Search-based testing of procedural programs:iterative single-target or multi-target approach?](https://reader033.vdocuments.us/reader033/viewer/2022051507/58a097941a28ab9f758b4a3d/html5/thumbnails/39.jpg)
Dario Di Nucci
University of Salerno
http://www.sesa.unisa.it/people/ddinucci/
Thanks for your attention!
Questions?