search-based testing of procedural programs: iterative ... · search-based testing of procedural...

39
Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? Andrea De Lucia Rocco Oliveto Dario Di Nucci Simone Scalabrino Giovanni Grano

Upload: others

Post on 12-Aug-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Search-based Testing of Procedural Programs:

Iterative Single-Target or Multi-Target Approach?

Andrea De LuciaRocco Oliveto

Dario Di NucciSimone Scalabrino Giovanni Grano

Page 2: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

“The overall cost of testing has been estimated at being at least half of the entire development cost, if not more.

Boris Beizer

Beizer. Software testing techniques. 2003. Dreamtech Press.

Page 3: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

“Software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.

Beller et al. When, how, and why developers (do not) test in their IDEs. ESEC/FSE 2015

Beller et al.

Page 4: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Test case generation

Page 5: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Test case generation

SBST Methodologies

Page 6: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Test case generation

SBST Tools

AUSTIN

eToc - evolutionary Testing of classes

Page 7: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

OCELOT

Optimal Coverage sEarch-based tooL for sOftware Testing

Page 8: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Test Case Generation for C

Fully Implemented in Java and C (through JNI)

Based on JMetal Framework

Structs and Pointers Handling

Check Unit Testing Framework

OCELOT

Features

Page 9: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

OCELOT

Why C?

Page 10: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Code Execution

CE

SS

CI

TS

PB

Makefile Generation

Program Building

Target Selection

Solution Search

Code Instrumentation

OCELOT

Process Overview

MG

Page 11: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

MOSAMulti-Objective Sorting Algorithm*

Panichella et al.

LIPSLinearly Independent Path based Search

Scalabrino et al.

OCELOT

Target Selection Algorithms

Page 12: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Many-ObjectiveSorting Algorithm

Fitness function reformulationFind a set of test cases that optimizes the branch coverage of each branch

Dominance and Pareto optimalityEach solution is evaluated in terms of Pareto dominance and optimality,

Preference SortingA new ranking algorithm for sorting the solutions

Page 13: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Linearly IndependentPath based Search

Inspired by McCabe baseline method and Dynamic Symbolic Execution

● Starts with random test data (t0)● Selects a new target at each iteration

a > 1

a++ a--

End

Start

Single target selection algorithm

Collateral coverage

Considers the targets serendipitously achieved

Search budget optimization

Allocates SBi/ni evaluations for each target

Seeding

Final population of the previous iteration is reused

Independent of the search algorithm

Current implementation based on GA

Page 14: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

EMPIRICAL EVALUATION

Page 15: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Research Questions

RQ1Effectiveness

BranchCoverage

RQ3Oracle Coast

Test SuiteSize

RQ2Efficiency

ExecutionTime

MOSA LIPS

Page 16: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Design

Experiment Details

Settings

ContextPopulation Size 100

Crossover Rate 0.90

Mutation Rate 1/#variables

Search Budget 200.000

35 C functions605 branches

Gimp: Gnu Image Manipulation ProgramGSL: Gnu Scientific LibrarySGLIB: a generic library for Cspice: analogue circuit simulator

30 runsAverage ValuesWilcoxon’s Test (p-value 0.05)Vargha-Delaney Test

Page 17: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Results RQ1:

Effectiveness

MOSA LIPS

84,73% 86.29%Overall Branch

Coverage

Cases in which is better

21 102

1 1 case with large effect size28 cases with medium/large effect size

Page 18: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

MOSA LIPS

14.80s 5.03sAverage

Execution Time

Cases in which is better

0 351

Results RQ2:

Efficiency

1 with large effect size

Page 19: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

MOSA LIPS

14.80s 5.03sAverage

Execution Time

Cases in which is better

0 351

Results RQ2:

Efficiency

1 with large effect size

Too much time for ranking the

Pareto Fronts!

Page 20: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Results RQ3:

Oracle Cost

MOSA LIPS

4.4 6.1Average

# Test Cases

Cases in which is better

351 0

1 33 cases with large effect size

Page 21: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Results RQ3:

Oracle Cost

MOSA LIPS

4.4 6.1Average

# Test Cases

Cases in which is better

351 0

1 33 cases with large effect size

LIPS does not directly handle the oracle

problem!

Page 22: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

LIPS*no collateral coverage

MOSA LIPS*

4.4 4.25Average

# Test Cases

Cases in which is better

6 7

1 Although better than MOSA

Worse than LIPS in terms of branch coverage Worse than LIPS in terms of execution time1

Page 23: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

LIPS + minimization

MOSA1 LIPS

3.61 3.66Average

# Test Cases

Cases in which is better

62 23

1 For a fair comparison the minimization was applied also to MOSA suites2 4 cases with medium/large effect size3 1 case with large effect size

Page 24: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

LIPS + minimization

1 For a fair comparison the minimization was applied also to MOSA suites2 4 cases with medium/large effect size3 1 case with large effect size

Minimizationexecution time < 1s

MOSA1 LIPS

3.61 3.66Average

# Test Cases

Cases in which is better

62 23

Page 25: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 26: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 27: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 28: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 29: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 30: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 31: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 32: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 33: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Conclusions

Page 34: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Future works

Page 35: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Future works

Replicate on larger dataset

Page 36: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Future works

Replicate on larger dataset

Study C landscapes

Page 37: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Future works

Replicate on larger dataset

Study C landscapes

Add LIPS to Evosuite

Page 38: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Future works

Replicate on larger dataset

Study C landscapes

Add LIPS to Evosuite

Go beyondGenetic Algorithm

Page 39: Search-based Testing of Procedural Programs: Iterative ... · Search-based Testing of Procedural Programs: Iterative Single-Target or Multi-Target Approach? ... Beizer. Software testing

Dario Di Nucci

University of Salerno

[email protected]

http://www.sesa.unisa.it/people/ddinucci/

Thanks for your attention!

Questions?