coverage and mutation testing tools - fbk | fondazione bruno kessler

Coverage and Mutation testing tools

So#ware Analysis and Tes1ngCu Nguyen Duy (cunduy at :k dot eu)

Alessandro Marche?o (marche?o at :k dot eu)

Paolo Tonella (tonella at :k dot eu)

Mariano Ceccato (ceccato at :k dot eu)

Academic Year 2010-‐2011

Tuesday, November 9, 2010

mailto:[email protected]








Coverage testing (recall)

Coverage measures describe the degree to which a program has been tested tell the effectiveness in terms of coverage of the test

set, help improve software quality inform quantitatively the project manager about the

progress of testingMany type of coverage measures

statementspathsmethods, classesrequirement specifications, etc. 2


Coverage tool features

Compile time: automatic instrument code for coverage recording

Runtime: structure-based coverage recording lines, blocks, conditions, methods and classes

Reportingquantitative coverage measure

percentage of code executedmost executed v.s. never executed code

visual navigationquickly navigate to code that is not executed to improve

the test set 3


Coverage tools

4

Emma: free, open sourcehttp://emma.sourceforge.net/

CodePro Analytix: use Emma, free, developed by Google http://code.google.com/javadevtools/

codepro/doc/index.htmlClover: commercial

http://www.atlassian.com/software/clover/ Many others


http://emma.sourceforge.net/

http://emma.sourceforge.net/

http://code.google.com/javadevtools/codepro/doc/index.html




http://www.atlassian.com/software/clover/

http://www.atlassian.com/software/clover/

Mutation testing


Introduction

6

program

Test CaseTest CaseTest

CaseTest CaseTest

Case


Introduction

6

program


CaseTest CaseTest

Case

How good these test cases are?- fault detection - coverage


Introduction

6

program


CaseTest CaseTest

Case


what if no fault foundwhat if 100% covered


Introduction

6

program


CaseTest CaseTest

Case


what if no fault foundwhat if 100% covered

- mutant core with mutation testing


Mutation testing terms

Mutant: a copy of the original program with a small change (seeded fault)

Mutation operator: applied to make change (automatically) to the original program

Mutant killed: if its behaviors/outputs differ from those of the original program

Mutant score: of a test case = number of killed mutants / total number of mutants

7


Mutation operators

Method-level operators, example:Arithmetic operator, e.g. “-” -> “+”Relational operator, e.g. “ > ” -> “ < ”Conditional operator, e.g. “ && ” -> “ || ”

Class-level operators, example: EncapsulationInheritancePolymorphism

8http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf

http://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf


http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf




Mutant creation

9

program

mutant 1

mutant 2

mutant N

Mutation operator


Mutant creation

9

program

mutant 1

mutant 2

mutant N

Mutation operator

package math;

public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}


Mutant creation

9

program

mutant 1

mutant 2

mutant N

Mutation operator

package math;


package math;

public class Add {

public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}


Mutant score

10

program

mutant 1

mutant 2

mutant N


Mutant score

10

program

mutant 1

mutant 2

mutant N

TC1


Mutant score

10

program

mutant 1

mutant 2

mutant N

TC1

✔

✘

✔

✔


Mutant score

10

program

mutant 1

mutant 2

mutant N

TC1

✔

✘

✔

✔

MS(TC1) = killed mutants/ N


Example

11


public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}


Example

11


public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}

TC1:Add o = new Add();print(o.sum(1,2));print(o.sum(1.0,2.0));

MutantScore(TC1) = ?


Mutation testing tools

12

Jumble: free, open sourcehttp://jumble.sourceforge.net/

MuJava: freehttp://cs.gmu.edu/~offutt/mujava/

MuClipse: free, built on top of MuJavahttp://muclipse.sourceforge.net/doesn’t support Java >= 5

Many others


http://jumble.sourceforge.net/

http://jumble.sourceforge.net/

http://cs.gmu.edu/~offutt/mujava/

http://cs.gmu.edu/~offutt/mujava/

http://muclipse.sourceforge.net/

http://muclipse.sourceforge.net/

coverage and mutation testing tools - fbk | fondazione bruno kessler

Documents