1/23/2003university of virginia1 korat: automated testing based on java predicates cs751...

1/23/2003 University of Virginia 1

Korat: Automated Testing Based on Java Predicates

CS751 Presentationby Radu Stoleru

C.Boyapaty, S.Khurshid, D.Marinov


Roadmap Why do they do it?

Statement of the Problem State of the Art

What do they do? How do they do it?

Test Input Generation Checking Correctness

Results & Evaluation Questions & Comments


Roadmap Why do they do it?

Statement of the Problem State of the Art

What do they do?What do they do? How do they do it?How do they do it?

Test Input GenerationTest Input Generation Checking CorrectnessChecking Correctness

Results & EvaluationResults & Evaluation Questions & CommentsQuestions & Comments


Why do they do it? Automated Testing. Why improve testing? Through manual testing:

significant errors are not found it takes 30% of development time automated testing is an industry standard validation

Automated Testing consists of: automated generation of test cases from

specifications automated execution of test cases automated validation


Why do they do it? Specification-based testing:

Z specification, UML statechart – no linked data structs

TestEra framework (’01) – new specification language JML+JUnit (’01) – no test case generation

Static Analysis Extended Static Checker (’98) – no complex structs TVLA (’98) – only limited program properties

Software model checking JavaPathFinder (’00), VeriSoft (’98) – no linked data

structs


Why do they do it? Korat:

automated generation of test cases for complex structs complete evaluation of correctness automatically generates counter-examples no new specification language

Testing Framework

JUnit JML+JUnit Korat

generating test cases

generating test oracle

running tests


Roadmap Why do they do it?Why do they do it?

Statement of the ProblemStatement of the Problem State of the ArtState of the Art

What do they do? How do they do it?How do they do it?




What do they do? use JML for formal specification (class

invariants, preconditions, postconditions) generate test inputs using preconditions

builds Java predicate builds a skeleton finitization prunes input state space generates isomorph-free test cases

evaluate correctness using postconditions using JML/JUnit




What do they do?What do they do? How do they do it?

Test Input Generation Checking Correctness



Exampleclass BinaryTree {

//@ public invariant //@ repOk(); Node root; int size;

static class Node { Node left; Node right; }

/*@ public normal_behavior @ requires has(n); @ ensures !has(n); @*/ void remove(Node n) { ... }}

boolean repOk() { if (root == null) return size == 0; Set visited = new HashSet(); visited.add(root); List workList = new LinkedList();

workList.add(root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; workList.add(current.right); } } if (visited.size() != size) return false; return true;}


Input Size 5 non-isomorphic solutions for 3 nodes:

N0

N1

N2

N0

N1

N2

(n+1)2n+1 candidates for n nodes (292 for 12 nodes) how to find them quickly?

left

right

right

right

N1

N2

left

right

N0

N1

N2

left

left

N0

N1 N2

left right

N0


Search Korat search algorithm:

void koratSearch(Predicate p, Finitization f) { initialize(f); while(hasNextCandidate()) { Object candidate = nextCandidate(); try { if(p.invoke(candidate)) output(candidate); } catch (Throwable t) {} backtrack(); } }

given a predicate and a finitization, candidate inputs are generated inputs are validated by invoking the predicate on them


Finitization a set of bounds that limits the size of the input Class Domain := a set of objects from one

class {N0, N1, N2} Field Domain := a set of values a field can

take. For Node.left it is {null, N0, N1, N2}Finitization finBinaryTree(int n, int min, int max) { Finitization f = new Finitization(BinaryTree.class); ObjSet nodes = f.createObjects(“Node”, n); nodes.add(null); f.set(“root”, nodes); // Field

Domain f.set(“size”, new IntSet(min, max)); // Field Domain f.set(“Node.left”, nodes); // Field Domain f.set(“Node.right”, nodes); // Field Domain return f; }

generated automatically by Korat

can be further specialized


State Space using a finitization, Korat:

allocates a given number of objects constructs candidate vectors using object fields:

‘root’, ‘left’, ‘right’: {null, N0, N1, N2} size: {3}

root size left right left right left right

BinaryTree N0 N1 N2

N1 N2

left

right

N0: [N0, 3, N1, N1, null, null, null, null]


Search for each candidate vector, Korat:

invokes repOk() and monitors the execution builds a field ordering (list of fields ordered by the

accessed time) if repOk() returns true, output the structure if repOk() returns false, backtracks on the last

accessed field, using the field ordering


Search

when repOk() returns false, the field ordering is:

N1 N2

left

right

N0

|root, N0.left, N0.right|

[N0, 3, N1, N1, null, null, null, null]

backtracking on N0.right, gives the next candidate: (increments the field domain index for the

field that is last in the field ordering)

N1 N2

leftright

N0[N0, 3, N1, N2, null, null, null, null]


Search

N1 N2

left

right

N0

[N0, 3, N1, N1, null, null, null, null]

N1 N2

leftright

N0 [N0, 3, N1, N2, null, null, null, null]

with backtracking, Korat prunes 44 candidates of type:

[N0, 3, N1, N1, _, _, _, _]


Nonisomorphism two candidates are isomorphic if:

; o, o’ OC,r ; f fields(o) ; p P .

o.f == o’ in C <=> (o).f == (o’) in C’ ando.f == p in C <=> (o).f == p in C’

isomorphism => state space partitioned only the lexicographically smallest candidate is

generated it is used to increment field domain indices by

more than 1.

N1

N2

left

left

N0

N0

N2

left

left

N1


Nonisomorphism-Algorithm

N2 N1

leftN0

N2

N1

left

right

N0

|root, N0.left, N0.right, N2.left, N2.right|

backtracking on a field f (pointer to object of of class cf):

class domain: cf {N0, N1, N2}

[N0, 3, N2, null, null, null, null, null]

[N0, 3, N2, null, null, null, null, N1] (?)


Generating Test Cases to generate test inputs for method m, Korat

builds a class that represents m’s inputs builds repOk() that checks m’s precondition generates all inputs that satisfy repOk()

class BinaryTree_remove { //@ invariant repOk(); BinaryTree This; BinaryTree.Node n; boolean repOk() { return This.repOk() && This.has(n); }}

class BinaryTree { //@ invariant repOk(); ... //@ requires has(n); void remove(Node n) { ... }}


Checking Correctness Korat uses:

JML toolset for generating oracles JUnit for executing tests and reporting errors

to test a method m, Korat invokes m on each input and test the output using the oracle






Results & Evaluation Questions & CommentsQuestions & Comments


Results & EvaluationBenchmark Size State

SpaceStructs

Generated

Time(sec)

BinaryTree 812

253

292

1430208,012

1.53233.59

HeapArray 68

220

229

131391,005,075

1.2142.61

LinkedList 812

291

2150

4,1404,213,597

1.32690.00

TreeMap 79

292

2130

35122

8.812148.5

0

HashSet 711

2119

2215

2386277,387

3.71926.71

AVTree 5 250 598,358 62.05


Results & EvaluationBenchmark Method Max

SizeTest

CasesGen Time (sec)

Test Time (sec)

BinaryTree remove 3 15 0.64 0.73

HeapArray extractMax

6 13,139 0.87 1.39

LinkedList reverse 2 8 0.67 0.76

TreeMap put 8 19,912 136.19 2.70

HashSet add 7 13,106 3.90 1.72

AVTree lookup 4 27,734 4.33 14.63






Results & EvaluationResults & Evaluation Questions & Comments


Questions & Comments non-Java environments? clear enough explanations for algorithms? proof for the search algorithm? paper quality: outstanding / good / bad /

awful ? anything else you want to add?

1/23/2003university of virginia1 korat: automated testing based on java predicates cs751...

Documents