edsketch: execution-driven sketching for java
TRANSCRIPT
![Page 1: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/1.jpg)
EdSketch: Execution-Driven Sketching for Java
Jinru Hua and Sarfraz Khurshid
University of Texas at Austin
7/14/2017
![Page 2: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/2.jpg)
Execution-Driven Sketching - Motivation
● A sketch is a partial program with holes, which can be completed by automated tools.
● Existing program sketching approaches translate sketch to SAT, which may lead to impractical problem to translate all libraries or create models for libraries.
● We present execution-driven sketching, a synthesis approach using backtracking search.
● Our key novelty is to introduce effective pruning strategies to explore actual behaviors with libraries.
2
![Page 3: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/3.jpg)
Outline
● Overview and Motivation
● Motivating Example
● Approach
● Evaluation
● Summary
3
![Page 4: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/4.jpg)
class LinkedList { Entry head; public void reverse() { Entry ln1= ; Entry ln2=null; Entry ln3=null; while ( ) { } } }
reverse(): reverse the singly linked list
Motivating Example - Program Sketch
Original Input Expected Output
next1 2 3
head
3 2 1
head
4
@Testpublic void testThreeEntries() { LinkedList list = new LinkedList(new int[]{1,2,3}); list.reverse(); assertEquals(“[3,2,1]”, list.toString());}
![Page 5: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/5.jpg)
@Testpublic void testThreeEntries() { LinkedList list = new LinkedList(new int[]{1,2,3}); list.reverse(); assertEquals(“[3,2,1]”, list.toString());}
class LinkedList { Entry head; public void reverse() { Entry ln1=EdSketch.EXP(...); Entry ln2=null; Entry ln3=null; while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...); } } }
reverse(): reverse the singly linked list
Motivating Example - A user-written sketch
Original Input Expected Output
next1 2 3
head
3 2 1
head
5
![Page 6: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/6.jpg)
class LinkedList { Entry head; public void reverse() { Entry ln1=EdSketch.EXP(...); Entry ln2=null; Entry ln3=null; while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...); } } }
reverse(): reverse the singly linked list
Motivating Example - A complete program
Original Input Expected Output
next1 2 3
head
3 2 1
head
6
1. EdSketch.EXP(...):head
2. EdSketch.COND(...):
ln1 != null
3. EdSketch.ASSIGN_BLOCK(...):
head = ln1; ln1 = head.next;head.next = ln2;ln2 = head;
![Page 7: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/7.jpg)
Outline
● Overview and Motivation
● Motivating Example
● Approach
● Evaluation
● Summary
7
![Page 8: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/8.jpg)
Architecture - Expr. Gen. & Instrumentation
8
![Page 9: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/9.jpg)
Approach - Expression Generation
● Generate field dereferences for all visible variables within a predefined bound of field deref., e.g., 1.
● Assign a unique identifier for each candidate.- E.g., head = ln1 is represented as [0, 1].- A 4-stmt block is represented as [0,1,1,4,4,2,2,0]
head = ln1; ln1 = head.next;head.next = ln2;ln2 = head;
● Non-deterministically select an identifier- int left_id = Verify.getInt(0,7);
9
head ln1 ln2 ln3 head.next ln1.next ln2.next ln3.next
0 1 2 3 4 5 6 7
![Page 10: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/10.jpg)
Instrumentation Example - Assignments
10
public void reverse() {... while (EdSketch.COND(...)) { EdSketch.ASSIGN_BLOCK(...);
}
![Page 11: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/11.jpg)
11
public void reverse() {... while (EdSketch.COND(...)) {+ for (Statement s: EdSketch.ASSIGN_BLOCK(...)) {+ EdSketch_Assign stmt = (EdSketch_Assign) s;+ Entry rhs_val = (Entry) stmt.getRHS();+ switch (stmt.getLHS_id()) {+ case 0: head = rhs_val; break;+ case 1: ln1 = rhs_val; break;
...+ case 7: ln3.next = rhs_val; break;+ }+ } }
0 head
1 ln1
2 ln2
3 ln3
4 head.next
5 ln1.next
6 ln2.next
7 ln3.next
Instrumentation Example - Local var. update
![Page 12: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/12.jpg)
Architecture - Pruning and Backtracking
12
![Page 13: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/13.jpg)
● If a candidate satisfies any rules, EdSketch prunes it without execution against tests.
- “E1 = E1;” 一 omitted (no-op)
- “V1 = V2; V2 = V1;” 一 subsumed by “V1 = V2;”
- “V1 = V2; V1 = V3;” 一 subsumed by “V1 = V3;”
- “V3 = V1; V2 = V1;” and id(V3)>id(V2)- isomorphic to “V2 = V1; V3 = V1;”
Approach - Assignment pruning rules
13
![Page 14: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/14.jpg)
Condition Pruning Strategies
● Condition Candidate Generation
○ Operators that combine expressions, e.g., ln1 != null■ Primitive types:■ Non-primitive types:
○ Based on symmetry, we combine an expression with the ones with bigger identifiers. ■ only consider e==f and e!=f, ignore f==e and f!=e.
○ Two constant values: true, false (e==e and e!=e)
● Condition Value Grouping○ Split condition candidates based on evaluated value (true or
false)14
![Page 15: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/15.jpg)
● Stateful backtracking using Java PathFinder (JPF)● Stateless backtracking inspired by VeriSoft
○ Using Explorer class from Juzi [Suen-UT-MS-05]
Approach - Backtracking Engines
15
![Page 16: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/16.jpg)
Outline
● Overview and Motivation
● Motivating Example
● Approach
● Evaluation
● Summary
16
![Page 17: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/17.jpg)
Evaluation - Subjects
● 10 data structure subjects, bounded exhaustive test suites, avg. 9 test cases (wrote oracles manually).
17
Type Name Test 1 2 3 4
A BSTAS 8 196 38.4 K 7.5 M 1.5 B
A MEDAS 7 16 256 4.1 K 66 K
A LLREV 4 64 4.1 K 262.4 K 16.7 M
A RBTAS 15 676 457 K 309 M 209 B
A DLLAF 4 196 38.4 K 7.5 M 1.5 B
A DLLAL 4 196 38 K 7.5 M 1.5 B
C BSTCD 8 392 190.5 K 74.7 M 29.3 B
C RBTRM 15 74 5.5 K 7.4 M 10.0 B
C MEDCD 7 96 9.2 K 884.9 M 84.9 M
C RBTCD 15 74 5.5 K 7.4 M 10.0 B
![Page 18: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/18.jpg)
● Sketch system is a SAT-based synthesizer
● Manually translate subjects to Sketch language
- Translate the sketch, e.g., while ({|(head|ln1|...|ln3.next) (==|!=) (head|ln1|...|ln3.next|null)|})
- Translate the test suite, e.g., harness void test() { LinkedList l = new LinkedList(...); ... //list “1,2,3” reverse(l.head); assert l.head.val == 3; ... }
Evaluation - Comparison with Sketch
18
![Page 19: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/19.jpg)
Evaluation - Performance
● Comparison with Sketch synthesizer
19
![Page 20: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/20.jpg)
Evaluation - Performance
● Comparison with Sketch synthesizer
20
EdSketch-JVM can sometimes even be faster than the SAT-based Sketch synthesizer.
![Page 21: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/21.jpg)
Evaluation - EdSketch Pruning Strategies
● Comparison of number of executed programs with and without pruning strategies.
21
![Page 22: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/22.jpg)
Evaluation - EdSketch Pruning Strategies
● Comparison of number of executed programs with and without pruning strategies.
22
Assignment Pruning--Avg. reduction rate: 21%Condition Pruning--Avg. reduction rate: 56%
![Page 23: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/23.jpg)
Evaluation - Larger Application● Our approach can handle sketching tasks from Java
applications. - A task from Apache commons-lang with 55 KLOC code and
3,274 test cases.
23
public String random(int end, char[] chars,...) {
if (chars != null) {
end = chars.length;
}...
}
/* Program sketch after manual transformation */
public String random(int end, char[] chars,...) {
if (EdSketch.COND(...)) {
EdSketch.ASSIGN_BLOCK(...);
}...
}
![Page 24: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/24.jpg)
Evaluation - Advanced Features● Our approach can sketch programs with reflection,
I/O, and native calls.
24
public boolean repOK() throws Exception {...
LinkedList<Node> workList=new LinkedList<Node>();
workList.add(root);
while (!workList.isEmpty()) {
Node cur= workList.removeFirst();
Node exp=(Node) EdSketch.EXP(...);
//expect current.left
if(Node.class.getField("left").get(exp)!=null) {
...//omit the rest
}
}
}
![Page 25: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/25.jpg)
Related Work
25
● Some related program synthesis approaches:○ Sketching partial programs [Sketch’06, JSketch’15]
○ Synthesizing API chains [Prospector’05, CodeHint’14, SyPet’17]
○ Embedding angelic choice construct with backtracking solver but no pruning strategies. [Angelic Programming’10]
● Some related strategies in program repair approaches:○ Assignment pruning rules are similar to adaptive equivalence
for program repair [AE’13]
○ Condition value grouping is similar to abstract condition values for repair [SPR’15]
![Page 26: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/26.jpg)
Summary
● We introduce a novel execution-driven approach to sketch Java programs using backtracking search.
● We introduce effective pruning strategies to explore actual behaviors with libraries.
● The experimental results show that the performance of EdSketch compares well with SAT-based Sketch synthesizer based on our subjects.
● EdSketch can complete some sketches in presence of complex constructs.
26
![Page 27: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/27.jpg)
27
![Page 28: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/28.jpg)
Backup Slides
28
![Page 29: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/29.jpg)
Partial Expression Syntax
● EdSketch.COND(...):- EdSketch.COND(Entry.class, new String[]{“head”,
“ln1”, “ln2”, “ln3”}, new Object[]{head, ln1, ln2, ln3});
● EdSketch.BLOCK(...):- EdSketch.BLOCK(Entry.class, new String[]{“head”,
“ln1”, “ln2”, “ln3”}, new Object[]{head, ln1, ln2, ln3});
● Set the number of assignment statements:- EdSketch.BLOCK(...).setLength(4);
29
![Page 30: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/30.jpg)
Approach - Condition Value Grouping
● Split condition candidates based on evaluated value (true or false)
30
![Page 31: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/31.jpg)
Instrumentation - Handle infinite loop
31
+ private int count = 0; public void reverse() {... while (EdSketch.COND(Entry.class)) {+ for (Statement s: EdSketch.ASSIGN_BLOCK(Entry.class)) {+ EdSketch_Assign stmt = (EdSketch_Assign) s;+ Entry rhs_val = (Entry) stmt.getRHS();+ switch (stmt.getLHS_id()) {+ case 0: head = rhs_val; break;+ case 1: _ln1_ = rhs_val; break;
...+ case 7: _ln3_.next = rhs_val; break;+ }+ }+ if (count++ > EdSketch.LOOPBOUND)+ EdSketch.backtrack(); }
0 head
1 ln1
2 ln2
3 ln3
4 head.next
5 ln1.next
6 ln2.next
7 ln3.next
![Page 32: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/32.jpg)
EdSketch-JPF vs. EdSketch-JVM
32
EdSketch-JPF EdSketch-JVM
● Quick to implement, off-the-shelf enabling technology
● Efficient
● Real backtracking:○ Handles JVM memory-
level side effects
● Resets relevant parts of state○ Assumes running the
same JUnit suite again gives the same outcome
● Can be extended to sketch multithreaded code
● Handles native code, complex libraries
![Page 33: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/33.jpg)
Discussion
● The efficacy of our pruning strategy is consistent in finding the first solution and all solutions.
● The order of test cases has negligible influence on sketching performance based on our subjects.
● We only compare the performance with one state-of-the-art synthesizers on our dataset.
● We manually translate Sketch program and tests.● Our bounded exhaustive test suite may still lead to
plausible sketching results, i.e., passing all tests but are incorrect based on manual inspection.
33
![Page 34: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/34.jpg)
Sketch Procedure
34
![Page 35: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/35.jpg)
EdSketch Condition Candidate Generation
35
![Page 36: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/36.jpg)
EdSketch Value Grouping Algorithm
36
![Page 37: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/37.jpg)
● If a candidate satisfies any rules, EdSketch prunes it without execution against tests.
- E1 = E2
- V1 = V2, V2 = V1
- V1 = V2, V1 = V3
- V3 = V1, V2 = V1 & id(V3)>id(V2) - isomorphic to V2 = V1, V3 = V1
Approach - Assignment pruning rules
37
ln1.next = ln1.next
ln1 = ln2, ln2 = ln1
ln1 = ln2, ln1 = ln3
ln3 = ln1, ln2 = ln1
![Page 38: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/38.jpg)
Evaluation - File I/O● Our approach can sketch programs with reflection,
I/O, and native calls.
38
public int getSum(String file) {
Scanner scan = new Scanner(new File(file));
int a = scan.nextInt();
int b = scan.nextInt();
//expect a+b
return (int) EdSketch.EXPOP(new double[]{a,b});
![Page 39: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/39.jpg)
Evaluation - Native Calls● Our approach can sketch programs with reflection,
I/O, and native calls.
39
public native String nativeToString(int x, int y);
public String swap(int x, int y) {
int tmp = x;
//expect x = y, y = tmp
EdSketch.ASSIGN_BLOCK(int.class);
String str = nativeToString(x,y);
return str;
}
![Page 40: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/40.jpg)
Counter-example-driven synthesis
40
Synthesize control c that completes S for a random input; check if the control c also works for all other inputs. If not, add counter example input and repeat
Reference:A.Solar-Lezama, L.Tancau, R.Bodik, V.Saraswat, S.Seshia. Combinatorial Sketching for Finite Programs. ASPLOS’06, page 404-415.
![Page 41: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/41.jpg)
How automated is EdSketch?
To create: conditions (“exp rop exp”) and assignment statements (“exp = exp;”), where rop ∈ {<, >, ≤, ≥, ==, !=}At the algorithm level:
Inputs: type T for exp, bound B for number of dereferences, bound S on number of statements
At the implementation level:Inputs: T, B, S, root pointers (Map<VarName, Var>)Manual instrumentation:
Inserting new fields to model local variablesInserting non-deterministic choices (e.g., switch-case statements) 41
![Page 42: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/42.jpg)
Reference - Program Synthesis[Sketch’06] A.Solar-Lezama, L.Tancau, R.Bodik, S.Seshia, V.Saraswat. Combinatorial sketching for finite programs. In ASPLOS, pages 404–415, 2006.
[JSketch’15] J.Jeon, X.Qiu, J.Foster, A.Solar-Lezama. JSketch: sketching for Java. In FSE, pages 934–937, 2015.
[Prospector’08] D.Mandelin, L.Xu, R.Bodik, D.Kimelman. Jungloid mining: helping to navigate the API jungle. In PLDI, pages 48–61, 2005.
[CodeHint’14] J.Galenson, P.Reames, R.Bodik, B.Hartmann, K. Sen. CodeHint: dynamic and interactive synthesis of code snippets. In ICSE, pages 653–663, 2014.
[SyPet’17] Y.Feng, R.Martins, Y.Wang, I.Dillig, T.Reps. Component-based synthesis for complex apis. In POPL, pages 599–612, 2017.
[Perelman et al.’12] D.Perelman, S.Gulwani, T.Ball, D.Grossman. Type-directed completion of partial expressions. In PLDI, pages 275–286, 2012.
42
![Page 43: EdSketch: Execution-Driven Sketching for Java](https://reader034.vdocuments.us/reader034/viewer/2022051710/5a6d408a7f8b9af8418b5175/html5/thumbnails/43.jpg)
Reference - Program Synthesis (contd.)[InSynth’10] V.Kuncak, M.Mayer, R.Piskac, P.Suter. Complete functional synthesis. In PLDI, pages 316–329, 2010.
[Slang’14] V.Raychev, M.Vechev, E.Yahav. Code completion with statistical language models. In PLDI, pages 419– 428, 2014.
[Bodik et al.’10] R.Bodik, S.Barman, S.Chandra, J.Galenson, D.Kimelman, C.Rodarmor, and N.Tung. Programming with angelic nondeterminism. In POPL, pages 339–352, 2010.
43