symdiff: leveraging program verification for comparing programs shuvendu lahiri research in software...
TRANSCRIPT
SymDiff: Leveraging Program Verification for Comparing Programs
Shuvendu LahiriResearch in Software Engineering (RiSE),
Microsoft Research, Redmond
Jointly withChris Hawblitzel (Microsoft Research, Redmond), Ming
Kawaguchi (UCSD), Henrique Rebelo (UPFE)
VSSE Workshop, 2012
Motivation
Microsoft Confidential
Ensuring compatibility– Programmers spend a large fraction of their time
ensuring (read praying) compatibility after changes
Does my bug-fix introduce a regression?
Does the refactoring change any observable
behavior?
How does the feature addition impact existing
features?
Compatibility: applicationsf() { Print(foo); g(); }
g() { ... Print(foo); }
g() { ... Print(foo); Print(bar); }
New featuresRefactoring
CompilersLibrary APIchanges
Bug fixes
Version Control
Compatibility: Microsoft
• Products– Windows APIs (Win32, ntdll)– Driver development kits– .NET frameworks, Base class library– Compilers (C#, JIT,…)– …..
• Windows updates– Security patches– Bug fixes
Every developer/tester
/auditor
Problem
• Use static analysis to–Improve the productivity of users trying
to ensure compatibility across program changes
• Potential benefits–Agility: fewer regressions, higher confidence in
changes, smarter code review, ..
Challenge
• Equivalence checking is too strong a spec– Most changes modify behavior
• Hard to formalize (separate expected changes from unexpected changes)– Refactoring behaviors intact– Bug fix non-buggy behaviors intact– Feature add existing feature behaviors intact– API change ??– Data change ??– Config changes ??– …
Challenge Opportunity
• Hard to formalize (separate expected changes from unexpected changes)– Refactoring behaviors intact– Bug fix non-buggy behaviors intact– Feature add existing feature behaviors intact– …….
Highlight “unexpected”
changes
Microsoft Confidential
Our approach– Provide a tool for performing semantic diff (diff over
behaviors)
Semantic Diff
Does my bug-fix introduce a regression?
Does the refactoring change any observable
behavior?
How does the feature addition impact existing
features?
Microsoft Confidential
Our approach– Provide a tool for performing semantic diff (diff over
behaviors)
Semantic Diff
Does my bug-fix
introduce a regression?
Does the refactoring change any observable behavior?
How does the feature addition
impact existing features?
What is SymDiff?
A framework to– Leverage and extend program
verification for providing relative correctness
Overview
• Demo• Semantic diff– Tool (in current form)
• An application – Compiler compatibility
• Making SymDiff extensible with contracts– Users can express “expected” changes– Mutual summaries and relative termination
Demo
1. Eval (bug1)2. Eval (func)3. StringCopy (bug fix)4. Recursive example
SymDiff tool
SymDiff
– Apply and extend program verification techniques towards comparing programs–Current form: Checks input/output
partial equivalence
[CAV ’12 tool paper]
SymDiff tool: language independent
SymDiff(Boogie+
Z3)
P1
P2P2≠P1
P1 = P2
Works at Boogie intermediate language
S1C/.NET/
x86/ARM
Boogie
S2C/.NET/
x86/ARM
Boogie
Boogie
• Simple intermediate verification language– [Barnett et al. FMCO’05]
• Commands– x := E //assign– havoc x //change x to an arbitrary value– assert E //if E holds, skip; otherwise, go wrong– assume E // if E holds, skip; otherwise, block– S ; T //execute S, then T– goto L1, L2, … Ln //non-deterministic jump to labels – call x := Foo(e1,e2,..) //procedure call
Boogie (contd.)
• Two types of expressions– Scalars (bool, int, ref, ..)– Arrays ([int]int, [ref]ref, …)
• Array expression sugar for SMT array theory– x[i] := y x := upd(x, i, y)– y := x[i] y := sel(x,i)
• Procedure calls sugar for modular specification
call Foo();assert pre;havoc V;assume post;
procedure Foo();requires pre;ensures post;modifies V;
Basic equivalence checkingvoid swap1(ref int x, ref int y){ int z = x; x = y; y = z;}void swap2(ref int x, ref int y){ x = x + y; y = x - y; x = x - y;}
z0 == x0 &&x1 == y0 &&y1 == z0 &&swap1.x == x1 && swap1.y == y1&&x1' == x0 + y0 &&y1' == x1' – y0 &&x2' == x1' – y1' &&swap2.x == x2' && swap2.y == y1' &&~ (swap1.x == swap2.x && swap1.y == swap2.y)
Z3theorem prover
UNSAT (Equivalent)
SAT (Counterexample)
Handling procedure calls
• Modular checking– Assume “matched” callees are deterministic and
have the same I/O behaviors– Modeled by uninterpreted functions [Necula ‘00, …,
Godlin & Strichman ‘08, …..]• Addition of postcondition for Foo, Foo’
modifies g;free ensures g == UF_Foo_g(x, old(g));free ensures ret == UF_Foo_ret(x, old(g));procedure Foo(x) returns (ret);
modifies g;free ensures g == UF_Foo_g(x, old(g));free ensures ret == UF_Foo_ret(x, old(g));procedure Foo’(x) returns (ret);
Modeling C/Java/C#/x86 Boogie
• Separation of concerns– Front end can be developed independently– Quite a few already exists • HAVOC/VCC for C, Spec#/BCT for .NET, ?? for Java, …
• Heap usually modeled by arrays– x.f := y Heap_f[x] := y
• Challenges– Deterministic modeling of I/O, malloc, …..– The entire heap is passed around
Application: Compiler compatibility
Microsoft Confidential
Compiler validation
X86
ARM
ARM+opt
Source
v1 v2 v3
Versions
X86+opt
v4
Compatibility: x86 vs. x86 example
G01: mov EAX, EDX
G02: and EAX, 255 push EAX mov EDX, 0x100000 call WriteInternalFlag2(int,bool)
__epilog: ret
G01: push ESI mov ESI, EDX
G02: and ESI, 255 push ESI mov EDX, 0x100000 call WriteInternalFlag2(int,bool)
G03: pop ESI ret
v2 v3
X86+opt
254
Large x86 vs. ARM example
Beyond equivalence
Beyond equivalenceType of change Check
Refactoring/Optimizations In1 = In2 Out1’ = Out2’
Bug fix In1 = In2 (Fail1’ || Out1’ = Out2’)
Feature addition In1 = In2 (UnImplemented1’ || Out1’ = Out2’)
Performance optimization In1 = In2 (Measure2’ <= Measure1’)
Differential assertion checking (DAC) (see POPL’12 on “Interleaved Bugs ….”)
In1 = In2 (Fail1’ || ~Fail2’)
Contracts over two programs
• Need an extensible contract mechanism for comparing two programs– Generalization of pre/post conditions
• Why– Allow users to express relative correctness
specifications (e.g. conditional equivalence)– Automated methods may not always suffice (even for
equivalence checking)• Challenge– Should be able to leverage SMT-based program verifiers
Mutual summaries
–A extensible framework for interprocedural program comparison
• Prior work (mostly automated):– Intraprocedural• Translation validation [Pnueli et al. ‘98, Necula ‘00, Zuck
et al. ’05,…]
– Coarse intraprocedural (only track equalities)• Regression verification [Strichman et al. ‘08]
Mutual summaries
– [MSR-TR-2011-112]
• Mutual summaries (MS)• Relative termination (RT)• Dealing with loops and unstructured goto
Example: Feature additionint f1(int x1){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x1] == 0) return Val[x1]; else if (Op[x1] == 1) return f1(a1) + f1(a2); else if (Op[x1] == 2) return f1(a1) - f1(a2); else return 0;}
int f2(int x2, bool isU){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x2] == 0) return Val[x2]; else if (Op[x2] == 1){ if (isU) return uAdd(f2(a1, T), f2(a2, T)); else return f2(a1, F) + f2(a2, F); } else if (Op[x2] == 2){ if (isU) return uSub(f2(a1, T), f2(a2, T)); else return f2(a1, F) – f2(a2, F); } else return 0;}
The programs are equivalent when
isU == False
Mutual summaries
• What is a mutual summary MS(F1, F2)? – An formula over two copies of • parameters, globals (g), returns and next state of globals
(g’)
void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); }}
void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); }}
MS(F1, F2): (x1 = x2 && g1 <= g2 && x1 >= 0) ==> g1’ <= g2’
Mutual summaries
• What does a mutual summary MS(F1, F2) mean?– For any pre/post state pairs (s1,t1) of F1, and (s2,t2)
of F2, (s1,t1,s2,t2) satisfy MS(F1,F2)
void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); }}
void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); }}
MS(F1, F2): (x1 = x2 && g1 <= g2 && x1 >= 0) ==> g1’ <= g2’
Exampleint f1(int x1){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x1] == 0) return Val[x1]; else if (Op[x1] == 1) return f1(a1) + f1(a2); else if (Op[x1] == 2) return f1(a1) - f1(a2); else return 0;}
int f2(int x2, bool isU){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x2] == 0) return Val[x2]; else if (Op[x2] == 1){ if (isU) return uAdd(f2(a1, T), f2(a2, T)); else return f2(a1, F) + f2(a2, F); } else if (Op[x2] == 2){ if (isU) return uSub(f2(a1, T), f2(a2, T)); else return f2(a1, F) – f2(a2, F); } else return 0;}
MS(f1, f2) = (x1 == x2 && !isU) ==> ret1 == ret2
Checking mutual summaries
• Given F1, F2, MS(F1, F2), define the following procedure:
void CheckMS_F1_F2(int x1, int x2){inline F1(x1);inline F2(x2);
assert MS(F1,F2);}
Modular checking: Instrumentation
1. Add “summary relations” R_F1, and R_F2
void F1(int x1);ensures R_F1(x1, old(g1)/g1, g1/g1’);
2. Use the summary relations to assume mutual summaries at call sites:
axiom (forall x1, g1, g1’, x2, g2, g2’:: {R_F1(x1, g1, g1’), R_F2(x2, g2, g2’)}(R_F1(x1, g1, g1’) && R_F2(x2, g2, g2’)) ==>MS_F1_F2(x1, g1, g1’, x2, g2, g2’));
Leveraging program verifiers
• Mutual Summary checking– Encode using contracts (postconditions), axioms– Verification condition generation (Boogie)– Checking using SMT solver (Z3)
• Next steps– Inferring the mutual summaries
Relative termination
• Specification relating the terminating behaviors of P2 wrt P1
• Not just for proving termination– Required for composing transformations– MS1(f,f’) && MS2(f’,f’’) (MS1 MS2) (f,f’’)– E.g. P_Eq(f,f’) && P_Eq(f’,f’’) P_Eq(f,f’’)
Relative termination condition
• What is a relative termination condition RT(F1, F2)? – An formula over two copies of
• parameters, globals (g)
void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); }}
void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); }}
RT(F1, F2): (x1 <= x2)
Relative termination condition
• What does relative termination condition RT(F1, F2) mean? – For pair of inputs states (s1,s2), if F1 terminates on s1,
and (s1,s2) satisfies RT(F1,F2), then F2 terminates on s2
void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); }}
void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); }}
RT(F1, F2): (x1 <= x2)
What about loops? int Foo2() { i = 0; if (n > 0) { t = g; v = 3; do2: a[i] := v; i := i + 1; v := v + t; While2: //FLABEL if (i < n) goto do2; } return i;}
int Foo2() { i = 0; if (n > 0) { t = g; v = 3; do2: a[i] := v; i := i + 1; v := v + t; return While2(i, t, v); } return i;}
(int ,int) While2(i2, t2, v2) {i2' := i2;v2' := v2;if (i2' < n) { a2[i2'] := v2'; i2' := i2' + 1; v2' := v2' + t2; return While2(i2', t2,v2');}return (i2‘,v2’);
}
Unrolling optimizationsvoid F2(int i2){ if (i2 < n) { a2[i2] = 1; F2(i2+1); return; } return;}
void F3(int i3){ if (i3 + 1 < n) { a3[i3] := 1; a3[i3+1] := 1; F3(i3+2); return; } if (i3 < n) a3[i3] := 1; return;}
Extra step• Inline F2 once inside F2 to “match up” with F3
MS(F2, F3) = (i2 == i3 && a2 == a3) ==> a2’ == a3’
Using mutual summaries
• Flow1. Specify the FLABELS to remove loops and gotos
into procedures2. Write mutual summaries for pairs of resulting
procedures3. Specify the inlining limit (if needed)
Express translation validation proofs of many compiler optimizations
– Copy propagation – Constant propagation – Common sub-expression
elimination – Partial redundancy
elimination – Loop invariant code
hoisting – Conditional speculation – Speculation
– Software pipelining – Loop unswitching – Loop unrolling – Loop peeling – Loop splitting – Loop alignment – Loop interchange – Loop reversal – Loop skewing – Loop fusion – Loop distribution
[Kundu, Tatlock, Lerner ‘09]
Order of updates differ in two versions
A nice example that uses MS, RT
void A(ref x){ if(x != nil){ A(next[x]); D(x); }}
next: ref ref; data: ref int;
void B(ref x){ if(x != nil){ D(x); B(next[x]); }}
void C(ref x){ ref i := x; if(i != nil){Do: D(i); i := next[i]; if (i != nil)
goto Do; }}
void D(ref x){ data[x] := U(data[x]);}
Recursive Tail-recursive Do-while
Overview
DemoSemantic diff– Tool (in current form)
An application – Compiler compatibility
Making SymDiff extensible with contracts– Mutual summaries and relative termination– General contracts for comparing programs
In summary
• Checking compatibility (statically) is a huge opportunity– Both formalizing the problem– Tools/techniques to solve it
• Likely to have impact on development cycle– Existing static analysis tools has failed to do so cost-
effectively, in spite of all the progress• Combining with dynamic analysis– To generate test cases when possible, or aid testing
achieve higher differential coverage
Resources
• SymDiff website
http://research.microsoft.com/symdiff/
• Binary release soon!– Contains C front end