taming concurrency: a program verification perspective shaz qadeer microsoft research
TRANSCRIPT
![Page 1: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/1.jpg)
Taming Concurrency:A Program Verification Perspective
Shaz Qadeer
Microsoft Research
![Page 2: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/2.jpg)
Reliable concurrent software?• Correctness problem
– does program behave correctly for all inputs and all interleavings?
• Bugs due to concurrency are insidious – non-deterministic, timing dependent– data corruption, crashes– difficult to detect, reproduce, eliminate
![Page 3: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/3.jpg)
P satisfies S
Undecidable problem!
Why is verification of concurrent programs more difficult than verification of sequential programs?
![Page 4: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/4.jpg)
P satisfies S
Assertions: Provide contracts to decompose problem into a collection of decidable problems• pre-condition and post-condition for each procedure• loop invariant for each loop
Abstractions: Provide an abstraction of the program for which verification is decidable• Finite-state systems
• finite automata• Infinite-state systems
• pushdown automata, counter automata, timed automata, Petri nets
![Page 5: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/5.jpg)
Interference
int t;
t := x;
t := t + 1;
x := t;
Correct
pre x = 0;
post x = 1;
![Page 6: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/6.jpg)
Interference
int t;
t := x;
t := t + 1;
x := t;
Incorrect!
pre x = 0;
post x = 2;
int t;
t := x;
t := t + 1;
x := t;
A B
![Page 7: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/7.jpg)
Controlling interference
int t;
acquire(l);
t := x;
t := t + 1;
x := t;
release(l);
Correct!
pre x = 0;
post x = 2;
int t;
acquire(l);
t := x;
t := t + 1;
x := t;
release(l);
A B
![Page 8: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/8.jpg)
Interference makes program verification difficult
• Annotation explosion with the assertion approach
• State explosion with the abstraction approach
![Page 9: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/9.jpg)
Annotation explosion
• For sequential programs– assertion for each loop – assertion refers only to variables in scope
• For concurrent programs– assertion for each control location– assertion may need to refer to private state of
other threads
![Page 10: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/10.jpg)
State explosion
Sequential Concurrent
Finite-statesystems
Pushdownsystems
P.G
(P.G)3
P = # of program locationsG = # of global states
PSPACEcomplete
Undecidable
n = # of threads
![Page 11: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/11.jpg)
Taming interference
• Atomicity for combating annotation explosion
• Interference-bounding for combating state explosion
![Page 12: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/12.jpg)
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}
![Page 13: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/13.jpg)
Atomicity violation in StringBuffer (Flanagan-Q, PLDI 03)
public final class StringBuffer { private int count; private char[ ] value; . . public synchronized StringBuffer append (StringBuffer sb) { if (sb == null) sb = NULL; int len = sb.length( ); int newcount = count + len; if (newcount > value.length) expandCapacity(newcount); sb.getChars(0, len, value, count); //use of stale len !! count = newcount; return this; }
public synchronized int length( ) { return count; }
public synchronized void getChars(. . .) { . . . }}
![Page 14: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/14.jpg)
Inadequate atomicity is a good predictor ofundesirable interference!• compared to data races
Numerous tools for detecting atomicity violations• static (Type systems, ESPC, QED)• dynamic (Atomizer, AVIO, AtomAid, Velodrome, …)
Significant effort cutting across communities • architecture and operating systems• programming languages and compilers • testing and formal methods
![Page 15: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/15.jpg)
Definition of atomicity
• deposit is atomic if for every non-serialized execution, there is a serialized execution with the same behavior
acq(l) r=bal bal=r+n rel(l)x y z
Serialized execution of deposit
acq(l) r=bal bal=r+n rel(l)x y z
acq(l) r=bal bal=r+n rel(l)x y z
Non-serialized executions of deposit
![Page 16: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/16.jpg)
S0 S1 S2 S3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)x y z
S0 T1 T2 T3 S4 S7T6S5
r=bal bal=r+n rel(l)x y zacq(l)
S0 T1 S2 T3 S4 S7S6S5
y r=bal bal=r+n rel(l)x acq(l) z
S0 T1 T2 T3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)x y z
S0 S1 S2 T3 S4 S7S6S5
acq(l) r=bal bal=r+n rel(l)y zx
Reduction (Lipton, CACM 75)
![Page 17: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/17.jpg)
B: both right + left commutes– variable access holding lock
A: atomic action, non-commuting – access unprotected variable
Four atomicities
L: left commutes– lock release
R: right commutes– lock acquire
![Page 18: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/18.jpg)
S0. S5
R* A L*x Y. . .
S0. S5
R* A L*x Y. . .
Sequential composition Theorem: Sequence (R+B)*;(A+); (L+B)* is atomic
CCCCCCCCCAAACCCLLLCARARRCARLBBCARLB;
; A
R;A;L ; R;A;L
C
A
; A
R; B ; A; L
A
R
![Page 19: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/19.jpg)
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l);}
RBBL
A RBLB
A
ARBL
C
Incorrect!
![Page 20: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/20.jpg)
Critical_Section l;/*# guarded_by l */int balance;
Bank account
/*# atomic */void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l);}
/*# atomic */int read( ) { int r; acquire(l); r = balance; release(l); return r;}
/*# atomic */void withdraw(int x) { acquire(l); int r = balance; balance = r – x; release(l);}
RBBL
A RBLB
A
RBBL
A
Correct!
![Page 21: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/21.jpg)
Taming interference
• Atomicity for combating annotation explosion
• Interference-bounding for combating state explosion
![Page 22: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/22.jpg)
Interference bounding
• An approach to the state-explosion problem
• Explore all executions with a bounded amount of interference– increase the interference bound iteratively
• Good idea if low interference bound– can be exploited by algorithms– is enough to expose bugs
![Page 23: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/23.jpg)
Context-bounded verification [Wu-Q, PLDI 04] [Rehof-Q, TACAS 05]
Context Context Context
Context switch Context switch
• Interference proportional to number of context switches
• Explore all executions with few context switches• Unbounded computation within each context
– Different from bounded model checking
![Page 24: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/24.jpg)
Context-bounding today
• CHESS [Musuvathi-Q, PLDI 07]• JMoped [Suwimonteerabuth et al., SPIN
08]• SPIN for multithreaded C programs [Zaks-
Joshi, SPIN 08]• CBA [Lal-Reps, CAV 08]• Static Driver Verifier
![Page 25: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/25.jpg)
Testing concurrent programs is HARD
• Bugs hidden in rare thread interleavings
• Today, concurrency testing = stress testing– Poor coverage of interleavings– Unpredictable coverage results in
“Heisenbugs”
• The mark of reliability of the system still remains its ability to withstand stress
![Page 26: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/26.jpg)
CHESS in a nutshell
• Replace the OS scheduler with a demonic scheduler
• Systematically explore all scheduling choices
ConcurrentProgram
Win32 API
Kernel SchedulerDemonicScheduler
![Page 27: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/27.jpg)
CHESS architecture
Kernel: Threads, Scheduler, Synchronization Objects
While(not done) { TestScenario()}
TestScenario() { …}
Program
CHESSCHESS runs the scenario in a loop • Each run takes different interleaving• Each run is repeatable
Win32 API
Intercept synchronization and threading calls• To control and introduce
nondeterminismDetect• Assertion violations• Deadlocks• Dataraces• Livelocks
![Page 28: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/28.jpg)
CHESS methodology generalizes
• CHESS works for– Unmanaged programs (written in C, C++)– Managed programs (written in C#,…)– Singularity applications
• With appropriate wrappers, can work for Java, Linux applications
.NETProgram
.NET CLR
CHESSWin32
Program
Win32 / OS
CHESSSingularityProgram
Singularity
CHESS
![Page 29: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/29.jpg)
CHESS: Systematic testing for concurrency
Kernel: Threads, Scheduler, Synchronization Objects
While(not done){ TestScenario()}
TestScenario(){ …}
Program
CHESS
Win32 API
CHESS runs the scenario in a loop • Each run is a different interleaving• Each run is repeatable
![Page 30: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/30.jpg)
x = 1; … … … … … x = k;
State-space explosion
x = 1; … … … … …x = k;
…
n threads
k steps each
• Number of executions
= O( nnk )
• Exponential in both n and k– Typically: n < 10 k > 100
• Limits scalability to large programs
Goal: Scale CHESS to large programs (large k)
Thread 1 Thread n
![Page 31: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/31.jpg)
x = 1;if (p != 0) { x = p->f;}
Preemption-boundingPrioritize executions with small number of preemptions
Two kinds of context switches: Preemptions – forced by the scheduler
e.g. Time-slice expiration
Non-preemptions – a thread voluntarily yieldse.g. Blocking on an unavailable lock, thread end
x = p->f;}
x = 1;if (p != 0) {
p = 0;
preemption
non-preemption
Thread 1 Thread 2
![Page 32: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/32.jpg)
Preemption-bounding in CHESS• The scheduler has a budget of c preemptions
– Nondeterministically choose the preemption points
• Resort to non-preemptive scheduling after c preemptions
• Once all executions explored with c preemptions– Try with c+1 preemptions
![Page 33: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/33.jpg)
x = 1; … … … …
x = 1; … … …
…x = k;
Property 1: Polynomial bound• Terminating program with fixed inputs and deterministic threads
– n threads, k steps each, c preemptions
• Number of executions <= nkCc . (n+c)!
= O( (n2k)c. n! )
Exponential in n and c, but not in k
x = 1; … … … … …x = k;
x = 1; … … … … … x = k;
… …
x = k;
• Choose c preemption points
• Permute n+c atomic blocks
Thread 1 Thread 2
![Page 34: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/34.jpg)
Property 2: Simple error traces• Finds smallest number of preemptions to the
error
• Number of preemptions better metric of error complexity than execution length
![Page 35: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/35.jpg)
Property 3: Coverage metric• If search terminates with preemption-
bound of c, then any remaining error must require at least c+1 preemptions
• Intuitive estimate for– The complexity of the bugs remaining in
the program– The chance of their occurrence in practice
![Page 36: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/36.jpg)
Property 4: Many bugs with 2 preemptions
Program Lines of code
Bugs
Work Stealing Q 4K 4
CDS 6K 1
CCR 9K 3
ConcRT 16K 4
Dryad 18K 7
APE 19K 4
STM 20K 2
TPL 24K 9
PLINQ 24K 1
Singularity 175K 2
37 (total)
Acknowledgement: testers from PCP team
![Page 37: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/37.jpg)
Coverage vs. Preemption-bound
![Page 38: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/38.jpg)
CHESS status
• Being used by testers in many Microsoft product groups
• Demo and session at Professional Developer Conference 2008
• CHESS for Win32 – http://research.microsoft.com/projects/chess
• CHESS for .NET– Coming soon
![Page 39: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/39.jpg)
So far …
• Interference makes program verification difficult
• Taming interference– atomicity – interference-bounding
![Page 40: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/40.jpg)
Whither next?
• Concurrent programs as a composition of modules implementing atomic actions
• Inteference-bounding for general concurrent systems
• Symbolic context-bounded verification
![Page 41: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/41.jpg)
Structuring concurrent programs• In addition to atomicity, we need
– behavioral abstraction analogous to pre/post-conditions for sequential code fragments
– intuitive and simple contract language
• A calculus of atomic actions (Elmas-Tasiran-Q, POPL 08)– QED verifier
![Page 42: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/42.jpg)
Interference-bounding for general concurrent systems
• Task = unit-of-work• Shared-memory systems
– Task (usually) corresponds to a thread• Message-passing systems
– Task corresponds to a sequence of messages• Need linguistic support for the task
abstraction in software models and implementations
![Page 43: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/43.jpg)
Context-bounded reachability is NP-complete
Unbounded Context-bounded
Finite-statesystems
Pushdownsystems
PSPACEcomplete
NP-complete
Undecidable NP-complete
P = # of program locationsG = # of global states
n = # of threadsc = # of context switches
![Page 44: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/44.jpg)
Symbolic context-bounded verification
• The transition relations of tasks are usually encoded symbolically– reachability analysis is PSPACE-complete even for a
single task
• Scalable tools for single symbolic task– Bebop, Moped, Getafix
• Can we extend these techniques to deal with multiple tasks?– Lal and Reps (TACAS 2008, CAV 2008)– Suwimonteerabuth et al. (SPIN 2008)
![Page 45: Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649dab5503460f94a99a47/html5/thumbnails/45.jpg)
Questions?