optimizing the automated programming stackbornholt/papers/thesis-phd19...symbolic profiling...

Optimizing theAutomated Programming StackJames Bornholt University of Washington

Software is everywhere

Bugs are everywhere

Bugs are everywhere

Automated programming tools

VerifierProgram

Specification✅

❌ + test caseLanguage

Automated programming tools

VerifierProgram

Specification✅

❌ + test case

SynthesizerSpecification✅

❌

+ program

Language

Language

Automated programming successes

Verified operating systems

[Nelson et al, SOSP’17]

Synthesized network configs

[McClurg et al, PLDI’15]

Verified SQL optimizers

[Chu et al, VLDB’18]

Synthesized crypto primitives

[Erbsen et al, Oakland’19]

Synthesized biology experiments

[Köksal et al, POPL’13]

Synthesized memory models

[Bornholt et al, PLDI’17]

Synthesized educational models

[Butler et al, VMCAI’18]

Challenges in automated programming

Intractability SpecificationMost problems in automated programming are intractable (many undecidable).

Automated programming requiresa specification, which is often difficult to construct and audit.

Challenges in automated programming

Intractability SpecificationMost problems in automated programming are intractable (many undecidable).

Automated programming requiresa specification, which is often difficult to construct and audit.

Domain specializationSpecialization reduces the size of the search space, eliminating irrelevant programs/behaviors.

Specialization allows for concise and expressive specificationsthat capture programmer intent.

Domain specialization

Automated programming stack

Domain-specific tools

Domain specialization

Automated programming stack

SAT/SMT solving improvements in scale and expressiveness

Domain-specific tools

Domain specialization

Automated programming stack

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Domain-specific tools

Domain specialization

Automated programming stack

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Domain-specific tools

Automated programming stack

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Domain-specific tools

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

MemSynth [PLDI’17]: an automated tool for synthesizing memory consistency models

Ferrite [ASPLOS’16]: a tool for synthesizing crash-safe file system code

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

MemSynth [PLDI’17]: an automated tool for synthesizing memory consistency models

Metasketches [POPL’16]: a strategy abstrac`on for synthesis problems

SymPro [OOPSLA’18]: a technique for systema`cally building scalable tools

Ferrite [ASPLOS’16]: a tool for synthesizing crash-safe file system code

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

MemSynth [PLDI’17]: an automated tool for synthesizing memory consistency models

Metasketches [POPL’16]: a strategy abstrac`on for synthesis problems

SymPro [OOPSLA’18]: a technique for systema`cally building scalable tools

Ferrite [ASPLOS’16]: a tool for synthesizing crash-safe file system code

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello? 3 41 2

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?

3 41 2

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?

3 41 2

1 23 4

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?nothing?

3 41 2

1 23 4

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?nothing?

3 41 2

1 23 4

3 41 2

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?nothing?both?

3 41 2

1 23 4

3 41 2

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?nothing?both?

3 41 2

1 23 4

3 41 2

No! (sequential consistency)

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

Can this print… hello?goodbye?nothing?both?

3 41 2

1 23 4

3 41 2

No! (sequential consistency)

Yeah! We wanna go fast!

All variables initialized to 0

Memory models define the memory ordering behavior of multiprocessors

…correctness of my compiler…

Compiler writers#

…rules to verify against…

Verifica`on tools✅

…possible low-level behaviors…

Kernel/library developers$

Memory models define the memory ordering behavior of multiprocessors

…correctness of my compiler…

Compiler writers#

…rules to verify against…

Verifica`on tools✅

…possible low-level behaviors…

Kernel/library developers$

Litmus tests and prose

Memory models define the memory ordering behavior of multiprocessors

…correctness of my compiler…

Compiler writers#

…rules to verify against…

Verifica`on tools✅

…possible low-level behaviors…

Kernel/library developers$

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Memory models define the memory ordering behavior of multiprocessors

…correctness of my compiler…

Compiler writers#

…rules to verify against…

Verifica`on tools✅

…possible low-level behaviors…

Kernel/library developers$

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

x86 [Sewell et al, CACM’10]

PowerPC [Alglave et al, CAV’10, etc]

ARM [Flur et al, POPL’16]

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Synthesize specifica`ons from litmus tests

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Synthesize specifica`ons from litmus tests

Detect ambigui`es in synthesized models

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Synthesize specifica`ons from litmus tests

Detect ambigui`es in synthesized models

x86: 2 seconds PowerPC: 12 seconds

x86: 4 ambigui`es PowerPC: 9 ambigui`es

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Litmus testsas rela`ons

Memory models as constraints

Synthesis via sketches

1 2 3

Litmus tests as relations

X = 1 Y = 1if Y == 0: print “hello”

if X == 0: print “goodbye”

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Litmus tests as relations

X = 1 Y = 1r0 = Y r1 = X

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Litmus tests as relations

X = 1 Y = 1r0 = Y r1 = X

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Encode programs and behaviors as relations in relational logic(like Alloy)

Litmus tests as relations

X = 1 Y = 1r0 = Y r1 = X

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Encode programs and behaviors as relations in relational logic(like Alloy)

Program relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order: (a,b) ∈ po if b is aoer a on the same thread

Litmus tests as relations

X = 1 Y = 1r0 = Y r1 = X

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Encode programs and behaviors as relations in relational logic(like Alloy)

Program relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order: (a,b) ∈ po if b is aoer a on the same thread

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from: (r,w) ∈ rf if r reads the value wripen by w

Litmus tests as relations

X = 1 Y = 1r0 = Y r1 = X

Thread 1 Thread 2

1

2

3

4

All variables initialized to 0

Encode programs and behaviors as relations in relational logic(like Alloy)

Program relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order: (a,b) ∈ po if b is aoer a on the same thread

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from: (r,w) ∈ rf if r reads the value wripen by w

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

M(T, E) ≜

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

M(T, E) ≜

(&& (&& (&& (in rf (& (-> Writes Reads) (join loc (~ loc)) (join data (~ data)))) (no (- (join rf (~ rf)) iden)) (all ((r4 (- Reads (join Writes rf)))) (= (join r4 data) Zero))) (&& (in ws (& (-> Writes Writes) (join loc (~ loc)))) (no (& iden ws)) (in (join ws ws) ws) (all ((r5 Writes)) (all ((r6 (- (& Writes (join loc (join r5 loc))) r5))) (or (in (-> r5 r6) ws) (in (-> r6 r5) ws)))) (in ws (join loc (~ loc))))) (no (& (^ (+ (+ rf ws (+ (join (~ rf) ws) (& (-> (- Reads (join Writes rf)) Writes) (join loc (~ loc))))) (& po (join loc (~ loc))))) iden)) (all ((r7 Writes)) (=> (&& (in r7 (- (join univ ws) (join ws univ))) (some (join (join r7 loc) finalValue))) (= (join r7 data) (join (join r7 loc) finalValue)))) (no (& (^ (+ (& po dp) ws (+ (join (~ rf) ws) (& (-> (- Reads (join Writes rf)) Writes) (join loc (~ loc)))) (-> none none) (+ (^ (+ (+ (join (:> po Syncs) po) (join (join (:> po Syncs) po) rf)) (join rf (join (:> po Syncs) po)))) (^ (+ (+ (& (join (:> po Lwsyncs) po) (+ (-> Writes Writes) (-> Reads MemoryEvent))) (:> (join rf (& (join (:> po Lwsyncs) po) (+ (-> Writes Writes) (-> Reads MemoryEvent)))) Writes)) (<: Reads (join (& (join (:> po Lwsyncs) po) (+ (-> Writes Writes) (-> Reads MemoryEvent))) rf))))))) iden)))

(no (& (^ (+ po rf)) iden))

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

M(T, E) ≜ (no (& (^ (+ po rf)) iden))

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

M(T, E) ≜ (no (& (^ (+ po rf)) iden))Constraining the possible values of rf…

…by forbidding cycles involving rf ∪ po

Memory models as relational constraintsProgram relations extracted from program text:

po = {( , ), ( , )}1 2 3 4

Program order

Execution relations describe dynamic behavior:

rf = {( , ), ( , )}2 3 4 1

Reads-from

A memory model constrains the allowed executions of a program

Written as a predicate in relational logic

M(T, E) ≜ (no (& (^ (+ po rf)) iden))Constraining the possible values of rf…

…by forbidding cycles involving rf ∪ po

A memory model allows a test T if there exists an execu`on E that sa`sfies the predicate

(no (& (^ (+ po rf)) iden))

Synthesis from a memory model sketch

M(T, E) ≜

(no (& (^ (+ po rf)) iden))

Synthesis from a memory model sketch

M(T, E) ≜ (no (& (^ (+ ?? ??)) iden))Expression holes for a synthesizer to complete

(no (& (^ (+ po rf)) iden))

Synthesis from a memory model sketch

M(T, E) ≜ (no (& (^ (+ ?? ??)) iden))po rf po + rf po & rf po - rf …

Expression holes for a synthesizer to complete

(no (& (^ (+ po rf)) iden))

Synthesis from a memory model sketch

M(T, E) ≜ (no (& (^ (+ ?? ??)) iden))po rf po + rf po & rf po - rf …

Expression holes for a synthesizer to complete

A sketch specifies things we know (e.g., want a happens-before ordering)…

(no (& (^ (+ po rf)) iden))

Synthesis from a memory model sketch

M(T, E) ≜ (no (& (^ (+ ?? ??)) iden))po rf po + rf po & rf po - rf …

Expression holes for a synthesizer to complete

A sketch specifies things we know (e.g., want a happens-before ordering)…

…and defines the shape of the parts we don’t know

Memory model frameworks

(no (& (^ (+ ws rf ppo grf)) iden))

[Alglave et al, CAV’10]

M(T, E) ≜

Preserved program order: same-thread reorderings

Memory model frameworks

(no (& (^ (+ ws rf ppo grf)) iden))

[Alglave et al, CAV’10]

Global reads-from: inter-thread reorderings

M(T, E) ≜

Preserved program order: same-thread reorderings

Memory model frameworks

(no (& (^ (+ ws rf ppo grf)) iden))

[Alglave et al, CAV’10]

Global reads-from: inter-thread reorderings

Sequential consistency

Total storeorder (x86)

po rf

po - (Wr→Rd) rf & SameThd

M(T, E) ≜

Preserved program order: same-thread reorderings

Memory model frameworks

(no (& (^ (+ ws rf ppo grf)) iden))

[Alglave et al, CAV’10]

Global reads-from: inter-thread reorderings

Sequential consistency

Total storeorder (x86)

po rf

po - (Wr→Rd) rf & SameThd

M(T, E) ≜ ?? ??

Ocelot DSL for relational logic with holes

M(T, E) ≜

Ocelot embeds rela`onal logic in the RoseZe solver-aided language [Torlak & Bodik 2014]

Also in use for SQL query synthesis and protocol reasoning

http://ocelot.tools

(no (& (^ (+ ws rf ppo grf)) iden))?? ??

Expression holes for a synthesizer to complete

The synthesis query

Allowed litmus tests

Forbidden litmus testsSynth Completed memory model M

Memory model sketch M̂

The synthesis query

Synth

x86

53

2 allowed tests

1 2 4 6

7 8 9 10

8 forbidden tests

Total store order

Memory model sketch M̂

The synthesis query

Allowed litmus tests

Forbidden litmus testsSynth Completed memory model M

Memory model sketch M̂

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. M allows T

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. M allows T

∃ M. ∀ T∈T-. M forbids T

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. M allows T

∃ M. ∀ T∈T-. M forbids T

Standard exists-forall quan`fier papern for synthesis

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. M allows T

∃ M. ∀ T∈T-. M forbids T

M allows T: ∃ E. M(T, E)Standard exists-forall quan`fier

papern for synthesis

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. ∃ E. M(T,E)

∃ M. ∀ T∈T-. ∀ E. ¬M(T,E)

M allows T: ∃ E. M(T, E)Standard exists-forall quan`fier

papern for synthesis

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. ∃ E. M(T,E)

∃ M. ∀ T∈T-. ∀ E. ¬M(T,E)

M allows T: ∃ E. M(T, E)

Higher-order quan`fica`on over rela`ons! 😱

Standard exists-forall quan`fier papern for synthesis

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. ∃ E. M(T,E)

∃ M. ∀ T∈T-. ∀ E. ¬M(T,E)

M allows T: ∃ E. M(T, E)

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. ∃ E. M(T,E)

∃ M. ∀ T∈T-. ∀ E. ¬M(T,E)

M allows T: ∃ E. M(T, E)

Handled by a quan`fied boolean formula (QBF) solver

The synthesis query

Allowed litmus tests

Forbidden litmus testsMemory model M

Memory model sketch M̂

∃ M. ∀ T∈T+. ∃ E. M(T,E)

∃ M. ∀ T∈T-. ∀ E. ¬M(T,E)

M allows T: ∃ E. M(T, E)

Handled by a quan`fied boolean formula (QBF) solver

Handled by incremental synthesis engine

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T T

T T T T T T T T

T1

T2

T T T T T

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T T

T T T T T T T T

T1 T2

T T T T T

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T T

T T T T T T T T

T1 T2

Handled by a quan`fied boolean formula (QBF) solver

T T T T T

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T T

T T T T T T T T

T1 T2

M’

Handled by a quan`fied boolean formula (QBF) solver

T T T T T

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T T

T T T T T T T T

T1 T2

M’

T

Handled by a quan`fied boolean formula (QBF) solver

Incremental synthesis

Allowed litmus tests

Forbidden litmus tests

Synth T T T T T T T

T T T T T T T T

T1 T2

M’T

Handled by a quan`fied boolean formula (QBF) solver

Incremental synthesis

Allowed litmus tests

Forbidden litmus testsCompleted memory model M

Synth T T T T T T T

T T T T T T T T

T1 T2

M’T

Handled by a quan`fied boolean formula (QBF) solver

Incremental synthesis

Allowed litmus tests

Forbidden litmus testsCompleted memory model M

Synth T T T T T T T

T T T T T T T T

T1 T2

M’T

Empirically, need very few itera`ons to converge

Handled by a quan`fied boolean formula (QBF) solver

Disambiguating synthesized models

Ambig

Key idea: aoer synthesis, is there a different memory model that also explains the input tests?

Disambiguating synthesized models

Allowed litmus tests

Forbidden litmus tests Ambig

Key idea: aoer synthesis, is there a different memory model that also explains the input tests?

Memory model sketch M̂

Disambiguating synthesized models

Allowed litmus tests

Forbidden litmus tests AmbigCompleted memory model M

Key idea: aoer synthesis, is there a different memory model that also explains the input tests?

Memory model sketch M̂

Disambiguating synthesized models

Allowed litmus tests

Forbidden litmus tests Ambig Litmus test T

Completed memory model M

Completed memory model M2

Key idea: aoer synthesis, is there a different memory model that also explains the input tests?

Memory model sketch M̂

Disambiguating synthesized models

Allowed litmus tests

Forbidden litmus tests Ambig Litmus test T

Completed memory model M

Completed memory model M2

Key idea: aoer synthesis, is there a different memory model that also explains the input tests?

Difference between M and M2 is not just syntac`c: they disagree about test T

Memory model sketch M̂

Synthesizing existing memory models

x86

PowerPC

Synthesizing existing memory models

x86

PowerPC 768 tests[Alglave et al, CAV’10]

10 tests

Synthesis

Synthesizing existing memory models

x86

PowerPC 768 tests[Alglave et al, CAV’10]

10 tests ✓ 2 seconds

✓ 12 seconds

Synthesis

Synthesizing existing memory models

x86

PowerPC 768 tests[Alglave et al, CAV’10]

10 tests ✓ 2 seconds

✓ 12 seconds

Not equivalent to TSO!

Synthesis

Synthesizing existing memory models

x86

PowerPC 768 tests[Alglave et al, CAV’10]

10 tests ✓ 2 seconds

✓ 12 secondsNot equivalent to published model!

Not equivalent to TSO!

Synthesis

Synthesizing existing memory models

x86

PowerPC 768 tests[Alglave et al, CAV’10]

10 tests ✓ 2 seconds

✓ 12 secondsNot equivalent to published model!

4 new tests

9 new tests

Ambiguity

Not equivalent to TSO!

mfence, xchg

sync, lwsync

MemSynth: automated programming for memory consistency models

Litmus tests and prose

∀∃ ∈

∧

∨

∩∪⊂

⋈⇒

Formalspecifica`ons

Synthesize specifica`ons from litmus tests

Detect ambigui`es in synthesized models

x86: 2 seconds PowerPC: 12 seconds

x86: 4 ambigui`es PowerPC: 9 ambigui`es

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Scaling a synthesis tool is hard work

12 seconds

Scaling a synthesis tool is hard workTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

Scaling a synthesis tool is hard workTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

3 hours

Scaling a synthesis tool is hard workTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

3 hours Finding these op`miza`on opportuni`es is the key to good performance and new func`onality

Symbolic profilingTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

3 hours A symbolic profiler identifies optimization opportunities in an automated tool.

Symbolic profilingTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

3 hours A symbolic profiler identifies optimization opportunities in an automated tool.

What makes scaling an automated programming tool hard?

How does symbolic profiling work?

How effective is symbolic profiling?

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

‘()

¬(even? 3)

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

‘()

‘(6)

¬(even? 3)

(even? 6)

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

‘()

‘(6)

‘(6 8)

¬(even? 3)

(even? 6)

(even? 8)

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

‘()

‘(6)

‘(6 8)

¬(even? 3)

(even? 6)

‘(6 8 2)

(even? 8)

(even? 2)

Symbolic evaluation executes all paths through a program

(filter even? ‘(3 6 8 2 ))

‘()

‘()

‘(6)

‘(6 8)

¬(even? 3)

(even? 6)

‘(6 8 2)

(even? 8)

(even? 2)

Does this expression always return only even numbers?

Values of list elements are unknown (e.g., verifying filter for all inputs)

Symbolic evaluation executes all paths through a program

(filter even? ‘(x0 x1 x2 x3))

‘()

‘()

‘(6)

‘(6 8)

¬(even? 3)

(even? 6)

‘(6 8 2)

(even? 8)

(even? 2)

Does this expression always return only even numbers?

Values of list elements are unknown (e.g., verifying filter for all inputs)

Symbolic evaluation executes all paths through a program

(filter even? ‘(x0 x1 x2 x3))

‘()

‘()

‘(x1)

‘(x1 x2)

¬(even? x0)

(even? x1)

‘(x0)

‘() ‘(x0) ‘(x0 x1)

‘(x2) ‘(x1) ‘(x0) ‘(x0 x2) ‘(x0 x1)

‘(x1 x2 x3)‘(x1 x3)‘(x1 x2)‘(x1)‘(x2 x3) ‘(x0)‘(x0 x3)‘(x0 x2)‘(x0 x2 x3)‘(x0 x3)

(even? x2)

(even? x3)

Does this expression always return only even numbers?

Values of list elements are unknown (e.g., verifying filter for all inputs)

Symbolic evaluation executes all paths through a program

(filter even? ‘(x0 x1 x2 x3))

‘()

‘()

‘(x1)

‘(x1 x2)

¬(even? x0)

(even? x1)

‘(x0)

‘() ‘(x0) ‘(x0 x1)

‘(x2) ‘(x1) ‘(x0) ‘(x0 x2) ‘(x0 x1)

‘(x1 x2 x3)‘(x1 x3)‘(x1 x2)‘(x1)‘(x2 x3) ‘(x0)‘(x0 x3)‘(x0 x2)‘(x0 x2 x3)‘(x0 x3)

(even? x2)

(even? x3)

Does this expression always return only even numbers?

(length )

Values of list elements are unknown (e.g., verifying filter for all inputs)

Symbolic evaluation executes all paths through a program

(filter even? ‘(x0 x1 x2 x3))

‘()

‘()

‘(x1)

‘(x1 x2)

¬(even? x0)

(even? x1)

‘(x0)

‘() ‘(x0) ‘(x0 x1)

‘(x2) ‘(x1) ‘(x0) ‘(x0 x2) ‘(x0 x1)

‘(x1 x2 x3)‘(x1 x3)‘(x1 x2)‘(x1)‘(x2 x3) ‘(x0)‘(x0 x3)‘(x0 x2)‘(x0 x2 x3)‘(x0 x3)

(even? x2)

(even? x3)

Does this expression always return only even numbers?

(length )

2 2 3 1 2 2 3 212

Symbolic evaluation techniques

Symbolic execution

Bounded model checking

Always fork into independent paths (more paths, but more concrete)

Merge aoer every fork (fewer paths, but less concrete)

Symbolic evaluation techniques

Symbolic execution

Bounded model checking

Always fork into independent paths (more paths, but more concrete)

Merge aoer every fork (fewer paths, but less concrete)

Rosepe [Torlak & Bodik 2014]Jalangi [Sen et al 2014]

Crucible [Galois, Inc.]

Symbolic evaluation techniques

Symbolic execution

Bounded model checking

Always fork into independent paths (more paths, but more concrete)

Merge aoer every fork (fewer paths, but less concrete)

Rosepe [Torlak & Bodik 2014]Jalangi [Sen et al 2014]

Crucible [Galois, Inc.]

Controlling the trade-off between these strategies is key to good scalability

Two data structures to summarize symbolic evaluation

‘()

‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0)

(even? x1)¬(even? x1) (even? x1)¬(even? x1)

(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

Symbolic evalua\on graph Reflects the evaluator’s strategyfor all-paths execu`on of the program

Symbolic heap Shape of all symbolic valuescreated by the program

Two data structures to summarize symbolic evaluation

‘()

‘() ‘(x0)

‘() ‘(x1) ‘(x0) ‘(x0 x1)

¬(even? x0) (even? x0)

(even? x1)¬(even? x1) (even? x1)¬(even? x1)

(even? x0)

∧

¬

∧∧ ∧

¬

(even? x1)

Symbolic evalua\on graph Reflects the evaluator’s strategyfor all-paths execu`on of the program

Symbolic heap Shape of all symbolic valuescreated by the program

Any symbolic evalua\on technique can be summarized by these two data structures

Analyzing symbolic data structures

For each procedure, measure metrics that summarize the evolu`on of the symbolic evalua`on graph and symbolic heap

Summarize metrics as a score to rank procedures in the program

Analyzing symbolic data structures

For each procedure, measure metrics that summarize the evolu`on of the symbolic evalua`on graph and symbolic heap

Summarize metrics as a score to rank procedures in the program

Analyzing symbolic data structures

The most likely bopleneck is not the slowest procedure

The most likely bopleneck is not the slowest procedure

Three symbolic profilersWe developed two implementa`ons:

• The RoseZe solver-aided language (Racket)

• The Jalangi dynamic analysis framework (JavaScript)

Since publica`on, based on our work:

• The Crucible symbolic simula`on library (C, Java, …) by Galois

Three symbolic profilersWe developed two implementa`ons:

• The RoseZe solver-aided language (Racket)

• The Jalangi dynamic analysis framework (JavaScript)

Since publica`on, based on our work:

• The Crucible symbolic simula`on library (C, Java, …) by Galois

Today

Symbolic profiling in practice

Refinement type checker for Ruby [VMCAI’18] 6× speedupCryptographic protocol verifier [FM’18] 29× speedupSQL query verifier [CIDR’17] 75× speedupSafety-critical radiotherapy system verifier [CAV’16] 290× speedup

Case studies: fixed 8 performance issues in 15 Rosette tools

Symbolic profiling in practice

Refinement type checker for Ruby [VMCAI’18] 6× speedupCryptographic protocol verifier [FM’18] 29× speedupSQL query verifier [CIDR’17] 75× speedupSafety-critical radiotherapy system verifier [CAV’16] 290× speedup

Used in produc`on at the UW Medical Center

Case studies: fixed 8 performance issues in 15 Rosette tools

Symbolic profiling in practice

Refinement type checker for Ruby [VMCAI’18] 6× speedupCryptographic protocol verifier [FM’18] 29× speedupSQL query verifier [CIDR’17] 75× speedupSafety-critical radiotherapy system verifier [CAV’16] 290× speedup

Case studies: fixed 8 performance issues in 15 Rosette tools

User study: 8 Rosette users tasked with finding known performance issues in 4 programs

Users solved every task more quickly when they had access to symbolic profiling

6 failures without symbolic profiling vs. none with it

Symbolic profilingTi

me

(sec

s)

0

3000

6000

9000

12000

January February March April May June

12 seconds

3 hours A symbolic profiler identifies optimization opportunities in an automated tool.

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Automated tools are worth building The case of memory models [PLDI’17]

Building them can be made systematic Symbolic profiling [OOPSLA’18]

The future is more automation Automating the automated programming stack

Automated programming abstractions

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Metasketches [POPL’16]

Symbolic profiling [OOPSLA’18]

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Diagnosing SMT solver behavior

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Metasketches [POPL’16]

Symbolic profiling [OOPSLA’18]

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness ?

Diagnosing SMT solver behavior

SAT/SMT solving improvements in scale and expressiveness

Diagnosing SMT solver behavior

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Metasketches [POPL’16]

Symbolic profiling [OOPSLA’18]

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Solver profiling

Self-optimizing automated tools

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Metasketches [POPL’16]

Symbolic profiling [OOPSLA’18]

Solver-aided languages front-end abstractions for verification/synthesis

Symbolic evaluation algorithms to translate programs to SAT/SMT

SAT/SMT solving improvements in scale and expressiveness

Solver profiling

Exploit this profiling data for profile-guided op`miza`on

Application opportunities

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Hardware accelerator design/programming

High-performancelow-precision kernels

2

4 . . .

3

5 ⇥. . .

⇤

<latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit>

File systems [ASPLOS’16, OSDI’16]

Operating systems [SOSP’17, OSDI’18]

Memory models [PLDI’17]

Metasketches [POPL’16]

Symbolic profiling [OOPSLA’18]

Solver profiling

Hardware accelerator design/programming

High-performancelow-precision kernels

2

4 . . .

3

5 ⇥. . .

⇤

<latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit><latexit sha1_base64="82XuOOPTGgCFx506OuCN1kMU+88=">AAAHl3icdVXfj9NGEPZBuUBa4ICnqi9WA4inKM4BOXQPICpV9A2qHiBlo9N6M05Wtz/M7hgSWf47K/4YpI5/0ATbseRkPN83387MjtdxqqTHyeTr0bXrP904Hty8Nfz5l9t37p7cu//e28wJuBBWWfcx5h6UNHCBEhV8TB1wHSv4EF/9UeIfPoPz0pp/cJvCQvOVkYkUHMl1efKJKUhwPmQxrKTJuXN8W+SiGLLzkLEhW1r0lXE+ZGCWDWHInFytcRH2R4syvozsi7k8GU3GZ7Ppi6fPQzLOTl88m9bGbDoLo/GkukZBc729vHd8h8REpsGgUNz7eTRJcUGyKIUCEs48pFxc8RXMyTRcg1/kVW+K8BF5lmFiHd0Gw8q7H5Fz7f1Wx8TUHNe+jZXOPmyeYXK2yKVJMwQj6oWSTIVow7LR4VI6EKi2ZHDhJOUaijV3XCBtx5AZ+CKs1pz6w/4u5tEip175zEG5Us6q9ZJ8FBUF23has6xzP0YoW7K7kQgbRKz/uRdSCumoeQVJ9StlRn4uKIm8MlqgsQYqsDJaoFyCqcDKaIOGCqVZxKKbnxf5X//DB9KSxiM3KDnCIYUdYafx6AcRVae/H21TcBytK8ckr/ADCehMoUzrBjtQfMNkorVdQp2B0+XusPOCgfIwilgi+3WMLb6LURfbbfJWww6vnlqMuoSG0LMLbll0Z0divjxUmAfdHRulYprNK8BwFIXMueapvRiX5Y6XITG98JVBjSBnh2gs7tfdVaLfHqnS26HSi9xlkrObnaZDE3xfig3SDUmSPjp5uxkrtSupfGgTYEMntt9xmucWDVqVP24TEnRZe2ybvCqkw084jWB/QA21IzBLe0aAm5WCev8r89D8pCCK+WlngFLu6Iheg5eeRM7D0ZTu03KY9pADmpgWOY0ylx7G0Qw2bB3bTf6QeTo6U/S4VcC81A/LSugL8v0zER423k/H0WQcvXs6evW6+ZbcDH4Lfg+eBFEwC14Fb4K3wUUggn+Db0fHR4PBr4OXgz8Hb2rqtaMm5kHwwzV49x8Tq9ah</latexit>

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

New abstractions and tools can empower programmers to build specialized automated

programming tools that improve software reliability.

Thanks!

bornholt@uw.edu https://unsat.org