© 2009 microsoft corporation. all rights reserved. automatic verification of heap manipulation...

© 2009 Microsoft Corporation. All rights reserved.

Automatic Verification of Heap Manipulation using Separation Logic

Josh Berdine

with thanks to• Byron Cook, Stephen Magill, Thomas Wies

• SLAyer project colleagues (MSR Cambridge)

• Cristiano Calcagno, Dino Distefano, Peter O’Hearn, Hongseok Yang

• SpaceInvader & Smallfoot project colleagues (London)

SOFSEM 09January 25, 2009


• 2000’s: impressive practical advances in automatic program

verification E.g.:

– SLAM: Protocol properties of procedure calls in device drivers, e.g.

any call to ReleaseSpinLock is preceded by a call to

AquireSpinLock

– ASTRÉE: no run-time errors in Airbus code

• However

– ASTRÉE assumes: no dynamic pointer allocation

– SLAM assumes: memory safety

• Important programs make serious use of heap memory... but

heap verification is hard

• Maybe one day… automatically prove:

– Memory safety of the operating system (kernel)

– The absence of any security holes due to abuse of memory

Context


• Heap-manipulating code

– dynamic allocation & deallocation: malloc & free

– pointer swing: x->Flink = y

• Want to know manipulation of pointers is “ok”

An old challenge

void filter(PSLL_ENTRY *a, int i) { PSLL_ENTRY t, *z = a;

while(*z != NULL) if((*z)->Data == i) { t = *z; *z = t->Flink; free(t); } else { z = &(*z)->Flink; }}


• Absence of memory errors

– dereference dangling pointers

– double free

– leak memory

• Doesn’t hold code very likely doesn’t behave “as expected”

• Focus on pointers over arrays: “pointer safety”

• Other properties

– data structure integrity

– accesses only a constrained portion of the heap

Memory safety (what does “ok” mean)


Memory Safety Bug


Memory Safety Bug

Blink Flink


Memory Safety Bug

Blink Flink

Irp

Blink Flink

Irp

Blink Flink


SLAyer:

Abstract interpreter based on a domain of separation logic formulae together with linear arithmetic.

Intention: to prove deeper properties about the heap than SLAM/SDV can handle.

Experimental integration with Static Driver Verifier in progress.

Joint work with the East London Massive

Memory Safety Bug

Blink Flink


Separation Logic: Heap Assertions

¤

x

(x 7! y)

y

(y 7! x)


• (sub)heaps are possible worlds (a form of modal logic)

• Classical Logic +

– emp : “the heap is empty”

– x y : “the heap has exactly one cell x, containing y”

– A ¤ B : “the heap can be divided so A holds of one part and B of

the other”

• Add inductive definitions

• …and other exotic things (“magic wand”)

• Standard model: RAM model

and many variations

Separation Logic: Heap Assertions

heap : N * ¯n Z


• A 0 A ¤ A

• (13 66) 0 (13 66) ¤ (13 66)

– Crucial for catching “no access after deallocation”

• A ¤ B 0 A

• (13 66) ¤ (42 7) 0 (13 66)

– Crucial for catching leaks

• (13 66) ¤ (13 66) ` false

– Crucial for catching interference

A Substructural Logic


Symbolic Execution of commands

• [(x f:-) ¤ P] [x.f] := 42 [(x f:42) ¤ P]

• [true] [x.f] := 42 [??]

• [(x -) ¤ P] free(x) [P]

• [true] free(x) [??]

• [P] x := alloc() [9 x’. P[x’/x] ¤ (x -)]

In-place Reasoning


x := null;

while(?) {

y := alloc();

[y.n] := x;

x := y;

}

Loops?

emp

emp Æ x=null

(y -) Æ x=null

(y {n:x}) Æ x=null

9x’. (x {n:x’}) Æ x’=null

Invariant:

(emp Æ x=null)

Ç (9x’. (x {n:x’}) Æ x’=null)


x := null;

while(?) {

y := alloc();

[y.n] := x;

x := y;

}

Loops?emp


9x’. (y -) ¤ (x {n:x’}) Æ x’=null

9x’. (y {n:x}) ¤ (x {n:x’}) Æ x’=null

9x’,x’’. (x {n:x’’}) ¤ (x’’ {n:x’}) Æ

x’=null

Invariant:

(emp Æ x=null)

Ç (9x’. (x {n:x’}) Æ x’=null)

Ç (9x’,x’’. (x {n:x’’}) ¤ (x’’ {n:x’}) Æ x’=null)

Ç (9x’,x’’,x’’’. (x {n:x’’’}) ¤ (x’’’ {n:x’’}) ¤ (x’’ {n:x’}) Æ

x’=null)

…


• lsn(E, F) , (E=F Æ emp)

Ç (9y’. (E {n:y’}) ¤ lsn (y’, F))

– listn(E) is shorthand for lsn (E, null)

ls(x,y) ¤ ls(y,x)

Linked List Segments

x

y



Ç (9y’. (E {n:y’}) ¤ lsn (y’, F))


lsn (x,t) ¤ (t {n:y}) ¤ listn(y)


x t y

Entailment: ` listn (x)



Ç (9y’. (E {n:y’}) ¤ lsn (y’, F))


lsn(x,t) ¤ (t {n:null}) ¤ listn(y)


x t y

Non-Entailment: 0 listn (x)


x := null;

while(?) {

y := alloc();

[y.n] := x;

x := y;

}

Loops?emp


9x’. (y -) ¤ (x {n:x’}) Æ x’=null

9x’. (y {n:x}) ¤ (x {n:x’}) Æ x’=null

9x’,x’’. (x {n:x’’}) ¤ (x’’ {n:x’}) Æ

x’=null

lsn(x,null)

Invariant:

(emp Æ x=null)

Ç (9x’. (x {n:x’}) Æ x’=null)

Ç lsn(x,null)

Generalize


x := null;

while(?) {

y := alloc();

[y.n] := x;

x := y;

}

a` lsn(x,null)

emp

lsn(x,null)

(y -) ¤ lsn(x,null)

(y {n:x}) ¤ lsn(x,null)

9x’. (x {n:x’}) ¤ lsn(x’,null)

` lsn(x,null)

Invariant:

(emp Æ x=null)

Ç (9x’. (x {n:x’}) Æ x’=null)

Ç lsn(x,null)

Loops?


Hierarchical Data Structures example


• hls(¤, E, F) , (E=F Æ emp)

Ç (9y’. ¤(E,y’) ¤ hls(¤, y’, F))

• For: ¤ls(x,y) = 9z’. (x {F:y, D:z’}) ¤ lsn (z’, null)

hls(¤ls, x, y)

Second-Order Linked List Segments

yxF F F

D D

n

n

n

n

n

n

D


• hls(¤, E, F) , (E=F Æ emp)

Ç (9y’. ¤(E,y’) ¤ hls(¤, y’, F))

• For: ¤emp(x,y) = 9z’. (x {F:y, D:z’})

hls(¤emp, x, y)

Second-Order Linked List Segments

yxF F F

D D D


trim(x,y) {

[hls(¤ls, x, y)]

if (x!=y) {

[9w’,z’. (x {F:w’, D:z’}) ¤ lsn(z’, null) ¤ hls(¤ls, w’, y)]

free_list(x->D);

[9w’,z’. (x {F:w’, D:z’}) ¤ emp ¤ hls(¤ls, w’, y)]

[9w’,z’. (x {F:w’, D:z’}) ¤ hls(¤ls, w’, y)]

trim(x->F,y);

[9w’,z’. (x {F:w’, D:z’}) ¤ hls(¤emp, w’, y)]

}

[hls(¤emp, x, y)]

}

Supposing: [lsn(x, null)] free_list(x); [emp]

Symbolic Execution with HO lists


• Adaptive shape analysis

– build in induction principles, rather than particular data structures

– automatic recognition of many complex variations on linked lists:

• singly-linked list segments

• …of non-empty doubly-linked lists

• …with back-pointers to the head node

• …of cyclic doubly-linked lists

• …

Hierarchical Data Structures


Recap

• Proof search by abstract interpretation

• Symbolically execute commands using in-place reasoning

• Check if invariant found using theorem prover

• Generalization of invariant approximations

– E.g. 9y’. (x {F:y’, D:42}) ¤ ls(y’,z) ` ls(x,z)

– Rules are specializations of prover’s


List* curr = null;int i = 0;while(i < n) { t = new(List); t->next = curr; curr = t; i++;}int j = 0;while(j < n) { curr = curr->next; j++;}

Transition Systems

0:

1:

2:

3:

0

1

2

curr = nulli = 0

[i < n]t = newt→next = currcurr = ti = i + 1

[i ≥ n]j = 0

[ j < n ]curr = curr→nextj = j + 1

3[ j ≥ n ]


Example

0

1

2

curr = nulli = 0


[i ≥ n]j = 0


3[ j ≥ n ]

[emp]

[emp]

curr = null

i = 0

[emp ∧ curr = null]

[emp ∧ curr = null ∧ i = 0]

0[emp]

1[emp ∧ curr = null

∧ i = 0]

curr = nulli = 0


Example

0

1

2

curr = nulli = 0


[i ≥ n]j = 0


3[ j ≥ n ]

[emp] 0[emp]

[emp ∧ curr = null∧ i = 0]

curr = nulli = 0


1 [i ≥ n] j = 0


Example

0

1

2

curr = nulli = 0


[i ≥ n]j = 0


3[ j ≥ n ]

[emp] 0[emp]


curr = nulli = 0


1 [i ≥ n] j = 0

1

[t ↦ next : curr’ ∧ curr’ = null

∧ i’ = 0 ∧ i’ < n∧ curr = t ∧ i = i’ + 1]

emp ∧ curr = null ∧ i = 0 ∧ i < nt ↦ – ∧ curr = null ∧ i = 0 ∧ i < nt ↦ next : curr ∧ curr = null ∧ i = 0 ∧ i < nt ↦ next : curr’ ∧ curr’ = null ∧ i = 0 ∧ i < n ∧ curr = tt ↦ next : curr’∧ curr’ = null ∧ i’ = 0 ∧ i’ < n

∧ curr = t ∧ i = i’ + 1

[i < n]t = new

t→next = currcurr = ti = i + 1




Example

0[emp]


curr = nulli = 0

1

1[t ↦ next : curr’ ∧ curr’ = null ∧ i’

= 0∧ i’ < n ∧ curr = t ∧ i = i’ + 1]

[i ≥ n] j = 0

[i ≥ n] j = 0

1

[t ↦ next : t’ * t’ ↦ next : curr’∧ curr’ = null ∧ i’ = 0

∧ i’ < n ∧ curr’’ = t’ ∧ i’’ = i’ + 1∧ i’’ < n ∧ curr = t ∧ i = i’’ + 1]




Example

0[emp]


curr = nulli = 0

1

1[t ↦ next : curr’ ∧ curr’ = null ∧ i’

= 0∧ i’ < n ∧ curr = t ∧ i = i’ + 1]

[i ≥ n] j = 0

[i ≥ n] j = 0

1

[t ↦ next : t’ * t’ ↦ {next : curr’}

∧ curr’ = null ∧ i’ = 0∧ i’ < n ∧ curr’’ = t’ ∧ i’’ = i’ + 1∧ i’’ < n ∧ curr = t ∧ i = i’’ + 1]

Generalization

Generalization


• lsK(E, F) , (K=0 Æ E=F Æ emp)

Ç (K>0 Æ 9y’. (E {next:y’}) ¤ lsK-1(y’, F))

– lsn(E, F) is shorthand for 9k. lsnk(E, F)

– lsK(E, F) represents a list segment from E to F (via next) of length

K

Generalization

[t ↦ next : t’ * t’ ↦ {next : curr’}

∧ curr’ = null ∧ i’ = 0∧ i’ < n ∧ curr’’ = t’ ∧ i’’ = i’

+ 1∧ i’’ < n ∧ curr = t ∧ i = i’’ +

1]

{ ls2(curr,null) ∧ i = 2}



⋮

{∃k,v. lsk(curr,null) ∧ i = v}

{∃k. lsk(curr,null)}


Symbolic Execution



0[emp]

[emp]

curr = nulli = 0

1

1[t ↦ next : null]

[i ≥ n] j = 0

[i ≥ n] j = 0

[∃k. lsk(curr,null)]


[i ≥ n]; j = 02


2


3[ j ≥ n ]

[ j < n ]2’

[∃k. lsk(curr,null)∧ j < n]

curr = curr →next

1


Shape-Arithmetic Correlations

Why is this okay?

Relationship between i andthe length of the list

∃k. lsk(curr,null)

i = k

presentin thecode

appears due togeneralization



• Need to know about correlations between shape and

arithmetic

• That is, refine shape information with arithmetic information

• Need to understand arithmetic

• Could use:

– Abstract Interpretation

– Interpolants

– Proofs

– Counter Automata

– Probably others

Arithmetic Engine


Modularity

• A framework that can utilize any such

refinement method

• Must preserve as much of the control flow

as possible

• Generate programs corresponding to

abstract traces (sets of concrete traces)

• Reason for safety of generated program

must transfer to our abstraction


Abstract States

∃k. lsk(curr, null)

∃v, w. x ↦ v * y ↦ w

must make theseappear in the generated

program

Variables in CEx program:original program varsexistential vars

Commands in CEx program:

updates to program vars

changes to witnesses ofexistential vars


Generating Code

∃y. Q(y)

∃x. P(x)

c


Generating Code

∃x. P(x)

∃y. Q(y)

c

P(a)


Generating Code

∃x. P(x)

∃y. Q(y)

c

P(a)

P’(a)⊦


Generating Code

∃x. P(x)

∃y. Q(y)

c

P(a)

P’(a)⊦

a := x

y := f(a)

∃-Elim

∃-Intro


Example



1 [∃k. lsk(curr,null)]

a := k[lsa(curr,null)]


[curr ↦ next : c’ * lsa(c’,null) ∧ i = i’ + 1 ∧ i’ < n ∧ curr = t]

generalize

[lsa+1(curr,null)]

k := a+1[∃k. lsk(curr,null)]


Encoding the Error

2’

[∃k. lsk(curr,null)∧ j < n]

curr = curr →next

Crash Condition of A[E] ≝ the condition under which

the command A[E] results in abort.

in this case: (k = 0)

if(k=0) goto ERROR;

Add to program:


Full Program (≡ an unrolling of:)


int k;int curr = 0;int i = 0;while(i < n) { t = ?; curr = t; i++; k++;}int j = 0;while(j < n) { curr = ?; j++; if(k = 0) goto ERROR; else k--;}


List* curr = null;int* i = new(0);while(*i < n) { t = new(List); t->next = curr; curr = t; *i = *i + 1;}int j = 0;while(j < n) { curr = curr->next; j++;}

Also Works For

[∃k,v. lsk(curr,null) ∧ i ↦ v]

[∃k,v. lsk(curr,null) ∧ i ↦ v]

a := k; b := v

[lsa+1(curr,null) ∧ i ↦ b + 1]

k := a+1; v := b+1

loopbody


1. Perform shape analysis

– Construct abstract transition system

• Separation Logic assertions at vertices

• Heap-manipulating commands on edges

2. Construct arithmetic abstraction

1. Add commands tracking values of existential variables

• “auxiliary variable introduction”

2. Throw away heap-manipulation commands

3. If ERROR unreachable in arithmetic abstraction

– According to any arithmetic prover

4. Then have a memory safety proof for original program

Combined Verification


• End up needing to know rather a lot about the data structures

• Is proving only memory safety underkill?

• What else can we use this info for?

Beyond Memory Safety


List* find(List* head, int fo) {

List* prev, entry, tmp;

prev = head;

entry = head->next;

while(entry != head) {

if (entry->data == fo) {

tmp = entry->next;

prev->next = tmp;

if (?) return entry;

entry->next = entry;

entry = entry->next;

} else {

prev = entry;

entry = entry->next;

}

}

return NULL;

}

Non-Terminating Arithmetic Abstraction

List* find(List* head, int fo) {

int k;

List* prev, entry, tmp;

prev = head;

entry = ?;

while(entry != head) {

assume(k>0);

if (?) {

tmp = ?;

if (?) return entry;

entry = ?;

} else {

prev = entry;

entry = ?;

k--;

}

}

return NULL;

}


• Arithmetic abstraction is sound also for termination:

– If original program admits an infinite trace

– Then so does the arithmetic abstraction (of its memory safety

proof)

– Despite the arithmetic abstraction being an over-approximation

• So to prove termination of heap-manipulating code:

1. Prove memory safety

2. Translate proof & program to heap-free program that

• Terminates only if original does

• Reveals potential progress measures for data structures

3. Run an arithmetic termination proverVariance analysis

Termination Proving


Concurrency

PLIST_ENTRY head;KSPIN_LOCK lock;

void initialize() { head = ExAllocatePool(PagedPool, sizeof(LIST_ENTRY)); InitializeListHead(&head);}

void producer() { KIRQL Irql; PLIST_ENTRY entry; while (nondet()) { KeAcquireSpinlock(&lock, &Irql); entry = ExAllocatePool(PagedPool, sizeof(LIST_ENTRY)); if (entry) { InsertHeadList(&head, entry); } KeReleaseSpinlock(&lock, Irql); }}

void consumer() { KIRQL Irql; PLIST_ENTRY entry; while (nondet()) { KeAcquireSpinlock(&lock, &Irql); entry = RemoveHeadList(&head); if (entry != head) { ExFreePool(entry); } KeReleaseSpinlock(&lock, Irql); }}

lock protects a (doubly-linked cyclic)

list pointed to by head

lock protects a (doubly-linked cyclic)

list pointed to by head


• Infer description of data structure each lock protects

• Descriptions induce partitioning of heap

– Changes from one program point to another

– No part accessed simultaneously by multiple threads

• Catch accesses outside of critical sections

Concurrency


• Thread-modular shape analysis

– supports concurrently accessed heap data

– coarse-grained locking

– works if there is not intricate coordination between the threads

– also checks for data races

– scalability: sub-exponential in number of threads

• Locks in the heap

– unboundedly many locks

– finer grained locking, e.g. hand-over-hand list locking

Concurrency


Post-Context

Source.c .h

VerificationEngine

Proof

Counter-Example

False alarm

Programming language &

runtime model

Potential Counter-example


• Problem is undecidable:

– For any program and any property, does program satisfy

property?

• Focus on class of programs and properties

– Trade-off to gain automatic inference of annotations

Inherently limited


• Not property driven

– Already focused on a class of properties

– Analyzer automatically finds & proves strongest specifications /

most information it can

• For a property of interest, analysis results

– possibly prove it

– are possibly overkill

– possibly do not (but may form basis of further proof)

• Property proved = Absence of whole class of bugs

Static analysis for verification


Post-Context

Source.c .h

VerificationEngine

Proof

Counter-Example

False alarm


runtime model



Post-Context

Source.c .h

VerificationEngine

API spec &OS model

Proof

Counter-Example

False alarm


runtime model



• Carefully chosen fragment of a logic for describing heap data

structures

• Theorem prover for that fragment

• In-place reasoning about commands

• Calculate invariants, generalizing using tweaked version of

prover’s proof theory

• Abstract (candidate) memory safety proofs to arithmetic

programs

• Use arithmetic analyses to complete safety proofs

• Use arithmetic termination provers

Summary

© 2009 microsoft corporation. all rights reserved. automatic verification of heap manipulation...

Documents