1

E. YahavSchool of Computer Science

Tel-Aviv University

Typestate Verification: Abstraction Techniques and

Complexity Results

J. Field, D. Goyal, G. RamalingamIBM T.J. Watson Research Center

2

Motivation

Componenta library with cleanly encapsulated state

Clienta program that uses

the library

3

Lightweight Specification"correct usage" rules a client must follow

"call open() before read()"

Certificationdoes the client program satisfy the lightweight specification?

Approach

Componenta library with cleanly encapsulated state

Clienta program that uses

the library

Tailor certification procedure to the specification being verified

Canvas Project (Component Annotation, Verification and Stuff)

4

Typestate Checking [Strom and Yemini 1986]

Statically verify that programs satisfy certain kinds of temporal safety properties A file should be opened before it can be read A file should not be read after it is closed …

Many recent approaches to verification based on typestate checking

Interaction of aliasing and typestate checking not well understood

5

Goals

How to increase precision of typestate checking for programs with aliasing

Specialized abstractions Exploit nature of property being verified Exploit nature of programs being verified

Complexity bounds Relate difficulty of verification to the nature of

the property/programs being verified

6

Specifications as Finite Automata

Automaton defines set of valid operation sequences that can be performed on an object

The set of valid sequences is prefix-closed all transitions from error state are to itself

Can also use regular expressions implicit prefix-closure (in this talk) read*;close

not-closed

closed

error

read

close readclose

closeread

7

Precise Typestate Checking and Aliasing

Precise modulo common assumption that all paths in the program are executable

No aliasing

O(|S||V||P|)

Two or morelevels of pointersw/o recursive data structures

PSPACE-hard

Recursive data structures

Undecidable

Shallow aliasing(1-level pointers)

?

8

Typestate Checking + Shallow Aliasing

x := new File();y := new File();z := y;if (?) {

y.close();z := x;

}z.read();

9

Common Approach: Two-Phase Analysis

Break certification into two phases separate points-to analysis apply typestate checking

Separation of analyses can lead to imprecision

Client Program

Lightweight Specification

not-closed

closed

error

read

close read

close

closeread

Points-to analysis

Typestate checking

10

Loss of Precision in Two-phased Approach

x := new File();y := new File();z := y;if (?) {

y.close();z := x;

}z.read();

file 1 created

file 2 closedfile 2 may be closed

read may be erroneous

file 2 created

z may point to file 1 or file 2

(independent attributes analysis)

11

Is Precise but Efficient Typestate Checking Possible?

Not all finite state properties are equally hard to verify!

read*;close open+;read

Polynomial PSPACE-complete

12

Plan

Property Classes Omission-Closed Properties Repeatable-Enabling Sequence Properties

Program Classes In-degree Bounded Aliasing

Summary

13

Omission-closed Properties

Every subsequence of a valid sequence is also valid read*;close read;read;read;read;close

There exists a finite set of forbidden subsequences s.t. a sequence is invalid iff it contains some forbidden subsequence as a subsequence close;read close;close

Verifiable in polynomial time!

14

Abstracting Aliasing and Typestate information

Predicate

Meaning

<A,S> there exists an object that all variables in A point to, and this object’s state is in set S

<Error> there exists an object in the error state

15

x := new File();y := new File();

z := y;

if (?) {y.close();

z := x;}

z.read();

Example

<{z},{closed,error}>,…

<{x},{closed,error}>,…

<{x,y},*>, <{x},{closed,error}>, <{y},{closed,error}>…

<{x,y},*>, <{x},{closed,error}>, <{z},{closed,error}>…

Selected false predicates

16

Ensuring Precise Polynomial Analysis

How do we perform verification in polynomial time without losing precision?

Ensuring polynomial time Independent attribute analysis Polynomial number of predicates

Ensuring no loss of precision Require set of predicates to be disjunctively

WP-closed

17

Selecting Predicates using Iterated Weakest Precondition

Process applied to a specific property and not to a specific client program

Start with a set of predicates of interest W = {<Error>}

Repeat for every predicate p in W and every statement form st,

compute WP(st,p) simplify WP(st,p) add the disjuncts of simplified formula to W

Resulting set W is disjunctively WP-closed

18

Applying IWP To read*close

not-closed closed error

read

close readclose

closeread

<Error>

x.read() x.close()

<Error><{x}, {closed,error}> <Error><{x}, {closed,error}>

<{x}, {closed,error}>

y.read()

<{x}, {closed,error}>

y.close()

<{x}, {closed,error}> <{x,y}, {not-closed,closed,error}>

<{x,y}, {not-closed,closed,error}>

19

x := new File();y := new File();

z := y;

if (?) {y.close();

z := x;}

z.read();

Example Revisited

<{z},{closed,error}>,…

<{x},{closed,error}>,…

<{x,y},*>, <{x},{closed,error}>, <{y},{closed,error}>…

<{x,y},*>, <{x},{closed,error}>, <{z},{closed,error}>…

Selected false predicates

20

Selecting Predicates for Omission-Closed Properties

Construct the (part of) predecessor graph on sets of states, reachable from {error}

read

not-closedclosed,error

closed,error

errorclose read

read

close

<{x,y}, {not-closed, closed, error}>

<Error>

<{x}, { closed, error}>

Relevant

predicate

families

21

Eureka Steps in Simplifying Weakest-Preconditions

WP (x.read(), <Error>) could be simplified to

<Error> <{x}, {closed,error}>

or

<Error> <{x}, {closed}>

Choosing the first form is critical … the second form will not lead to a polynomial time solution!

22

What about properties that are not omission-closed?

Can we always find similar polynomial-time “precise” verification algorithms?

Unfortunately, that is unlikely …

23

Repeatable Enabling Sequence Properties

RES properties a sequence is invalid, but + is valid open+;read read open;open;read

Verification of RES properties NP-complete for acyclic programs PSPACE-complete for general programs

no polynomial bound exists for the length of the shortest error path (unless PSPACE = NP)

24

The Intuition

…p1.open(); …

p2.open(); ……pk.open(); …

q.read();

open+;read example

…p1.close(); …

p2.close(); ……pk.close(); …

q.read();

read*;close example

a hard question k easy questions

Is there a path along whichq p1 & … & q pk ?

Can q = p1?…Can q = pk?

25

Is that all?

No… some properties are neither omission-closed

nor repeatable-enabling-sequence propertiesFor example

open;read not omission-closed - open;read valid but subsequence

read not valid open is an enabling sequence (enables read), but not

repeatable - open;open;read is not valid more results in the paper…

(lock;unlock)*

26

Taking A Different Angle

Recap: some property classes are hard to verify e.g., open+;read

What to do?Exploit properties that programs usually haveA single object usually not simultaneously

pointed by a large number of program variables Is it possible to do finite-state verification

efficiently at least for such programs?

27

Limited Aliasing

Arbitrary finite state properties can be verified in time O(nk+1) programs of size n, maximum aliasing factor k

Naive approaches take exponential time

28

Limited Aliasing – Key Ideas

Use a new predicate vocabulary[A, s] denotes

all variables in A point to the same object O O is in state s no variable not in A points to object O

Use a forward propagation analysis Lazily introduce predicates Complexity proportional to number of

predicate instances that take non-false value

29

Beyond Shallow Programs:Heap Analysis

Use instrumented heap analysis - heap analysis guided by properties of heap objects of interest Parametric shape analysis via 3-valued logic (Sagiv, Reps,

Wilhelm, TOPLAS 02) Establishing Local Heap Safety Properties with

Applications to memory-management (Shaham, Yahav, Kolodner, Sagiv, SAS 03)

Our abstractions can be used to define heap object properties of interest Improve precision of the analysis

30

Related Work

Type-based Deline and Fahndrich, PLDI 2001 (Vault) Foster, Terauchi, and Aiken, PLDI 2002

Selective path-sensitivity Das, Lerner, and Seigle, PLDI 2002 (ESP) Engler, Chelf, Chou and Hallem, OSDI 2000

Complete abstract interpretations Giacobazzi, Ranzato, and Scozzari JACM 2000

Other Corbett et.al., ICSE 2000 (Bandera) Ball and Rajamani, SPIN 2001 (SLAM) Henzinger et al., POPL 2002 (BLAST) Naumovich, Clarke, Osterwell, and Dwyer, FLAVERS

(1997) …

31

Omission-closed

Repeatable enabling sequence

Almost omission-closed

Other

E.g. read*;close open+;read open;read (lock;unlock)*

Defn. valid()valid()

valid()valid()

kk valid()valid()

Acyclic programs

P NP-complete P ?

Cyclic programs

P PSPACE-complete

PEP => PGeneral: ?

?

Bounded aliasing

P

Summary

32

http://www.cs.tau.ac.il/~yahave

33

BACKUP SLIDES

34

Omission Closed WP Rules

Stmt WP(Stmt,<A,S>)

x:=y <A[xy],S>

x := new() <A,S> if x Afalse if x A A {x}true if A={x} init Sfalse if A={x} init S

x.op() <A,S> if pre(S,op) = S<A{x},pre(S,op)> <A,S>

At program entry

true if |A|=1 init Sfalse if |A|1 init S

35

Bounded Aliasing Flow Rules

Stmt flow(Stmt)([A,])

x:=y {[A{x},]} if y A{[A{x},]} if y A

x := new() {[{x},init], [A{x},]} if x A{[A,]} if x A

x.op() {[A,succ(,op)]} if x A{[A,]} if x A

36

DISCARDED SLIDES

Slides beyond this point are discarded slides

37

Verification by counting

Counting “how often” a predicate holds true can be done efficiently for acyclic programs and certain kinds of predicates

Certain finite state properties can be verified efficiently using such a counting approach in the case of acyclic graphs

Can be applied to programs with loops by “unrolling loops” … effective technique for properties where bounds can be

established on the length of shortest error paths

38

Example

x := new File();y := new File();z := y;if (?) {

y.close();z := x;

}z.read();

z points to unclosed file z points to

closed file

z points to unclosed file

read is safe

z points to unclosed file

39

Example

f := new File();While (?) {

f.read(); …if (?) {

f.close();f := new File();

}}

read is safe

40

IWP: Omission-Closed Properties

WP (x.op(), <A,S>) = ?pred(S,op) = set of states whose successor on

operation op is in Scase 1: if pred(S,op) = S, then

WP (x.op(), <A,S>) = <A,S>case 2: if pred(S,op) S, then

WP (x.op(), <A,S>) = <A {x}, pred(S,op)> <A,S>

41

IWP: Omission-Closed Properties

case 3: if NOT (pred(S,op) S) ? then WP (x.op(), <A,S>) is more complicated … it cannot

be expressed as a predicate of the form <A,S>.

However, any predicate <A,S> that is relevant for verification of an omission-closed property satisfies pred(S,op) S for every op …

42

Relevant Predicates forOmission-closed Properties

Which predicates are relevant?Base

< { }, {error} > is relevant

Inductionif <A,S> is relevant after x.op() and

if pred(S,op) S, then

<A {x}, pred(S,op)> is relevant

43

Applying IWP To read*close

not-closed closed error

read

close readclose

closeread

WP (x.read(), <{ }, {error}>) = <{ }, {error}> <{x}, {closed,error}>

WP (x.close(), <{ }, {error}>) = <{ }, {error}> <{x}, {closed,error}>

WP (y.read(), <{x}, {closed,error}>) = <{x}, {closed,error}>

WP (y.close(), <{x}, {closed,error}>) =

<{x}, {closed,error}> <{x,y}, {not-closed,closed,error}>

44

Abstract Transformers

Computed using weakest preconditionWP(x.read(),<Error>)

<Error>

x.read()

<Error><{x}, {closed,error}>

45

Iterated Weakest Precondition: A Methodology For Designing Analyses

Start with a set B of predicates of interestIteratively compute a superset T of B such

that for every predicate p of interest and every

possible statement st, compute WP(st,p) simplify WP(st,p) and identify the disjuncts of

simplified formula as predicates of interest

Resulting set is disjunctively WP-closed

46

Independent Attribute Predicate Abstraction

A special class of abstract interpretationsDefined by a set of predicates W

Predicate = Boolean function of the concrete state “there exists an object in the error state” “variables x and y point to the same object”

Concrete state abstracted by value of the set of predicates A function from W to {true, false}

Independent attribute analysis A set of concrete states abstracted by a function from W

to {true, false, maybe}

47

Precise Predicate Abstraction Using Independent Attributes

W = set of predicates for a program P For every predicate pi in W, and every statement St in P,

there exist predicates pj1, pj2

,…, pjm

in W

WP (St, pi) = pj1 pj2

… pjm

Answering “exists a path making predicate w true at program point q” can be done in O(|W||P|) time

48

Predicates For read*close

For a program P with a set of variables VThe set of predicates W

< Error > < {x}, {closed,error}> for every variable x <{x,y}, {...}> for every pair of variables x and y

O(|V|2) predicatesPrecise verification can be done in time O(|

V|2|P|)

49

open;read

open;read can be expressed as two constraints open;open, read;open, and read;read are forbidden

subsequences an omission-closed property that can be efficiently verified

read should be preceded by open the repeatable-enabling-sequence open+;read

Is open+;read easy to verify for programs guaranteed not to contain the forbidden subsequences open;open, read;open, and read;read ?

50

The Intuition

…p1.open(); …p2.open(); ……pk.open(); …

q.read();

open;read example Can pi = pj, for some i <> j?then, stop; Error!

Is there a path along whichq <> p1 & q <> p2 & … & q <> pk ?

not the same Error!How often is q.read() executed?

How often is q = pi for some i?

How often is q = p1?…How often is q = pk?

51

Verification by Counting

X Pf Pt X,,u exact,u

{x} read 1 1 2 -

{x} open;read

1 1 2 -

{y} read 1 1 2 -

{y} open;read

0 1 1 -

read 2 2 4 1

open;read

1 2 3 3Pf Pt

x := new File(); y := new File(); x.open(); if (?) { y.open(); } x.read(); y.read();u:

52

Almost Omission-Closed Properties

There is an integer k such that every subsequence of a valid sequence of length greater than k is also valid open;read

Polynomial time verification for acyclic programsPolynomial time verification for programs that

have a polynomial bound on the length of shortest error path