Carnegie Mellon University

Symbolic Approaches to Symbolic Approaches to Invariant Checking and Invariant Checking and

Automatic Predicate Automatic Predicate AbstractionAbstraction

Symbolic Approaches to Symbolic Approaches to Invariant Checking and Invariant Checking and

Automatic Predicate Automatic Predicate AbstractionAbstraction

http://www.cs.cmu.edu/~bryant

Randal E. Bryant

Contributions by graduate students:Sanjit Seshia, Shuvendu Lahiri

– 2 –

OutlineOutline

TaskTask Prove safety properties of term-level systems

MethodMethod Generate & prove inductive invariants by predicate

abstraction

FeaturesFeatures Symbolic approach using SAT enumeration Handles important class of quantified predicates Augment with automated predicate discovery

– 3 –

ReachableStates

Verifying Safety PropertiesVerifying Safety Properties

State Machine ModelState Machine Model State encoded as Booleans, integers, and functions Next state function expresses how updated on each step

Prove: System will never reach bad stateProve: System will never reach bad state

ResetStates

BadStates

PresentState

NextState

Inputs(Arbitrary)

Reset

– 4 –

• • •

Rn

R2

True Model CheckingTrue Model Checking

Reach Fixed-PointReach Fixed-Point Rn = Rn+1 = Reachable

Impractical for Term-Level Impractical for Term-Level ModelsModels

Many systems never reach fixed point

Can keep adding elements to buffer

Convergence test undecidable

BadStates

R1

ResetStates

– 5 –

I

Inductive Invariant CheckingInductive Invariant Checking

Key Properties of System that Make it Operate CorrectlyKey Properties of System that Make it Operate Correctly Formulate as formula I

Prove InductiveProve Inductive Holds initially I(s0)

Preserved by all state changes I(s) I((i, s))

ReachableStates

ResetStates

BadStates

– 6 –

Inductive InvariantsInductive Invariants

Formulas Formulas II11, …, , …, IInn

Ij(s0) holds for any initial state s0, for 1 j n

I1(s) I2(s) … In(s) Ij(s ) for any current state s and successor state s for 1 j n

Overall CorrectnessOverall Correctness Follows by induction on time

Restricted form of invariantsRestricted form of invariants x1x2…xk (x1…xk)

(x1…xk) is a CLU formula without quantifiers

x1…xk are integer variables free in (x1…xk)

– 7 –

Restricted Invariants and ProofsRestricted Invariants and Proofs

Proving invariants inductive requires quantifiersProving invariants inductive requires quantifiers|= (x1x2…xk (x1…xk)) y1y2…ym (y1…ym)

|= x1x2…xk (x1…xk) y1y2…ym (y1…ym)

Universally Quantified Variables “Free”Universally Quantified Variables “Free” Validity proves holds for all interpretations

Existentially Quantified Variables Make Problem Existentially Quantified Variables Make Problem UndecidableUndecidable Automatic instantiation of x1…xk with concrete terms

Sound but incomplete method

Reduce the quantified formula to a CLU formulaCan use the decision procedure for CLU

– 8 –

Constructing Invariants from PredicatesConstructing Invariants from Predicates

Recipe: Invariants

Result: Correctness

reg.valid(r)

r,t.reg.valid(r) reg.tag(r) = t (rob.head reg.tag(r) < rob.tail rob.dest(t) = r )

rob.head reg.tag(r)

reg.tag(r) = trob.dest(t) = r

– 9 –

Automatic Predicate AbstractionAutomatic Predicate Abstraction

Graf & Saïdi, CAV ‘97

IdeaIdea Given set of predicates P1(s), …, Pk(s)

Boolean formulas describing properties of system state

View as abstraction mapping: States {0,1}k

Defines abstract FSM over state set {0,1}k

Form of abstract interpretationDo reachability analysis similar to symbolic model checking

ImplementationImplementation Early ones had weak inference capabilities

Call theorem prover or decision procedure to test each potential transition

Recent ones make better use of symbolic encodings

– 10 –

Abstract State SpaceAbstract State Space

ConcreteStates

AbstractStates

P1(s), …, Pk(s)

s

AbstractionFunction

t

Abstraction

ConcreteStates

AbstractStates

s t

ConcretizationFunction

Concretization

– 11 –

Abstract State MachineAbstract State Machine

Transitions in abstract system mirror those in concrete

Abstract

ConcreteSystem

AbstractSystem

s

Concretize

t t

s

Concrete Transition

Abstract Transition

– 12 –

Generating Concrete InvariantGenerating Concrete Invariant

Reach Fixed-Point on Reach Fixed-Point on Abstract SystemAbstract System Termination guaranteed,

since finite state

Equivalent to Computing Equivalent to Computing Invariant for Concrete Invariant for Concrete SystemSystem Strongest possible

invariant that can be expressed by formula over these predicates

• • •Rn

R2

R1

ResetStates

A

AbstractSystem

Concretize

ConcreteSystem

I

ResetStates

C

– 13 –

Predicate Abstraction ExamplePredicate Abstraction Example

State SpaceState Space State variables: { x, y }

Initial StateInitial State { (2, 1) }

Next State BehaviorNext State Behavior x x y y

Verification TaskVerification Task Prove all bad states unreachable

InitialState

BadStates

– 14 –

Precise AnalysisPrecise Analysis

Reachable StatesReachable States { (2, 1), (2, 1) } Reachable

States

BadStates

– 15 –

PredicatesPredicates

Use 3-valued predicates in this example

cx:3

L

E

G

cx:y

L

E

G

cy:0

G

E

L

– 16 –

Abstract Initial StateAbstract Initial State

Reached Set #0{ LGG }

cx:3

L

cx:y

G

cy:0

G

– 17 –

Step 1: Concretize Reached Set #0Step 1: Concretize Reached Set #0Reached Set #0

{ LGG }

L

G

G

cx:3 cx:y cy:0

(Note loss of precision)

s

Concretize

s

Concretize

s

Concretize

– 18 –

Compute Possible Successor StatesCompute Possible Successor States

x xy y

s

Concrete Transition

s

Concretize

s

Concretize

s

Concretize

s

Concrete Transition

s

Concrete Transition

– 19 –

Abstract Newly Reached StatesAbstract Newly Reached States

0 0 0

Reached Set #1{ LLL, LGG }

L L

L

cx:3 cx:y cy:0

s

Concrete Transition

Abstract

Abstract

Abstract

s

Concretize

s

Concretize

s

Concretize

s

Concrete Transition

s

Concrete Transition

– 20 –

Step 2: Concretize Reached Set #1Step 2: Concretize Reached Set #1Reached Set #1

{ LLL, LGG }

L

cx:3 cx:y cy:0

(Note loss of precision)

L

L

s

Concretize

s

Concretize

s

Concretize

– 21 –

Compute Possible Successor StatesCompute Possible Successor States

x xy y

s

Concrete Transition

s

Concretize

s

Concretize

s

Concretize

s

Concrete Transition

s

Concrete Transition

– 22 –

Abstract Newly Reached StatesAbstract Newly Reached States

Reached Set #2{ LLL, LGG, EGG, GGG }

cx:3

G

E

cx:y

G

cy:0

G

s

Concrete Transition

Abstract

Abstract

Abstract

s

Concretize

s

Concretize

s

Concretize

s

Concrete Transition

s

Concrete Transition

– 23 –

EGG

Final Reached State SetFinal Reached State Set

LLLLGG GGG

BadStates

– 24 –

Conventional Implementation of P.A.Conventional Implementation of P.A.BasisBasis

Abstract state sets described as formulas over Boolean variables B = b1, …, bk

Current state given by formula (b1, …, bk)

Check whether candidate state (b1, …, bk) is successor

ConcreteSystem

AbstractSystem

Concretize

[P/B]

[P/B]

Concretize

Intersect?

Abstract Transition?

[P/B][/S] Predecessor

– 25 –

Drawbacks of Conventional ImplementationDrawbacks of Conventional Implementation

Very SlowVery Slow Guess at possible next state Construct term-level formula and test for satisfiability Possibly 2k calls to decision procedure

Can Only Handle Proposition PredicatesCan Only Handle Proposition Predicates Cannot construct quantified invariants

[P/B]

Intersect?

[P/B][/S]

[P/B] [P/B][/S]Satisfiable?

– 26 –

Symbolic Approach to P.A.Symbolic Approach to P.A. Lahiri, Bryant, Cook, CAV 2003

Generate Quantified Formula Describing Next Abstract State Generate Quantified Formula Describing Next Abstract State SetSet Current state given by formula (B) Generate formula (B) describing all successors

AbstractSystem

All Abstract Transitions

S, X

(B, S, X)How to reach abstract state B via concrete states S and X

– 27 –

Symbolic Approach (cont.)Symbolic Approach (cont.)

Transform into Quantified Boolean FormulaTransform into Quantified Boolean Formula Formula of form Next(B) = S, X (S, X, B)

S, X: Integer and function variables

B: Abstract state variables

Translate into Boolean formula of form A (A, B)A: Boolean variables encoding integer & function values

Key Property{ B | (S, X, B) satisfiable } = { B | (A, B) satisfiable }

Solve using either SAT enumeration or BDD quantification

– 28 –

Symbolic Formulation of Step 2Symbolic Formulation of Step 2

Concretized State SetConcretized State Set Encode each 3-valued {L, E, G}

predicate with 2 Boolean variables (l, g)

Represent state set as formula(l1 g1 l2 g2 l3 g3)

(l1 g1 l2 g2 l3 g3)

Reached Set #1{ LLL, LGG }

LLLLGG

l1:

x < 3

g1:

x > 3

l2:

x < y

g2:

x > y

g3:

y > 0

l3:

y < 0

– 29 –

Next-State PredicatesNext-State Predicates

Next State (Next State (xx, , yy )) Get predicates l1, l2, l3 , g1, g2, g3

Determine conditions under which predicates will hold in next state Express in terms of current state (x, y)

Next State Next State PredicatePredicate

ConditionCondition x = x

y = y

CurrentCurrent

StateState

MatchesMatches

l1x < 3 x < 3 x > 3 —

l2x < y x < y x > y g2

l3y < 0 y < 0 y > 0 g3

g1x > 3 x > 3 x < 3 —

g2x > y x > y x < y l2

g3y > 0 y > 0 y < 0 l3

– 30 –

Consistency ConstraintsConsistency Constraints Eliminate impossible predicate

combinations In general, may need to introduce

additional variablesTo express more complex transitivity

constraints

l1

g1

l2

g2

g2

l2

g3

l3

l3g3

l1

g1

(g1 g1)

(g1 l1)

(g2 g3 l1)

– 31 –

Symbolic FormSymbolic Form

FormulationFormulation Express compatible combinations of current-state & next-

state variables Quantify out current-state variables Gives formula over next-state variables

[ (ll1 gg1 ll2 gg2 ll3 gg3)

(ll1 gg1 ll2 gg2 ll3 gg3) ]

CurrentState

ll11, , ll22, , ll33 , , gg11, , gg22, , gg33

ConsistencyConstraints

(gg1 gg1) (gg1 ll1) (gg2 gg3 ll1)

ll2 gg2 gg2 ll2

ll3 gg3 gg3 ll3

– 32 –

l1 g1 l2 g2 l3 g3 l1 g1 l2 g2 l3 g3

Next State

1 0 1 0 1 0 0 0 0 1 0 1 EGG

1 0 1 0 1 0 0 1 0 1 0 1 GGG

1 0 1 0 1 0 1 0 0 1 0 1 LGG

1 0 0 1 0 1 1 0 1 0 1 0 LLL

l1 g1 l2 g2 l3 g3 l1 g1 l2 g2 l3 g3

Next State

1 0 1 0 1 0 0 0 0 1 0 1 EGG

Extracting Next-State SetExtracting Next-State Set Run SAT checker over formula Generate blocking clause for each newly generated state

[ (ll1 gg1 ll2 gg2 ll3 gg3)

(ll1 gg1 ll2 gg2 ll3 gg3) ]

(gg1 gg1) (gg1 ll1) (gg2 gg3 ll1)

ll2 gg2 gg2 ll2

ll3 gg3 gg3 ll3

(l1 g1 l2 g2 l3 g3)

– 33 –

Quantified Invariant GenerationQuantified Invariant Generation(Lahiri & Bryant, VMCAI 2004) User supplies predicates containing free variables Generate globally quantified invariant

ExampleExample Predicates

p1: reg.valid(r)

p2: rob.dest(t) = r

p3: reg.tag(r) = t

Abstract state satisfying (p1 p2 p3) corresponds to concrete state satisfying r,t[reg.valid(r) reg.tag(r) = t

rob.dest(t) = r] rather than r[reg.valid(r)] r,t[reg.tag(r) = t]

r,t[rob.dest(t) = r]

– 34 –

Systems Verified with Predicate AbstractionSystems Verified with Predicate Abstraction

Very general modelsUnbounded processes, buffers, cache lines, …

Safety properties only

Model Predicates Iterations CPU Time

Out-Of-Order Execution Unit 25 9 1,207s

German’s Cache Protocol 13 9 14s

German’s Protocol, unbounded channels

24 17 427s

Bounded Retransmission Buffer 22 9 11s

Lamport’s Bakery Algorithm 33 18 471s

– 35 –

Automatic Predicate DiscoveryAutomatic Predicate Discovery

Strength of Predicate AbstractionStrength of Predicate Abstraction If give it right set of predicates, PA will put them together

into invariant

WeaknessWeakness Gets nowhere without right set of predicates Typical failure mode: Generate “true” as invariant

ChallengesChallenges Too many predicates will overwhelm PA engine Our use of quantified invariants precludes counterexample-

generated refinement techniques

– 36 –

Iterative Generation of PredicatesIterative Generation of PredicatesLahiri & Bryant, CAV ’04Lahiri & Bryant, CAV ’04

Generate new set of predicates if current predicates not sufficientGenerate new set of predicates if current predicates not sufficient

Generate inductive invariant using predicate abstraction over P

WP-based new

predicate generation

from P

Property proved?

P

No

Initial set of atomic predicates

from the invariant

Yes, Done

– 37 –

Case Study 1: N-BakeryCase Study 1: N-Bakery

N-Process mutual exclusion protocol N-Process mutual exclusion protocol [Lamport ’76] Each process contains

An unbounded ticket A counter with range [1…N] Other Boolean state variables

Safety: Mutual exclusion property

Constructs inductive invariant in 3 iterations of WP-Constructs inductive invariant in 3 iterations of WP-based predicate discoverybased predicate discovery1. Iteration 1: # Predicates = 1, Time to construct inv = .81s

Does not imply mutual exclusion

2. Iteration 2: # Predicates = 18, Time = 55.8s Does not imply mutual exclusion

3. Iteration 3: # Predicates = 33, Time = 471s Implies mutual exclusion

– 38 –

Case Study 2: German’s Cache ProtocolCase Study 2: German’s Cache ProtocolN-Client Directory based Cache Coherence Protocol N-Client Directory based Cache Coherence Protocol [German, IBM]

Each client contains Boolean state variables 3 single-entry channels to communicate with central process

Central “home” process contains Directory : [1..N] {0,1} of clients sharing a line Current client id [1…N] Boolean variables

Safety: Mutual exclusion property

Constructs inductive invariant in 4 iterations of WP-based Constructs inductive invariant in 4 iterations of WP-based predicate discoverypredicate discovery

1. Iteration 1: # Predicates = 4, Time (to construct inv) = 1.46 s

2. Iteration 2: # Predicates = 12, Time = 11.94 s

3. Iteration 3: # Predicates = 24, Time = 207 s

4. Iteration 4: # Predicates = 28, Time = 1266s Implies mutual exclusion

– 39 –

Extension of German’s cache protocol Extension of German’s cache protocol Each client communicates with home with unbounded Each client communicates with home with unbounded

FIFO channelsFIFO channels Unbounded number of unbounded channels

Verification complexity goes up considerablyVerification complexity goes up considerably 2 manually provided predicates for FIFOs required

Predicates involved constant offsets

Time to construct inductive invariant = 3 hours

– 40 –

Predicate Abstraction ConvergencesPredicate Abstraction Convergences

Powerful method for generating & evaluating abstract model of system

Applicable to variety of systems with different modeling levels

HardwareHardware SoftwareSoftware

Word-LevelWord-Level UCLID

Seshia, Lahiri, Bryant, CAV ‘02

SLAM

Ball, Rajamani, SPIN ‘01

Bit-LevelBit-Level Clarke, Talupar, Wang, SAT ‘03

CBMC

Kroening, Clarke, ICCAD ‘04

– 41 –

ObservationsObservations

Predicate AbstractionPredicate Abstraction Combines features of theorem proving & model checking Very general and powerful technique Lots of ways to generalize

Making More EfficientMaking More Efficient Symbolic formulation very general SAT enumeration limits capacity to ~25 predicates

Making Easier to UseMaking Easier to Use Automatic predicate discovery Limitation: Hard to find counterexamples