turing award lecture

8/8/2019 Turing Award Lecture

1/57

Model CheckingModel Checking

My 27 year quest to overcome theMy 27 year quest to overcome thestate explosion problemstate explosion problem

Edmund ClarkeEdmund Clarke

Computer Science DepartmentComputer Science Department

Carnegie Mellon UniversityCarnegie Mellon University


2/57

Intel Pentium FDIV BugIntel Pentium FDIV Bug

Try 4195835 4195835 / 3145727 * 3145727.Try 4195835 4195835 / 3145727 * 3145727. In 94 Pentium, it doesnt return 0, but 256.In 94 Pentium, it doesnt return 0, but 256.

Intel uses the SRT algorithm for floating point division.Intel uses the SRT algorithm for floating point division.

Five entries in the lookup table are missing.Five entries in the lookup table are missing. Cost: $500 millionCost: $500 million

Xudong Zhaos Thesis on Word Level Model CheckingXudong Zhaos Thesis on Word Level Model Checking


3/57

Recent Rumor: New AMD TLBRecent Rumor: New AMD TLBBug??Bug??

AMD Family 10h revision B2 processors suffer from an issue in theAMD Family 10h revision B2 processors suffer from an issue in theprocessor TLBprocessor TLB ((TranslationTranslation LLookasideookaside BBufferuffer).).

Launch date of these pLaunch date of these processorsrocessors was delayed in September, 2007.was delayed in September, 2007.

AMD doesnt have official announcement yet, but you can google AMDAMD doesnt have official announcement yet, but you can google AMDBBarcelona bugarcelona bug for plenty of discussion. for plenty of discussion.


4/57

Temporal Logic ModelTemporal Logic ModelCheckingChecking

Model checking is anModel checking is an automatic verification techniqueautomatic verification techniquefor finite state concurrent systems.for finite state concurrent systems.

Developed independently byDeveloped independently by Clarke and EmersonClarke and Emerson andandbyby Queille and SifakisQueille and Sifakisin early 1980s.in early 1980s.

SpecificationsSpecifications are written inare written in propositional temporalpropositional temporallogiclogic..

Verification procedure is anVerification procedure is an exhaustive search of theexhaustive search of thestate spacestate space of the design.of the design.


5/57

Advantages of ModelAdvantages of ModelCheckingChecking

No proofs!!!No proofs!!!

Fast (compared to other rigorous methods such asFast (compared to other rigorous methods such astheorem proving)theorem proving)

Diagnostic counterexamplesDiagnostic counterexamples

No problem with partial specificationsNo problem with partial specifications

Logics can easily express many concurrency propertiesLogics can easily express many concurrency properties


6/57


7/57

Main Disadvantage Contd.Main Disadvantage Contd.

1

2

3

a

b

c

|| n states,m threads

1,a

2,a 1,b

2,b3,a 1,c

3,b 2,c

3,c

nm states


8/57

Main Disadvantage Contd.Main Disadvantage Contd.

State Explosion ProblemState Explosion Problem::

navoidable in worst case, but steady progress over the past 27

ars using clever algorithms, data structures, and engineering


9/57

Determines Patterns on Infinite TracesDetermines Patterns on Infinite Traces

Atomic PropositionsAtomic Propositions

Boolean OperationsBoolean Operations

Temporal operatorsTemporal operators

aa a is true nowa is true now

X aX a a is true in the nea is true in the neXXt statet stateFaFa a will be true in thea will be true in the FFutureutureGaGa a will bea will be GGlobally true in the futurelobally true in the futurea U ba U b a will hold truea will hold true UUntil b becomes truentil b becomes true

LTL - Linear Time LogicLTL - Linear Time Logic

a


10/57






X aX a a is true in the neXt statea is true in the neXt stateFaFa a will be true in thea will be true in the FFutureutureGaGa a will bea will be GGlobally true in the futurelobally true in the futurea U ba U b a will hold truea will hold true UUntil b becomes truentil b becomes true


a


11/57






X aX a a is true in the nea is true in the neXXt statet stateFaFa a will be true in the Futurea will be true in the FutureGaGa a will bea will be GGlobally true in the futurelobally true in the futurea U ba U b a will hold truea will hold true UUntil b becomes truentil b becomes true


a


12/57






X aX a a is true in the nea is true in the neXXt statet stateFaFa a will be true in thea will be true in the FFutureutureGaGa a will be Globally true in the futurea will be Globally true in the futurea U ba U b a will hold truea will hold true UUntil b becomes truentil b becomes true


a a a a a


13/57






X aX a a is true in the nea is true in the neXXt statet stateFaFa a will be true in thea will be true in the FFutureutureGaGa a will bea will be GGlobally true in the futurelobally true in the futurea U ba U b a will hold true Until b becomes truea will hold true Until b becomes true


a a a a b


14/57

Branching TimeBranching Time


15/57

CTL: Computation Tree LogicCTL: Computation Tree Logic

EFg g will possibly become true


16/57


AFg g will necessarily become true


17/57


AGg g is an invariant


18/57


EGg g is a potential invariant


19/57


CTL uses the temporal operatorsCTL uses the temporal operators

AX, AG, AF, AUAX, AG, AF, AU

EX, EG, EF, EUEX, EG, EF, EU

CTL*CTL* allows complex nestings such asallows complex nestings such as

AXX, AGX, EXF, ...AXX, AGX, EXF, ...

CTL: linear model checking algorithm !CTL: linear model checking algorithm !


20/57

Model Checking ProblemModel Checking Problem

LetLet MMbe abe a state-transition graphstate-transition graph..

LetLet be thebe the specificationspecification in temporal logic.in temporal logic.

Find all statesFind all states ss ofofMMsuch thatsuch that M, s |= M, s |= ..

CTL Model Checking: CE 81; CES 83/86; QS 81/82.CTL Model Checking: CE 81; CES 83/86; QS 81/82. LTL Model Checking: LP 85.LTL Model Checking: LP 85. Automata Theoretic LTL Model Checking: VW 86.Automata Theoretic LTL Model Checking: VW 86. CTL* Model Checking: EL 85.CTL* Model Checking: EL 85.

CTL Model Checking: CE 81; CES 83/86; QS 81/82.CTL Model Checking: CE 81; CES 83/86; QS 81/82. LTL Model Checking: LP 85.LTL Model Checking: LP 85. Automata Theoretic LTL Model Checking: VW 86.Automata Theoretic LTL Model Checking: VW 86. CTL* Model Checking: EL 85.CTL* Model Checking: EL 85.


21/57

State-transition graphdescribes system evolvingover time.

Model of computationModel of computation

~ Start

~ Close

~ Heat

~ Error

Start

~ Close

~ Heat

Error

~ Start

Close

~ Heat

~ Error

~ Start

Close

Heat

~ Error

Start

Close

Heat

~ Error

Start

Close

~ Heat

~ Error

Start

Close

~ Heat

Error

Microwave Oven Example


22/57

Temporal Logic and ModelTemporal Logic and ModelCheckingChecking

The oven doesntThe oven doesnt heat upheat up until theuntil the door is closeddoor is closed..

NotNotheat_upheat_up holdsholds untiluntildoor_closeddoor_closed

((~~heat_upheat_up))UUdoor_closeddoor_closed


23/57

Transition System(Automaton, Kripke structure)

Hardware Description(VERILOG, VHDL, SMV)

InformalSpecification

Temporal Logic Formula(CTL, LTL, etc.)

compilation

manualalgorithmic

verification

Model CheckingModel Checking


24/57

Hardware Example: IEEEHardware Example: IEEEFuturebusFuturebus++

In 1992 we used Model Checking to verify theIn 1992 we used Model Checking to verify the IEEEIEEEFuture+ cache coherence protocolFuture+ cache coherence protocol..

Found a number ofFound a number ofpreviously undetected errorspreviously undetected errors in thein thedesign.design.

First time that formal methods were used to findFirst time that formal methods were used to finderrors in anerrors in an IEEE standardIEEE standard..

Development of the protocol began inDevelopment of the protocol began in 19881988, but, butprevious attempts to validate it were informal.previous attempts to validate it were informal.


25/57

Symbolic Model CheckingSymbolic Model Checking

Burch, Clarke, McMillan, Dill, and Hwang 90;Burch, Clarke, McMillan, Dill, and Hwang 90;

Ken McMillans thesis 92Ken McMillans thesis 92

The Partial Order ReductionThe Partial Order Reduction

Valmari 90Valmari 90

Godefroid 90Godefroid 90

Peled 94Peled 94

Four Big Breakthroughs onFour Big Breakthroughs onState Space Explosion Problem!State Space Explosion Problem!


26/57

Four Big Breakthroughs on StateFour Big Breakthroughs on StateSpace Explosion Problem (Cont.)Space Explosion Problem (Cont.)

BoundedBoundedModel CheckingModel Checking

Biere, Cimatti, Clarke, Zhu 99Biere, Cimatti, Clarke, Zhu 99

Using Fast SAT solversUsing Fast SAT solvers

Can handle thousandsCan handle thousands

of state elementsof state elements

Can the given property fail in k-steps?

I(V0) T(V0,V1) T(Vk-1 ,Vk) (: P(V0) :

P(Vk))

Can the given property fail in k-steps?

I(V0) T(V0,V1) T(Vk-1 ,Vk) (: P(V0) :

P(Vk)) k-steps Property failsin some stepInitial state

BMC in practice: Circuit with 9510 latches, 9499 inputsBMC formula has 4 106 variables, 1.2 107 clausesShortest bug oflength 37 found in 69 seconds

Four Big Breakthroughs onFour Big Breakthroughs on


27/57

Four Big Breakthroughs onFour Big Breakthroughs onState Space Explosion ProblemState Space Explosion Problem(Cont.)(Cont.)

Localization ReductionLocalization Reduction Bob Kurshan 1994Bob Kurshan 1994

Counterexample Guided Abstraction Refinement (CEGAR)Counterexample Guided Abstraction Refinement (CEGAR) Clarke, Grumberg, Jha, Lu, Veith 2000Clarke, Grumberg, Jha, Lu, Veith 2000

Used in most software model checkersUsed in most software model checkers


28/57

From Hardware to Software:From Hardware to Software:

Natural Question: Is it possible to model checkNatural Question: Is it possible to model checksoftware?software?

According toAccording to Wired NewsWired News on Nov 10, 2005:on Nov 10, 2005:

When Bill Gates announced that the technologyWhen Bill Gates announced that the technologywas under development at the 2002 Windowswas under development at the 2002 WindowsEngineering Conference, he called it the holyEngineering Conference, he called it the holygrail of computer sciencegrail of computer science


29/57

Grand Challenge:Grand Challenge:Model Check Software !Model Check Software !

What makesSoftware Model CheckingSoftware Model Checking

different ?


30/57

What Makes Software ModelWhat Makes Software ModelChecking Different ?Checking Different ?

Large/unbounded base types:Large/unbounded base types: int, float, stringint, float, string

User-defined types/classesUser-defined types/classes

Pointers/aliasing + unbounded #s of heap-allocated cellsPointers/aliasing + unbounded #s of heap-allocated cells Procedure calls/recursion/calls through pointers/dynamic methodProcedure calls/recursion/calls through pointers/dynamic method

lookup/overloadinglookup/overloading

Concurrency + unbounded #s of threadsConcurrency + unbounded #s of threads


31/57

What Makes Software ModelWhat Makes Software ModelChecking Different ?Checking Different ?

Templates/generics/include filesTemplates/generics/include files

Interrupts/exceptions/callbacksInterrupts/exceptions/callbacks

Use of secondary storage: files, databasesUse of secondary storage: files, databases Absent source code for: libraries, system calls, mobile codeAbsent source code for: libraries, system calls, mobile code

Esoteric features: continuations, self-modifying codeEsoteric features: continuations, self-modifying code

Size (e.g., MS Word = 1.4 MLOC)Size (e.g., MS Word = 1.4 MLOC)


32/57

What Does It Mean to Model CheckWhat Does It Mean to Model CheckSoftware?Software?

1.1. Combine static analysis and model checkingCombine static analysis and model checking UseUse static analysisstatic analysisto extract ato extract a model Kmodel Kfrom a booleanfrom a boolean

abstraction of the program.abstraction of the program.

Then check that f is true in K (KThen check that f is true in K (K f), where f is thef), where f is thespecification of the program.specification of the program.

SLAM (Microsoft)SLAM (Microsoft) Bandera (Kansas State)Bandera (Kansas State) MAGIC, SATABS (CMU)MAGIC, SATABS (CMU) BLAST (Berkeley)BLAST (Berkeley) F-Soft (NEC)F-Soft (NEC)


33/57


2.2. Simulate program along all paths inSimulate program along all paths incomputation treecomputation tree

Java PathFinder (NASA Ames)Java PathFinder (NASA Ames)

Source code + backtracking (e.g., Verisoft)Source code + backtracking (e.g., Verisoft)Source code + symbolic execution + backtrackingSource code + symbolic execution + backtracking

(e.g., MS/Intrinsa Prefix)(e.g., MS/Intrinsa Prefix)

3.3. Use finite-state machine to look for patternsUse finite-state machine to look for patterns

in control-flow graphin control-flow graph [Engler][Engler]


34/57


4.4. Design with Finite-State Software ModelsDesign with Finite-State Software ModelsFinite state software models can act as missing linkFinite state software models can act as missing link

between transition graphs and complex software.between transition graphs and complex software.

StatechartsStatechartsEsterelEsterel


35/57


5.5. Use Bounded Model Checking and SATUse Bounded Model Checking and SAT [Kroening][Kroening]

Problem: How to compute set of reachable states?Problem: How to compute set of reachable states?

Fixpoint computation is too expensive.Fixpoint computation is too expensive.

Restrict search to states that are reachable from initialRestrict search to states that are reachable from initial

state withinstate within fixed numberfixed numbern of transitionsn of transitions

Implemented byImplemented by unwindingunwinding program and usingprogram and using

SAT solverSAT solver


36/57

Key techniques for Software ModelKey techniques for Software ModelCheckingChecking

Counterexample Guided Abstraction RefinementCounterexample Guided Abstraction Refinement

- Kurshan, Yuan Lu, Clarke et al JACM, Ball et al- Kurshan, Yuan Lu, Clarke et al JACM, Ball et al

- Uses- Uses counterexamplescounterexamples to refine abstractionto refine abstraction

Predicate AbstractionPredicate Abstraction

- Graf and Saidi, Ball et al, Chaki et al, Kroening- Graf and Saidi, Ball et al, Chaki et al, Kroening

- Keeps track of- Keeps track ofcertain predicates on datacertain predicates on data

-- Captures relationship between variablesCaptures relationship between variables


37/57

Transition System

InformalSpecification


Safety Property:bad state unreachable:

satisfied

Initial State

CounterexamplesCounterexamples

Program


38/57

Transition System

ProgramInformalSpecification


Initial State

Safety Property:bad state unreachable

Counterexample



39/57

Transition System

ProgramInformalSpecification


Initial State

Safety Property:bad state unreachable


Counterexample


40/57

Existential AbstractionExistential Abstraction

M

M

Given an abstraction function : S S

, the concrete

states are grouped and mapped into abstract states :

Preservation Theorem ?


41/57

Preservation TheoremPreservation Theorem

Theorem (Clarke, Grumberg, Long)Theorem (Clarke, Grumberg, Long)If property holds onIf property holds onabstract modelabstract model, it holds on, it holds on concrete modelconcrete model

Technical conditionsTechnical conditions Property is universal i.e., no existential quantifiersProperty is universal i.e., no existential quantifiers Atomic formulas respect abstraction mappingAtomic formulas respect abstraction mapping

Converse implication is not valid !Converse implication is not valid !


42/57

Spurious BehaviorSpurious Behavior

AGAF redEvery path necessarily leadsback to red.

Spurious Counterexample: ...

red

go

Artifact of the abstraction !

H t d fi Ab t tiHo to define Abstraction


43/57

How to define AbstractionHow to define AbstractionFunctions?Functions?

Abstraction too fineAbstraction too fine State ExplosionState Explosion

Abstraction too coarseAbstraction too coarse Information LossInformation Loss

AutomaticAutomatic Abstraction MethodologyAbstraction Methodology


44/57

Automatic AbstractionAutomatic Abstraction

MOriginal Model

Refinement

Refinement

M Initial AbstractionSpurious

Spurious

counterexample

Validation or

Counterexample Correct !

CEGARCEGAR


45/57

CEGARCEGARCCounterounterEExample-xample-GGuideduided AAbstractionbstractionRRefinementefinement

C

Program

C

Program

InitialInitial

AbstractionAbstraction

Simulator

No errorNo error

or bug foundor bug found

PropertyPropertyholdsholds

SimulationSimulationsucessfulsucessful

Bug foundBug found

Abstraction refinementAbstraction refinement Refinement

Model

Checker

VerificationVerification

Spurious counterexampleSpurious counterexample

CounterexampleCounterexample

AbstractModel

S ft E l D i D iS ft E l D i D i


46/57

Software Example: Device DriverSoftware Example: Device DriverCodeCode

Also according toAlso according to Wired NewsWired News::

Microsoft has developed a tool called Static DeviceMicrosoft has developed a tool called Static DeviceVerifier or SDV, that uses Verifier or SDV, that uses Model CheckingModel Checking to toanalyze the source code for Windows drivers andanalyze the source code for Windows drivers andsee if the code that the programmer wrote matches asee if the code that the programmer wrote matches amathematical model of what a Windows device drivermathematical model of what a Windows device driver

should do. If the driver doesnt match the model, theshould do. If the driver doesnt match the model, theSDV warns that the driver might contain a bug.SDV warns that the driver might contain a bug.


47/57

Back to Hardware!Back to Hardware!

Ease of design

increases

Gate level (netlists)

Register Level

System

Behavioral

Formal verification

support

Gate Level (netlist):


48/57

Register Level Verilog:

module counter_cell(clk, carry_in,carry_out);

input clk;input carry_in;output carry_out;reg value;assign carry_out = value & carry_in;initial value = 0;

always @(posedge clk) begin// value =(value + carry_in) % 2; case(value)

0: value = carry_in;1: if(carry_in ==0)

value = 1; else value = 0; endcaseendendmodule

Gate Level (netlist):

.model counter_cell

.inputs carry_in

.outputs carry_out

.names value carry_in _n2

.def 01 1 1.names _n2 carry_out$raw_n1- =_n2.names value$raw_n30.names _n60.names value _n6 _n7

.def 00 1 11 0 1.r value$raw_n3 value0 01 1

.. (120 lines)


49/57

Lack of verification supportLack of verification support

Gate level (netlists)

Register Level

System

Behavioraluse techniquesfromsoftwareverification

Must be automaticand scalable!!

Model Checking at the RegisterModel Checking at the Register


50/57

Model Checking at the RegisterModel Checking at the RegisterLevelLevel

Gatelevel(netlists)

Register Level

System

Behavioral

Model check

Abstraction Refinement loopAbstraction Refinement loop


51/57

Abstraction-Refinement loopAbstraction-Refinement loop(CEGAR)(CEGAR)

C

Program

C

Program

InitialInitial

AbstractionAbstraction

Simulator

No errorNo error

or bug foundor bug found

PropertyPropertyholdsholds

SimulationSimulation

sucessfulsucessful

Bug foundBug found

Abstraction refinementAbstraction refinement Refinement

Model

Checker

VerificationVerification

Spurious counterexampleSpurious counterexample

CounterexampleCounterexample

AbstractModel


52/57

BenchmarksBenchmarks

Ethernet MAC from opencores.orgEthernet MAC from opencores.org 5000 lines of RTL Verilog5000 lines of RTL Verilog

Checked three properties:

1. Transmit module simulatesstate machine on left. (ETH0)

2. Checks transitions out of stateBackOff (ETH1)

3. Checks transitions out of stateJam (ETH2)

Defer

IPG Preamble

Data0BackOff Jam

Data1

FCS PAD

Idle

Transmit Module In Ethernet MAC

(self-loop on each state not shown)


53/57

Experimental ResultsExperimental Results

Benchmark Latches Time (sec) #Preds #Iters

ETH0 359 44 21 55

ETH1 359 127 93 51

ETH2 359 161 94 111


54/57

Challenges for the FutureChallenges for the Future

Exploiting the Power ofExploiting the Power ofSATSAT, Satisfiability Modulo Theories (, Satisfiability Modulo Theories (SMTSMT))

Compositional Model CheckingCompositional Model Checking of both Hardware and Softwareof both Hardware and Software

Software Model CheckingSoftware Model Checking, Model Checking and, Model Checking and Static AnalysisStatic Analysis

Verification of Embedded SystemsVerification of Embedded Systems (Timed and Hybrid Automata)(Timed and Hybrid Automata)

Model Checking and Theorem ProvingModel Checking and Theorem Proving (PVS, STEP, SyMP, Maude)(PVS, STEP, SyMP, Maude)

ProbabilisticProbabilistic andand StatisticalStatistical Model CheckingModel Checking

InterpretingInterpreting CounterexamplesCounterexamples

Scaling upScaling up even more!!even more!!

My goal:My goal:


55/57

My goal:My goal:

Verification of Safety-Critical EmbeddedVerification of Safety-Critical Embedded

SystemsSystems

Do you trust your car?Do you trust your car?

Embedded Systems are as important in Europe asComputer Security is in the U.S.!

, - ,, - ,Vi itVisitors


56/57

, ,, ,VisitorsVisitorsPh.D. Students:Ph.D. Students:

Sergey BerezinSergey Berezin

Michael BrowneMichael Browne

Jerry BurchJerry Burch

Sergio CamposSergio Campos

Sagar ChakiSagar Chaki

Pankaj ChauhanPankaj Chauhan

David DillDavid Dill

Allen EmersonAllen Emerson

Alex GroceAlex Groce

Anubhav GuptaAnubhav Gupta

Vicki Hartonas-GarmhausenVicki Hartonas-Garmhausen

Himanshu JainHimanshu Jain

Sumit JhaSumit Jha

William KlieberWilliam Klieber

David LongDavid Long

Yuan LuYuan Lu

Dong WangDong Wang

Will MarreroWill Marrero

Ken McMillanKen McMillan

Marius MineaMarius Minea

Bud MishraBud Mishra

Christos NikolaouChristos Nikolaou

Nishant SinhaNishant Sinha

Prasad SistlaPrasad Sistla

Muralidhar TalupurMuralidhar Talupur

Xudong ZhaoXudong Zhao

Post-docs:

Constantinos Bartzis

Armin Biere

Lei Bu

David Deharbe

Alexandre Donze

Azadeh Farzan

Ansgar Fehnker

Wolfgang Heinle

Tamir Heyman

James Kapinski

Daniel Kroening

Axel Legay

Daniel Milam

Alaexandar Nanevski

Joel Ouaknine

Karsten Schmidt

Subash Shankar

Ofer Strichman

Prasanna Thati Micheal Theobald

Tayssir Touili

Helmut Veith

Silke Wagner

Karen Yorav

Haifeng Zhu

Yunshan Zhu

Visitors:

Y. Chen

Y. Feng

T. Filkorn

M. Fujita

P. Granger

O. Grumberg

H. Hamaguchi

H. Hiraishi

S. Kimura

S. Krischner

G.H. Kwon

X. Li A. Platzer

R. Raimi

H. Schlingloff

S. Shanker

Y.Q. Sun

T. Tang

F. Tiplea Y. Tsay

J.P. Vidal

B. Wang

F. Wang

P. Williams

W. Windsteiger

Kwang Yi

T. Yoneda


57/57

Questions?Questions?

turing award lecture

Documents