42nd DAC, June 16, 2005

Formal Verification – Is It Real Enough? Formal Verification – Is It Real Enough?

Yaron Wolfsthal Yaron Wolfsthal

Haifa Research LabHaifa Research Lab

Rebecca GottRebecca Gott

Systems and Systems and Technology Technology GroupGroup

3

Assertion-Based VerificationAssertion-Based Verification

““How can one check a large routine in the senseHow can one check a large routine in the sensethat it's right?that it's right?

… … make a number of definite make a number of definite assertionsassertions which which can becan be

checkedchecked individually, and from which the individually, and from which the correctnesscorrectness

of the whole program easily follows.”of the whole program easily follows.”

Alan Turing, 24 June 1950 Alan Turing, 24 June 1950 "Checking a Large Routine”"Checking a Large Routine”

2005: IEEE P1850 PSL -- standard assertion language

4

What Are the Options?What Are the Options?

Dynamic assertion checkingDynamic assertion checking– On top of NC-Sim, VCS, ModelSim…On top of NC-Sim, VCS, ModelSim…– Relatively easy to do – and to argue forRelatively easy to do – and to argue for

Static assertion checking Static assertion checking – Formal Verification – Formal Verification

5

PositionPosition

Formal Verification is absolutely for real Formal Verification is absolutely for real – here and nowhere and now

Carries unique, significant benefits and ROI Carries unique, significant benefits and ROI

(without PhDs)(without PhDs)

Insights based on many projects, ~10 yearsInsights based on many projects, ~10 years– IBM servers, game processors, ASICsIBM servers, game processors, ASICs– Customer projectsCustomer projects

6

Switch Chip I, IBM High-End System Switch Chip I, IBM High-End System

11/06/98 03/26/99 08/13/99 12/31/99 05/19/00 10/06/00

Week Ending

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

240

250

Cu

mu

lati

ve

Issu

es

Chip Sim

Gate Sim

Designer Sim

RuleBase

System Sim

Element Sim

Walk Thru

Lab

Chip Sim II

Timing

Verification Effectiveness on Switch ChipsProblems Found By Method

7

Switch Chip II, IBM High-End SystemSwitch Chip II, IBM High-End System

04

/09

04

/16

04

/23

04

/30

05

/07

05

/14

05

/21

05

/28

06

/04

06

/11

06

/18

06

/25

07

/02

07

/09

07

/16

07

/23

07

/30

08

/06

08

/13

08

/20

Week

0

10

20

30

40

50

Issu

es

Rulebase

Chip Sim

Elem Sim

Gate Sim

Issue Discovery Area

1838.3%

612.8%

24.3%

24.3%

1225.5%

12.1%

12.1%

510.6%

PortMgtOutput PortInput PortCQCentrlMgtCQMgtXBarLPE

Chip Sim Bug Discoveries

PortMgt1

Output Port19

CentrlMgt8

PortMgtOutput PortInput PortCQCentrlMgtCQMgtXBarLPE

RuleBase Bug Discoveries

8

0

2

4

6

8

10

12

148

/4/2

002

8/1

8/20

02

9/1

/200

29

/15/

200

29

/29/

200

21

0/13

/20

021

0/27

/20

021

1/10

/20

021

1/24

/20

02

12/

8/20

02

12/

22/2

002

1/5

/200

31

/19/

200

3

2/2

/200

32

/16/

200

3

3/2

/200

33

/16/

200

33

/30/

200

3

4/1

3/20

03

4/2

7/20

03

5/1

1/20

03

5/2

5/20

03

6/8

/200

36

/22/

200

3

7/6

/200

37

/20/

200

3

Weeks

#B

ug

s

FV

nonFV

integration discovered by FV

Data collected from IBM Ethernet Core Project

IBM Ethernet CoreIBM Ethernet Core

9

Benefits of Formal VerificationBenefits of Formal Verification

Early AvailabilityEarly Availability

High CoverageHigh Coverage

Enabling effective integrationEnabling effective integration

Quality, productivity/schedule and cost gains Quality, productivity/schedule and cost gains

10

Unit sizeUnit size

Number of assertionsNumber of assertions

Amount of resources Amount of resources

Extent of logic coveredExtent of logic covered

Some Usage NumbersSome Usage Numbers

11

FV of a Communication CoreFV of a Communication Core

BlocBlockk

ScheduleScheduledd

TimeTime

Actual Actual TimeTime

BugsBugs SizeSize InterfacesInterfaces

In OutIn Out

B1B1 120H120H 160-160-170170

1515 3029330293 21 4821 48

B2B2 180H180H 163163 66 2738427384 25 2725 27

B3B3 100H100H 5656 33 62746274 13 2113 21

B4B4 100H100H 8585 -- 1628616286 29 1429 14

B5B5 80H80H 7070 22 1625716257 23 1523 15

B6B6 100H100H 161161 44 1995919959 14 1814 18

B7B7 80H80H 106106 11 1834518345 17 817 8

12

IBM eServer™ p690 (Power4™)IBM eServer™ p690 (Power4™)

"We applied FV to some extent on approximately "We applied FV to some extent on approximately 40 design components40 design components throughout the processor and throughout the processor and found more than found more than 200 design flaws at various stages 200 design flaws at various stages and of varying complexityand of varying complexity. . At least one bug was found by almost every application At least one bug was found by almost every application of FV. of FV. In most cases, FV began significantly later than In most cases, FV began significantly later than verification. verification. It is estimated that 15% of these bugs were of It is estimated that 15% of these bugs were of extreme complexity and would have been difficult extreme complexity and would have been difficult for traditional verification.for traditional verification. In some cases, a late bug found in verification or in the In some cases, a late bug found in verification or in the laboratory was recreated and its correction verified laboratory was recreated and its correction verified efficiently with FV."efficiently with FV."Ludden et al.,IBM Journal of R&D 46(1), 2002

13

IBM Engineering Services UnitExperience ReportIBM Engineering Services UnitExperience Report Formal VerificationFormal Verification of an average logic of an average logic

modulemodule– Requires up to a monthRequires up to a month– Involves the development / debugging of 50-Involves the development / debugging of 50-

70 assertions70 assertions– Consumes 20% of designer’s time for Consumes 20% of designer’s time for

supporting the worksupporting the work– Designer can realistically support 3-5 modulesDesigner can realistically support 3-5 modules

– Involves running 5 – 10 assertions concurrentlyInvolves running 5 – 10 assertions concurrently– Consumes few CPUsConsumes few CPUs– Scales up with additional CPUs Scales up with additional CPUs

In an average project, some 40% of the In an average project, some 40% of the logic modules are formally verified with logic modules are formally verified with RuleBaseRuleBase

Source: DAC’04 / PSL Consortium Source: DAC’04 / PSL Consortium MeetingMeeting

14

Formal Verification of Gigabit Ethernet Core, 2002-2003Formal Verification of Gigabit Ethernet Core, 2002-2003 400,000400,000 gates gates

40%40% of logic went through Formal ABV of logic went through Formal ABV

Formal ABV practiced by Formal ABV practiced by 3 engineers3 engineers out of out of a team of 10a team of 10

Formal ABV found Formal ABV found 33%33% of documented of documented design bugsdesign bugs

ZeroZero bugs found in logic that went through bugs found in logic that went through Formal ABV Formal ABV

Late Formal ABV found bugs in areas that Late Formal ABV found bugs in areas that were heavily simulatedwere heavily simulated

IBM Microelectronics, Haifa Design Centerhttp://www/pslsugar.org/papers/ABV-in-IBM-Haifa.pdf

15

Bug Classification Bug Classification

Bugs found due to schedule advantage Bugs found due to schedule advantage

Holes in simulation coverageHoles in simulation coverage

True corner casesTrue corner cases

Performance bugsPerformance bugs

Impossible bugsImpossible bugs

DeadlocksDeadlocks

16

Position cont.Position cont.

Formal Verification is for real, here and now, Formal Verification is for real, here and now, and it carries unique, significant benefits and and it carries unique, significant benefits and ROI ROI (without PhDs)(without PhDs)

BUTBUT

There can be some “Hindering Factors”There can be some “Hindering Factors”– The state-space explosionThe state-space explosion– Soft considerationsSoft considerations

17

“Hindering Factors”“Hindering Factors”

The state-space explosionThe state-space explosion

Soft considerationsSoft considerations– Perception of cost and difficultyPerception of cost and difficulty– ““What we have is Good Enough”What we have is Good Enough”

18

Hindering Factors : State-Space Explosion

Hindering Factors : State-Space Explosion

Need to have proper methodology in placeNeed to have proper methodology in place– Divide and conquerDivide and conquer– Falsify vs VerifyFalsify vs Verify

Must have battle-hardened technology and Must have battle-hardened technology and toolstools– Different designs require different search Different designs require different search

strategies, and the tool must be able to strategies, and the tool must be able to transparently support ittransparently support it

– Strength alongside versatilityStrength alongside versatility

Ballpark: 1000s (formal) Ballpark: 1000s (formal) 10000s 10000s (semiformal)(semiformal)

19

Hindering Factors cont : Perception

Hindering Factors cont : Perception

Assertions, and Formal Verification is Assertions, and Formal Verification is perceived as a “difficult” and “costly” perceived as a “difficult” and “costly” techniquetechnique

However, concrete data suggests otherwiseHowever, concrete data suggests otherwise1.1. Undergraduates routinely employed in FV Undergraduates routinely employed in FV

projectsprojects– Cf. DAC’04 / PSL Consortium MeetingCf. DAC’04 / PSL Consortium Meeting

http://www.pslsugar.org/papers/pm2_EyalGonenDAC04.pdfhttp://www.pslsugar.org/papers/pm2_EyalGonenDAC04.pdf

2.2. DeepChip survey on assertions DeepChip survey on assertions 3.3. FV learning curve survey by IBMFV learning curve survey by IBM

20

Proliferation of AssertionsProliferation of Assertions

21

User Assessment of “The Formal Verification Learning Curve”

User Assessment of “The Formal Verification Learning Curve”

61.6

68.8

65.6

100

90.4

30.4

25.6

97

0 10 20 30 40 50 60 70 80 90 100

Maintenance of properties

Conceiving properties

Maintenance of constraints

Writing constraints to cope w/size

Modifying design to cope w/size

Using non-determinism

Learning PSL

Design understanding

22

Hindering Factor cont : “Principle of Good Enough”

Hindering Factor cont : “Principle of Good Enough”

SatisficingSatisficing - People will tend to make - People will tend to make choices based on their most important choices based on their most important currentcurrent needs rather than through a rational needs rather than through a rational process. process.

Engineers typically look at Engineers typically look at various constraints and find trade-offs various constraints and find trade-offs to try to meet all requirements to try to meet all requirements “well enough” to allow the product “well enough” to allow the product to be builtto be built

A new way of thinking, which would A new way of thinking, which would require some upfront investment, require some upfront investment, requires some activation energyrequires some activation energy

Herbert SimonHerbert SimonTuring Award, 1975Turing Award, 1975

Bank of Sweden Prize Bank of Sweden Prize in Economic Sciencesin Economic Sciences

in Memory of in Memory of Alfred Nobel, 1978Alfred Nobel, 1978

23

Recommendations for DeploymentRecommendations for Deployment

Good engineering practicesGood engineering practices

(see proceedings)(see proceedings)

24

Position SummaryPosition Summary

Formal Verification is for real, here and now, Formal Verification is for real, here and now, and it carries unique, significant benefits and and it carries unique, significant benefits and ROI ROI (without PhDs)(without PhDs)

BUTBUT

There can no shortcutsThere can no shortcuts– Need a solid Need a solid technologytechnology foundation foundation– Same for Same for methodologymethodology– Management Management commitmentcommitment

25

Complex chips, challenging verification problems?

Welcome to the club of Formal Verification

Thank You

26

Backup SlidesBackup Slides

27

Formal Verification:Myths and FactsFormal Verification:Myths and Facts Doing Formal Verification requires PhDsDoing Formal Verification requires PhDs

Only small designs can be subjected to FVOnly small designs can be subjected to FV

FV is a push-button technologyFV is a push-button technology

FV is predictable FV is predictable

FV is beneficial, helps increase quality, improve FV is beneficial, helps increase quality, improve TTM TTM

Introducing FV into the design flow is a strategic Introducing FV into the design flow is a strategic decision that needs investment as and decision that needs investment as and commitment from managementcommitment from management

Once you have found some good corner-case Once you have found some good corner-case bugs, the designers become your captive bugs, the designers become your captive audience!audience!

28

Technology FoundationTechnology Foundation

Formal Verification is about analyzing very Formal Verification is about analyzing very large state spaces – tools must be battle-large state spaces – tools must be battle-hardenedhardened– AlgorithmsAlgorithms (optimization of FSMs, fast search) (optimization of FSMs, fast search) – Parallel computationsParallel computations

– Assuring horsepower as well as versatilityAssuring horsepower as well as versatility

Corporate CAD, focused at solving the real-Corporate CAD, focused at solving the real-life formal verification problems, is a winning life formal verification problems, is a winning strategystrategy

29

Methodology FoundationMethodology Foundation

Design for FVDesign for FV

Involve FV engineers in spec process from Involve FV engineers in spec process from early stagesearly stages

Define the verification targets – unit function Define the verification targets – unit function and sizeand size

Determine verification strategy – assurance Determine verification strategy – assurance vs bug huntingvs bug hunting

FV ReviewsFV Reviews

Develop properties systematicallyDevelop properties systematically

Etc..Etc..

30

Management CommitmentManagement Commitment

A critical factor of successA critical factor of success

Invest-first, harvest laterInvest-first, harvest later

Identify focal pointIdentify focal point

Leverage existing interest, experienceLeverage existing interest, experience

Initiation and educationInitiation and education