effective verification for do-254 projects · effective verification for do-254 projects ......

Effective Verification for DO-254 Projects

David Landoll Applications Architect May 1, 2008

DL, Effective Verificatin for DODL, Effective Verificatin for DO--254254 Projects, May Projects, May 20082008 22

Agenda

Verification Challenges and Safety-Critical Design DO-254 Requirements for Verification Safety-Critical Verification: Recommendations Conclusion

2


Verifying for Safety Critical Know that end product is customer safe Track that requirements are thoroughly verified Ensure verification processes to meet compliance to pertinent standards Keep project on schedule

3

RTL

3

XML ASCII

Avionics Integration – New Challenges

DL, Effective Verificatin for DODL, Effective Verificatin for DO--254254 Projects, May Projects, May 20082008 44 4

Flight Management

Maintenance Diagnostics

Image Processing

Integrated Avionics Processing

Flight Control & Avoidance Systems

Boeing 787: Integration’s Next Step From its central processor to its common data network, surveillance system and navigation system, the theme of the Boeing 787 Dreamliner is integration. James W. Ramsey

Communications

Weather Radar

4


Verification Challenges

5

HDL Design Files

Growing Gate Counts

Very Difficult to Verify!

50K 1M

100M

Highly Integrated Devices

O n b o a r d C o m p u t e r S y s t e m o n C h i p / F P G A

M e m o r y C o n t r o l l e r

T i m e r s IR Q C t r l

U A R T D M A C o n t r o l l e r

A H B / A P B B r i d g e

A M B A A H B

3 2 b i t D a ta b u s

D a t a M e m o r y P a r i t y M e m o r y

C A N C o n t r o l l e r

H D L C C o n t r o l l e r

S p a c e W i r eC o n t ro l l e r

A M B A A P B

F P U

H D L C C o n t r o l l e r

C A N S w i t c h

L e o n C P U

E D A C C o n t r o l l e r

B o o t P R O M

S D R A MS D R A M

A d d r e s s / C o n t ro l b u s

R S 4 2 2 S p a c e W i r eH D L C 1

U p / D o w n l i n kH D L C 0

U p /D o w n l in k C A N 0 C A N 1

D u a l C A N t r a n s c e i v e r

1 .5 V L i n e a r R e g u l a t o r

+ 3 .3 V

+ 3 .3 V + 1 .5 V

C L K G e n e r a to r

C o n f i g u r a t i o n P R O M

J T A G

I /O p o r t

P P S o u tP P S i n

DMA Controller UART

CPU AMBA AHB

Verifying Requirements (100’s, 1000’s, etc)

Increasing Design Concurrency

Many Clock Domains (Metastability)

Is Verification Complete?

Major Issues

(Exhaustively) Verifying Safety-Critical Reqs


Questions for the Verification Manager How do you know your tests really do comprehensively verify the requirements?

— Design performs its intended function

How do you ensure you’re testing the interactions between requirements (i.e., concurrency)?

— Design has no unintended functionality

How do you ensure you catch anomalous behaviors that might not be tied to requirements? How do you manage your verification effort, measure your progress, and prove that you’re done?

6

Typical Verification Manager


Agenda


7


Verification

The purpose of DO-254 is design assurance Designs must work as intended Quality verification is essential

Verifying complex designs is very challenging

8

Concept Design Concept Design

Synthesis Synthesis

Verify RTL Design

Verify RTL Design

Place & Route Place & Route

Program Device Program Device

Requirements Requirements

Planning Planning

HW Test & Debug HW Test & Debug

Why it is Important

RTL Design RTL Design

Verify Gate- Level Design Verify Gate-

Level Design

Note: In this presentation we will not be talking about testing the physical HW item, even though this is a requirement of “verification” for DO-254


Verification

Verification Independence so designer doesn’t test own code

Requirements-based test on both RTL and Gate-Level design representations (as well as end hardware item)

Traceability from Requirements to tests and results

Coverage to ensure verification is complete

Advanced Methods for level A/B projects

Reporting data for audits and management

9

What Does DO-254 Require?


Verification

An effective verification plan to drive all verification activities

Cost effective methods to ensure profitability

Resources used wisely

Metrics for monitoring progress and completion

Assurance of high quality results

Compliance to DO-254 requirements

10

What Does Your Business Require?


Agenda


11


Directed Test A Traditional Approach

A good approach for traditional design styles Manually-written tests exercise requirements via specified stimulus Testbench applies stimulus/checks results Log file includes results of test Code coverage metrics determine if tests exercise RTL code

12

Directed Tests

Traditional Testbench

Stimulus

Q Q SET

CLR S R

33 MHz

Q Q SET

CLR S R

u 1 x 2 x 1 f ( x 1 ... x n )

Element

Stimulus Results Check

Log File (1 per

Requirement)

Response

Coverage Requirements

Note: This method begins to fail with increased device complexity, integration and a large number of requirements

Traditional Coverage Limitation

These bugs exist, but are undetected Failures only appear if test propagates it to the output

DL, Effective Verificatin for DODL, Effective Verificatin for DO--254254 Projects, May Projects, May 20082008 1313 13 13

Reference Model

Lurking bugs: Missed by Traditional Coverage

Lurking bugs: Missed by Traditional Coverage

Design Under Test

•Tests Pass •100% coverage

* Code coverage * Branch coverage * Toggle coverage

Directed Tests


Evolution of Verification Methods

More complex designs can benefit from newer techniques

14

DATE: 1990’s 1995 1999 2001 2004 2007

Directed Test

Code Coverage

Testbench Automation

Directed Random

Functional Coverage

Constrained Random Coverage

Driven

Automatic Application of Formal

Clock Domain Standards SystemVerilog

PSL Properties (Assertions)

Unified Coverage Database

Open Verification

Methodology (OVM)

Formal Verification

■ Most aerospace companies use this traditional approach (directed test/code coverage)


Automating Test Stimulus vs. Directed Test

Directed tests: — Test writer must code

each specific scenario to specify intent explicitly

— Prone to overestimating completeness of testing

— Doesn’t scale with design complexity

Automated test stimulus: — Engine uses constraints and

randomness to exercise a wide variety of possible scenarios

— Completeness driven by progress towards functional coverage goals

— Scales very efficiently with design complexity

15 15

Test CR

Test

How do you build an Automated Testbench?

1616

ChooseChoose PacketPacket TypeType

Contro

l

Contro

l

DataData

ChooseChoose ControlControl TypeType

ReadRead

Request

Request

AbortAbort

Choose Choose attributesattributes ((Device, Device, Address, Address, LengthLength))

RetryRetry Device,Device, IDID

Device,Device, IDID

. . .. . .etcetc

1. Engineer encodes traffic structure and rules per requirements (Testbench) 2. SystemVerilog Simulator then chooses paths (Stimulus), per rules (if any) 3. Coverage measurements assures all paths taken per requirements

RulesRules

ValidValid ChoicesChoices

•• Must be NEW Must be NEW transactiontransaction

•• Device not busyDevice not busy •• etcetc

……etcetc

. . .. . .etcetc

……etc etc

. . .. . .etcetc

. . .. . .etcetc


Directed Test vs. Automated Test Stimulus

1 test/scenario (1 day each) Immediate progress!

Up front infrastructure 5X productivity increase!

17 17

Directed Test Automated Test Stimulus

5 verification engrs 500 requirements

Week2 Week4 Week6 Week8 Week10 Week12 Week14 Week16

Coverage Goal

50 scenarios

100 scenarios

150 scenarios

200 scenarios

250 scenarios

300 scenarios

350 scenarios

400 scenarios

450 scenarios

Add assertions and cover points

Set up CR test environment

500 scenarios


Monitoring and Covering Requirements Assertion Based Verification

Assertions are like comments that describe how the design is supposed to work (requirements) They actively monitor the design to ensure it does! Assertions provide traceability to requirements

18 18

Requirement

Assertion

Assertion Failure

property RQ62_LANDING_GEAR_LOCK; @(posedge clk) GEAR_down_notification |-> ##[1:$] Gear_down_lock_notification; endproperty cover property RQ62_LANDING_GEAR_LOCK;

“The flight crew shall be aurally warned if the gear is down but not locked”

Automated Test Generation Applied to DO-254 Modern Testbench Approach

More complete verification Requires fewer directed tests/resources Direct link back to requirements


Directed Tests

Traditional Testbench

Stimulus

Q Q SET

CLR S R

33 MHz

Q Q SET

CLR S R

u 1 x 2 x 1 f ( x 1 ... x n )

Element

Directed Test

(As needed)


Coverage Points

Log File (1 per

Requirement)

Response

Coverage

Q Q SET

CLR S R

33 MHz

Q Q SET

CLR S R

u 1 x 2 x 1 f ( x 1 ... x n )

Element


Requirements

Modern Testbench

19


Formal Methods vs. Directed Test Directed tests

— Simulation-based method that requires input stimulus

— Test writer must code a scenario that hits a bug

If stimulus doesn’t exercise a bug, the bug is missed

Formal Methods — Mathematical analysis done on RTL

--no stimulus needed Assertion provides description of requirement to be checked

— Formal engine analyzes assertion against every possible scenario (state)

Exhaustive!

20 20

*Note: Formal methods should be used in conjunction with (not as a replacement for) directed test and/or automated testing.

Test FV


Example: Formal Model Checking for DO-254 Exhaustively Verify Safety-Specific Requirements

Formal Model Checking finds all possible scenarios

— Example: enabling reverse thrusters

Unexpected paths to this situation are called “sneak paths”

— Is there any way for some event to happen other than the correct way?

How to apply: — Add an assertion stating that the event

cannot happen in implementation — Apply formal model checking — Investigate/fix all unwanted situations — Repeat process until

no unwanted paths exist

21 21

Requirement

Assertion assert always fire_reverse_thrusters |-> Gear_down_lock_notification @(clk'event and clk = '1')

Reverse thrusters shall never fire in mid-air.


Managing Verification for DO-254

Requirements-based test and traceability

Coverage

Verification Mgmt and Reporting data

22

Needs Mentor Provides the Solution

Verification activities mapped to requirements-driven test plan with links for traceability

Unified coverage database to store coverage data from a variety of sources, with a variety of metrics

Verification management facilitates reporting of progress (coverage) of requirements


Verification Management and Unified Coverage Quality, Progress and Requirements Traceability

23 23

Test Merging

Generate Coverage • Code coverage • Functional coverage • Assertion coverage


Verification with Mentor

Assertions Auto Test Stimulus Functional Coverage Verification Management and Unified Coverage

24

Advanced Methods Mentor Leads in Advanced Verification

Virtual lab for design and analysis of distributed mechatronic systems

Advanced methods can improve both safety and efficiency!

SystemVision

Formal Verification Clock-Domain Crossing

Logic Equivalency Checking

System Modeling

Assure two models are functionally equivalent

Mathematical analysis to exhaustively prove safety-critical requirements, … Check clock-domain crossings to eliminate metastability

Actively monitor adherence to requirements Automated stimulus generation to reach many more scenarios than directed test Measure coverage against design requirements Manage and report on verification progress


Agenda


25

Conclusion

Mentor can help you establish a methodology that is efficient, reusable, and certifiable

— Industry leading solutions in wide use — Supporting DO-254 objectives — Scalable methods for the simplest to the most complex safety-

critical project Applying advanced methods will:

— Improve verification efficiency and thoroughness — Reduce development costs — Improve safety of hardware systems



More Information Visit our web site: www.mentor.com/go/do-254 Here you will find numerous resources including the following verification-related publications

— “Achieving Quality and Traceability in FPGA/ASIC Flows for DO-254 Aviation Projects”

— “The Use of Advanced Verification Methods to Address DO-254 Design Assurance”

— “Effective Functional Verification Methodologies for DO-254 Level A/B and Other Safety-Critical Devices”

— “Assessing the ModelSim Tool for Use in DO-254 and ED-80 Projects” — “Automating Clock-Domain Crossing Verification for DO-254 (and other

Safety-Critical) Designs” — “DO-254 Compliant Design and Verification with VHDL-AMS”

27

effective verification for do-254 projects · effective verification for do-254 projects ......

Documents