1

New Development Techniques:New Challenges for Verification and Validation

Mats HeimdahlCritical Systems Research GroupDepartment of Computer Science and Engineering

University of Minnesota

4-192 EE/CS; 200 Union Street SE

Minneapolis, MN 55455

http://ww

w.cs.um

n.edu/crisys

2

Domain of Concern

http://ww

w.cs.um

n.edu/crisys

3

How we Develop SoftwareConcept

Formation

Requirements Specification

Design

Implementation

Integration

System

Unit Test

Integration Test

System Test

Object Code

Test

Analysis

http://ww

w.cs.um

n.edu/crisys

4

Validation and VerificationConcept

Formation

Requirements Specification

Design

Implementation

Integration

Verification:Are we building the thing right?

Validation:Are we building the right thing?

System

http://ww

w.cs.um

n.edu/crisys

5

Model-Based Development

SpecificationModel

Visualization PrototypingTesting

Code

Analysis

Properties

http://ww

w.cs.um

n.edu/crisys

6

Model-Based Development Tools

• Commercial Products Esterel Studio and

SCADE Studio from Esterel Technologies

SpecTRM from Safeware Engineering

Rhapsody from I-Logix Simulink and Stateflow

from Mathworks Inc. Rose Real-Time from

Rational Etc. Etc.

http://ww

w.cs.um

n.edu/crisys

7

Research Tools (many):RSML-e and Nimbus

RSML-e Formal Models (~20 running concurrently)

Simulations of environment

http://ww

w.cs.um

n.edu/crisys

8

SystemSpecification/Model

How we Will Develop SoftwareConcept

Formation

Requirements

Implementation

Integration

PropertiesAnalysi

s

Integration Test

System

Test

Specification Test

http://ww

w.cs.um

n.edu/crisys

9

FGS/FMS Mode LogicRSML-e and Nimbus

RSML-e Formal Models (~20 running concurrently)

Simulations of environment

http://ww

w.cs.um

n.edu/crisys

10

Sample RSML-e Specification

http://ww

w.cs.um

n.edu/crisys

11

Capture Requirements as Shalls

http://ww

w.cs.um

n.edu/crisys

12

Translated All the Shalls into SMV Properties

http://ww

w.cs.um

n.edu/crisys

13

Early Validation of Requirements Using Model-Checking (NuSMV)

• Prove Over 300+ Properties in Less Than an Hour

• Found Several Errors in Our Models Using Model-Checking

• Substantially Revised the Shalls to Correct Errors

http://ww

w.cs.um

n.edu/crisys

14

Early Validation of Requirements Using Theorem Proving (PVS)

• Proved Several Hundred Properties Using PVS• More Time Consuming than Model-Checking• Use When Model-Checking Won’t Work

http://ww

w.cs.um

n.edu/crisys

15

Model-Based Development ExamplesCompany Product Tools Specified & Autocoded Benefits Claimed

Airbus A340 SCADE With Code Generator

70% Fly-by-wire Controls 70% Automatic Flight Controls 50% Display Computer 40% Warning & Maint Computer

20X Reduction in Errors Reduced Time to Market

Eurocopter EC-155/135 Autopilot

SCADE With Code Generator

90 % of Autopilot

50% Reduction in Cycle Time

GE & Lockheed Martin

FADEDC Engine Controls

ADI Beacon Not Stated

Reduction in Errors 50% Reduction in Cycle Time Decreased Cost

Schneider Electric

Nuclear Power Plant Safety Control

SCADE With Code Generator

200,000 SLOC Auto Generated from 1,200 Design Views

8X Reduction in Errors while Complexity Increased 4x

US Spaceware

DCX Rocket MATRIXx Not Stated

50-75% Reduction in Cost Reduced Schedule & Risk

PSA Electrical Management System

SCADE With Code Generator

50% SLOC Auto Generated 60% Reduction in Cycle Time 5X Reduction in Errors

CSEE Transport

Subway Signaling System

SCADE With Code Generator

80,000 C SLOC Auto Generated Improved Productivity from 20 to 300 SLOC/day

Honeywell Commercial Aviation Systems

Primus Epic Flight Control System

MATLAB Simulink

60% Automatic Flight Controls 5X Increase in Productivity No Coding Errors Received FAA Certification

http://ww

w.cs.um

n.edu/crisys

16

A Simplified Development Model

Requirements and Specification

Code

Unit Test

System Test

Time

http://ww

w.cs.um

n.edu/crisys

17

Ongoing Research

SpecificationModel

Visualization PrototypingTesting

Code

Analysis

Properties

CMU, SRI, Stanford, UC Berkley, VERIMAG, NASA, Etc., Etc.

RSML-e, SCR, SpecTRM, Statecharts, Esterel, SCADE, Simulink, Etc. Etc.

RSML-e, SCR, SpecTRM, Statecharts, Esterel, SCADE, Simulink, Etc. Etc. –UML

RSML-e, SCR, SpecTRM, Statecharts, Esterel, SCADE, Simulink, Etc. Etc. –UML

Minnesota, Pennsylvania, George Mason, NRL, NASA Ames, Etc.

Proof carrying code, Provably correct compilers, Test for correctness

http://ww

w.cs.um

n.edu/crisys

18

Problems…

SpecificationModel

Visualization PrototypingTesting

Code

Analysis

Properties

Are the languages usable—syntax and semantics?Can they play nice together?

Can we trust execution environment?

Can we trust execution environment?

Trust the results?

Tested enough?

Can we really trust the code?

http://ww

w.cs.um

n.edu/crisys

19

Benefits of Modeling

Time Savings

Fewer “Bugs”Fewer “Bugs”

http://ww

w.cs.um

n.edu/crisys

20

Code Generation

Time Savings

Fewer “Bugs”Fewer “Bugs”

Coding effort greatly reduced

http://ww

w.cs.um

n.edu/crisys

21

Qualified Code Generation(theory)

Time Savings

Unit testing eliminated for generated code

Unit testing moved here.

http://ww

w.cs.um

n.edu/crisys

22

SystemSpecification/Model

Code Generation ConcernsConcept

Formation

Requirements

Implementation

Integration

Properties

Can we trust the

code generator?

Is our model “right”?Can we trust the execution environment?Can we trust our analysis tools?Can we trust our properties?

http://ww

w.cs.um

n.edu/crisys

23

“Correct” Code Generation• Provably correct compilers

Very hard (and often not convincing)

• Proof carrying code Total correctness required

• Base all specification testing on the generated code

Loose the benefits of working at the specification level

• Generate test suites from specification

Compare specification behavior with generated code to better trust your specification testing

Unit testing is now not eliminated, but completely automated

Specification/Model

Implementation

Specification Based Tests

Output

Output

Generate

http://ww

w.cs.um

n.edu/crisys

24

Specification Testing

• Certify the execution environment Too costly and

probably impossible

• Specification based testing Any discrepancy and

either the code generator is

wrong, or the execution

environment is wrong, or

the target platform is faulty

• When have we tested enough? Specification coverage

criteria What is adequate

coverage? Criteria for

measurement are not good for generation

– Technically covering the specification, but with useless tests

Do we reveal faults Tradeoff between the

impossible and the inadequate

http://ww

w.cs.um

n.edu/crisys

25

Proof Techniques (theory)

Time Savings

Reduced testing since properties

proved correct in specification stage

Proofs performed here

http://ww

w.cs.um

n.edu/crisys

26

SystemSpecification/Model

Verification TrustConcept

Formation

Requirements

Implementation

Integration

Properties

Proof validity in production

environment?

We need properties (requirements)!!!

Often lost in the modeling “frenzy”

How do we trust our proofs?

http://ww

w.cs.um

n.edu/crisys

27

Proof Techniques

• Certify analysis tools Too costly and probably

impossible

• Use redundant proof paths Technically feasible, but is the

redundancy “trustworthy”?? Cost…

• Automation is key Must keep analysis cost under

control

• Generate test suites from specification

Low cost since it is already done for the code generator

Trusted Translators

?

RSML-e

State Exploration

Model Checker

Theorem Prover

Translation

Translation

Translation

Trusted Translators?

Many languages and many analysis

techniques

http://ww

w.cs.um

n.edu/crisys

28

Proof Techniques (worst case)

Time Savings

Most analysis is not easy, nor cheap!

Added burden that cannot be leveraged later

http://ww

w.cs.um

n.edu/crisys

29

Regression Verification100s, if not 1000s, of properties

Large Evolving Model

Analysis Result

Iterated Weekly? Daily?

Hourly?

• Abstraction cost amortized

• Impact of change on abstraction

• Approximate techniques in day-to-day activities

http://ww

w.cs.um

n.edu/crisys

30

Can We Achieve the Goal?

Time Savings

Abbreviated system testing augmented

with generated tests

Redundant proof process (PVS, SMV,

Prover, SAL,…)

Specification testingTest case generation

Verifiable code

generator

Automated unit testing (to MC/DC?)—to check code generator and specification

execution environment

Yes!

?

?

? ?

?

http://ww

w.cs.um

n.edu/crisys

31

Perfection is Not Necessary

• We only need to be better than what we are now… How do we demonstrate this?

Empirical studies are of great importance

≥Missed Faults

http://ww

w.cs.um

n.edu/crisys

32

Education of Regulatory Agencies

• Regulatory agencies are very conservative And rightly so… Avionics software is very good

• We need to understand regulatory and industry concerns to get our techniques into practice

• We need to have convincing evidence that our techniques work and are effective

http://ww

w.cs.um

n.edu/crisys

33

New Challenges for V&V• Validate models

The models must satisfy the “real” requirements Validate the properties used in analysis Model testing crucial to success

• Validate tools We will rely a lot on tools for model

validation, can we trust them? Creative use of testing necessary

• Verify and Validate generated code Can we trust that the translation was correct? Test automation crucial Includes output to analysis tools

• Adapt to the various modeling notations Models will not come in one language Translation between notations and tools

http://ww

w.cs.um

n.edu/crisys

34

Discussion