mapping assurance to the software engineering process alfred h. kromholz, ph.d. the mitre...

Mapping Assurance to the Software Engineering Process

Alfred H. Kromholz, Ph.D.

The MITRE Corporation

kromholz@ mitre.org +1-703.883.7331

Copyright © 2004 The MITRE Corporation

Copyright © 2004 The MITRE Corporation2

Relation between Waterfall Process & V-Chart

Plans & RqmtsProd. Design

Detail Design

Unit TestInteg. Test

Syst. Test

Code

Concept

Accep. TestConcept

Plans and Requirements

Product Design

Code

Detail Design Unit Test

Integration Test

System Test

Acceptance Test

Change the shape from waterfallto fall and rise

Traditional Waterfall

The obvious question: Why bother?


Conceptual

Plans and Requirements

Product Design

Code

Detail Design Unit Test

Integration Test

System Test

Acceptance Test

Needs

System Requirements

Subsystem Requirements

Elements

Element Requirements Element

Test

Subsystem Test

Development V-ChartStarts with a Product Focus to Process Abstraction

Analysis phase(Decomposition)

Synthesis phase(Verification)

Development V-ChartShift from Product Focus to Process Abstraction


For efficiency, focus on

interfaces & interactionsSystem

Requirements


Element Requirements

Subsystem Test

Development V-ChartProcess and Product

Needs

Elements

Element Test

System Test

Acceptance Test

Analysis phase(Product Explosion)

Synthesis phase(Product Implosion)

The structure of the decomposition is the product architecture.

Analysis phase Synthesis phase


Development V-ChartTypical Generic Products of the Process

Needs

System Requirements


Elements


Element Test

Subsystem Test

System Test

Acceptance Test

NeedsStmt

System DesignSpec

Sub-syst DesignSpec

Detail DesignSpec

Test Report

Test Report

Test Report

Test Report

+


Needs

System Requirements


Elements


Element Test

Subsystem Test

System Test

Acceptance Test

Development V-ChartWhen Do We Establish Test Criteria?

Test Report

Test Report

Test Report

Test Report

+

NeedsStmt

DesignSpec

DesignSpec

DesignSpec

AcceptCriteria

TestSpec

TestSpec

TestSpec

Development V-ChartTypically, We Establish Test Criteria on the Way Up

Same products on this side

Earlier products on this side

AcceptCriteria

TestSpec

TestSpec

TestSpec

Development V-ChartWe Should Establish Test Criteria on the Way Down

Generally, just barely before we test


Validate / Verify

V&V

V&V

V&V

Development V-ChartVerify and Validate Before Staring Upward

NeedsStmt

DesignSpec

DesignSpec

DesignSpec

AcceptCriteria

TestSpec

TestSpec

TestSpec

Test Report

+

Test Report

Test Report

Test Report

Needs

System Requirements


Elements


Element Test

Subsystem Test

System Test

Acceptance Test


Validate / Verify

V&V

V&V

V&V

Development V-ChartValidate, Plan for Verify & DFT/DFM Early

NeedsStmt

DesignSpec

DesignSpec

DesignSpec

AcceptCriteria

TestSpec

TestSpec

TestSpec

Test Report

+

Test Report

Test Report

Test Report

Needs

System Requirements


Elements


Element Test

Subsystem Test

System Test

Acceptance TestDFT/DFM Feedback

DFT/DFM

DFT/DFM


Needs

System Requirements



NeedsStmt

System DesignSpec

Sub-syst DesignSpec

Detail DesignSpec

Element Test

Subsystem Test

System Test

Acceptance Test

Let’s Go Back to the Original Development V-Chart

Rationale

Rationale

Rationale

Rationale

Test Report

+

Test Report

Test Report

Test Report

Elements

The “Missing” Rationale

How did we get

from here

How did we get

from here to here?

Or here?

Or here?

Or here?

Does it need to be captured?

Recording the rationale for

decomposition can help us


Where Does Assurance Fit In?

Two broad customer circumstances

• Functional expectations but no assurance requirements– Expectations may be informal – mass-market, “shrink-wrapped” products

– Expectations may be formal – supplier requirements documents & specifications– Customer says, “Convince me that the product meets expectations.”

– Customer may say, “Why should I have confidence in your claims?”

• Both functional requirements and assurance requirements– Customer tells supplier what it takes to convince

– Assurance requirements may be direct – i.e., included in the specifications

– Assurance requirements may be indirect – i.e, specifications identify product and/or process standards to be followed


Customer ActivitiesSupplier ActivitiesThese activities take place during the

a posteriori phase

after the product is produced

Sub-claim evaluation


Claim evaluation

Acceptance

Assurance V-Chart

AssuranceClaim

Evidence

Evidence

Evidence

Evidence

Body of Evidence

Sub-Claims

Sub-Claims

Sub-Claims

No customer-specified assurance requirements

Two groups of assurance-related

activities


Expectations, standards, desiderata

Claims re Expectations

Sub-claims re Expectations

Sub-claims re Expectations

a priori phase

a posteriori phase

Body of Evidence



Claim evaluation

Acceptance

Argument

Argument

Argument

Argument

The specific problem domain implies both the applicable

standards and the Assurance Level required

Assurance V-Chart (Augmented)Customer-specified assurance requirements

But how do we know these are necessary and sufficient with respect to the previous level?

Evidence-Item Classes

We add an a priori phase that sets up an assurance case before a

product is designed and producedAs in the previous

chart

Tells the supplier what evidence is expected (and maybe in what form to present it)

are decomposed into

We add an “argument” that justifies how sub-claims support a claim and evidence supports

the subclaims –

are decomposed into

are decomposed into

are decomposed into

for each decomposition

activity


DFT/DFM Feedback

Evidence

DFT/DFM

Evidence

DFT/DFM

Evidence

Needs

System Requirements


Elements


Element Test

Subsystem Test

System Test

Evidence

Evidence

Evidence

Evidence

Evidence

Evidence

Evidence

Evidence

Evidence

Acceptance Test

Development V-ChartEvidence for Assurance Comes from All Development Activities

Validate&Verify

Evidence

V&V

Evidence

V&V

Evidence

V&V

Evidence

Evidence is gathered in accordance with the

evidence classes identified in the a priori phase.

Remember the “rationale” for

decomposition?This is where it comes in handy.








Assuranceprocess

Development process

Minimum SeparationDevelopment starts after 1 or 2 levels of Assurance

decomposition

Ideal SeparationAll necessary evidence

identified before Development starts

Assurance CompletionOccurs after all evidence

from Development is received

Assurance SynthesisStarts as early as evidence classes

are identified and evidence instantiations are available

Time Relationship of Assurance & Development V-Charts

How much time?

Clearly, assurance requirements should be

identified before development begins.

Current Work & Future Directions


Using Adelard Safety Case Editor to Argue a Safety Case

This illustrates the idea of a supplier making a claim to a purchaser that a product is safe.Thus the arrows go upward from evidence to argument to claim.


Arguing a Safety Case: Basic Supplier Approach

CLAIMArchitecture

is safe

ARGUMENTPartitioning is

proper

ARGUMENTMulti-versioning Dissimilarity is

properly provided

ARGUMENTSafety

Monitoring is provided

EVIDENCEPartitioning

Documentation

EVIDENCEMulti-versioning Documentation

EVIDENCESafety-Monitoring

Documentation

Note that there is no justification provided to support the assertion that the given arguments support the claim.

Argument is made on the basis of the

evidence.

Evidence is produced prior to making the claim.

Collection of arguments is used to

support the claim.


Arguing a Safety Case: Standards-Based Supplier Approach

CLAIM2.3

Architecture is safe

ARGUMENT2.3.1

Partitioning is proper

ARGUMENT2.3.2

Multi-versioning Dissimilarity is

properly provided

ARGUMENT2.3.3

Safety Monitoring is

provided

EVIDENCE2.3.1

Partitioning Documentation

EVIDENCE2.3.2

Multi-versioning Documentation

EVIDENCE2.3.3

Safety-Monitoring Documentation

However, there is still no justification provided. Indeed, many standards documents not only omit but deliberately exclude rationale.

The supplier can include references to governing documents

as part of the assurance package to help

support assertions.

(Numbering here is adapted from DO-178B, Software Considerations in Airborne Systems and Equipment Certification)


CLAIM2.3


CLAIM2.3.1


CLAIM2.3.2


properly provided

CLAIM2.3.3

Safety Monitoring is provided

EVIDENCE2.3.1


EVIDENCE2.3.2


EVIDENCE2.3.3


ARGUMENTPartitioning, multi-versioning,

and safety monitoring are necessary and sufficient for a

safe architecture

Arguing a Safety Case: Justified Supplier Approach

Justification can be provided by the supplier to support the claim ...

but there is no guarantee that this will satisfy the

customer.


REQUIREMENT2.3


REQUIREMENT2.3.1


REQUIREMENT2.3.2


properly provided

REQUIREMENT2.3.3

Safety Monitoring is

provided

EVIDENCE class2.3.1


EVIDENCE class2.3.2


EVIDENCE class2.3.3


ARGUMENT 2.3Partitioning, multi-versioning,

and safety monitoring are necessary and sufficient for a

safe architecture

Arguing a Safety Case: Requirements-Based Approach

In the Requirements-Based approach, the arrows are top-down

Customer defines what “safe architecture” means

(at least internally to the acquiring organization)

Customer identifies

expectations regarding proof that architecture

is safeNote:The bottom level calls for classes of evidence, not specific documents.

Customer defines that architecture must be safe

Customer establishes

requirements based on

decomposed definition


REQ’MENT2.3


REQ’MENT2.3.1


REQ’MENT2.3.2


properly provided

REQ’MENT2.3.3


EVIDENCE Class2.3.1


EVIDENCE Class2.3.2


EVIDENCE Class2.3.3

Safety-Monitoring

Documentation

ARGUMENT 2.3Partitioning, multi-

versioning, and safety monitoring are necessary and sufficient for a safe

architecture

Arguing a Safety Case: Consolidated Flow

CLAIM2.3


EVIDENCE2.3.1


EVIDENCE2.3.2


EVIDENCE2.3.3

Safety-Monitoring

Documentation

CLAIM2.3.1


CLAIM2.3.2


properly provided

CLAIM2.3.3


ARGUMENT 2.3Partitioning, multi-

versioning, and safety monitoring are necessary and sufficient for a safe

architecture

Customer specifications

(a priori)

Supplier evidence

(a posteriori)Works remarkably like

a V-Chart, doesn’t it?

mapping assurance to the software engineering process alfred h. kromholz, ph.d. the mitre...

Documents

test criteria

test report elements

test concept plans

test code concept accep

functional requirements

mitre corporation slide

vv development vchart

mitre corporation kromholz