validation of guidance control software requirements specification for reliability and...

SEDS Research Group School of EECS, Washington State University

Annual Reliability & Maintainability SymposiumJanuary 30, 2002

Frederick T. Sheldon and Hye Yeon Kim

Software Engineering for Dependable Systems (SEDS) Research Laboratory

School of Electrical Engineering and Computer Science Washington State University

Validation of Guidance Control Software Requirements Specification for Reliability and Fault-Tolerance

SEDS Research Laboratory School of EECS, Washington State University

Overview Goal: Show the feasibility of this analysis approach

using a industrial strength SRS to ensure:Completeness and ConsistencyFault-tolerance

Specification Under StudyA NASA provided Guidance and Control Software (GCS)

development specification for the Viking Mars Lander. Analysis Approach

Using Zed to specify the data Using Statecharts : Statemate for dynamical analysis

Summary and Future study

SEDS Research Laboratory School of EECS, Washington State University

IntroductionWhy Requirements Specification?

Cost, Time, and Effort

Defect detected phase Typical cost of correctionRequirements Specification $100 - $1,000Coding/Unit Testing $1,000 or moreSystem Testing $7,000 - $8,000Acceptance Testing $1,000 - $100,000After Implementation Up to millions of dollars

SEDS Research Laboratory School of EECS, Washington State University

Reliable SpecificationIs Correct

Complete, consistent and robust

Can the specification be trusted while minimizing the risk of costly errors?

How to analyze the specification to prevent the propagation of errors into the downstream activities?

SEDS Research Laboratory School of EECS, Washington State University

Consistency and CompletenessCompleteness: The lack of ambiguity

Incomplete if …… the system behavior is not specified

precisely because the required behavior for some events or conditions is omitted or is subject to more than one interpretation.

ConsistencyThe Specification is free from conflicting

requirements and undesired nondeterminism.

SEDS Research Laboratory School of EECS, Washington State University

Fault ToleranceFaults

A fault is a feature of a system that precludes it from operating according to its specification

– H. Ammar, B. Cukic, C. Fuhrman, and A. Mili, A comparative Analysis of HW and SW fault tolerance: Impact on software reliability engineering, 1999

Fault ToleranceThe ability to respond to unexpected system

failure (detection and mask/recover)

SEDS Research Laboratory School of EECS, Washington State University

Guidance and Control Software Software Requirements – GCS Dev. Spec.

The system was designed to provide software control of the embedded sensors and actuators of the Viking Mars Lander during the terminal decent (landing) phase.

ARSP (Altimeter Radar Sensor Processing)The ARSP module reads data provided by the

altimeter radar sensor to determine the lander’s altitude from the Mars surface.

SEDS Research Laboratory School of EECS, Washington State University

Mars Lander trajectories

SEDS Research Laboratory School of EECS, Washington State University

Velocity – Altitude Contour

SEDS Research Laboratory School of EECS, Washington State University

EXTERNALRUN_PARAMETERS

SENSOR_OUTPUTGUIDANCE_STATE

TDLRSP.3

GSP.4

ARSP.2

ASP.1

TSP.5

TDSP.6

SEDS Research Laboratory School of EECS, Washington State University

SEDS Research Laboratory School of EECS, Washington State University

Zed OverviewClarifying ambiguitiesIdentify assumptionsCorrectness checkingMathematical proofsGiving an in-depth understanding of the SRS

SEDS Research Laboratory School of EECS, Washington State University

StatechartsVisual formalism: Diagrammatic in natureTestability is provided through symbolic

simulationPredevelopment evaluation through

Fault InjectionStatemate consists of …

Activity chartStatecharts

SEDS Research Laboratory School of EECS, Washington State University

Natural Language based SRS

Inherently ambiguous risking the possibility of multiple interpretations

SEDS Research Laboratory School of EECS, Washington State University

Zed Schema

SEDS Research Laboratory School of EECS, Washington State University

From NL to ZedDiscovered Ambiguities

The confusing definition of variable “Rotation”, and direction of the rotation.

The type assigned to the AR_COUNTER variable was unclear.

An undefined 3rd order polynomial.Where the AR_COUNTER should be

modified?

SEDS Research Laboratory School of EECS, Washington State University

Statecharts Model: Activity chart

SEDS Research Laboratory School of EECS, Washington State University

Statecharts Model: Statechart 1

SEDS Research Laboratory School of EECS, Washington State University

Statecharts Model: Statechart 2

SEDS Research Laboratory School of EECS, Washington State University

Some Theory …Set of Inputs

()

Set of Outputs

Unknowns ()

KnownKnown

SafeUnsafe

AssumedSafe

Sources: Normal Operation Hardware Failures Human Intervention Models/Simulators

SEDS Research Laboratory School of EECS, Washington State University

SEDS Research Laboratory School of EECS, Washington State University

Paradigms …Software Failures:

“Software does not fail - it just does not perform as intended” Professor Nancy Leveson, MIT

Design and test for functionality:Also specify what the system should not do. . .. . . then test it!

SEDS Research Laboratory School of EECS, Washington State University

Some Theory… lets take a second look

Set of Inputs ()

Set of Outputs

Unknowns ()

Known

Known SafeUnsafe

AssumedSafe

Sources: Normal Operation Hardware Failures Human Intervention Models/Simulators

Fault Injection(added known)

SEDS Research Laboratory School of EECS, Washington State University

Testing and Fault InjectionBy using symbolic simulation in

StatemateBoundary Testing

Input that is inside of the variable rangeInput that is outside of the variable range

Fault InjectionState variable alternationState transition redirection

SEDS Research Laboratory School of EECS, Washington State University

Testing Results ARSP Specification Test Input and Output

  Variable Case 1 Case 2 Case 3 Case 4 Case 5

Input

FRAME_COUNTER 2 2 1 1 3

AR_STATUS - - [0, 0, 0, 0] - [0, 1, 0, 0]

AR_COUNTER -1 19900 -1 20000 -1

Output

AR_STATUS KP KP [1, 0, 0, 0] [0, -, -, -] [1, 0, 1, 0]

K_ALT KP KP [1, 1, 1, 1] [1, -, -, -] [0, 1, -, 1]

AR_ALTITUDE KP KP [*, -, -, -] [2000,-,-,-] KP

SEDS Research Laboratory School of EECS, Washington State University

Detailed Testing Results - Case 1

Initial valuesFinal values Initial variable

values are being calculated based on the given equations.

  Variable Case 1

Input

FRAME_COUNTER 2

AR_STATUS -

AR_COUNTER -1

Output

AR_STATUS [1, 0, 0, 0]

K_ALT [1, 1, 1, 1]

AR_ALTITUDE [2000, -, -, -]

[1, 1, 0, 0]

[1, 1, 1, 1]

[2000, 2000, -, -]

SEDS Research Laboratory School of EECS, Washington State University

Fault Injection ResultsAltered state variable

FRAME_COUNTER AR_COUNTER AR_STATUS Fault injected State Case

1 Case

2 Case

3 Case

4 Case

5 Case

1 Case

2 Case

3 Case

4 Case

5 Case

1 Case

2 Case

3 Case

4 Case

5

CURRENT_STATE x x x x x x x x x x x x x x x

KEEP_PREVIOUS_VALUE b b b b b b b b b b b b b b b CALCULATION b b b b b b b x x x b b x b x

ODD b b b b b b b x x x b b x b x ESTIMATE_ALTITUDE b b b b b b b N/A b b b b N/A b b

CALCULATE_ALTITUDE b b b b b b b b x b b b b b b KEEP_PREVIOUS b b b b b b b b b b b b b b b

DONE b b b b b b b b b b b b b b b x incorrect outputs, b no defect

SEDS Research Laboratory School of EECS, Washington State University

Detailed Fault Injection Results

Change FRAME_COUNTER at CURRENT_STATE from 2 to 3

  Variable Case 1

Input

FRAME_COUNTER 2

AR_STATUS -

AR_COUNTER -1

Output

AR_STATUS [1, 0, 0, 0]

K_ALT [1, 1, 1, 1]

AR_ALTITUDE [2000, -, -, -]

[1/0, 1, 0, 0]

[1, 1, 1, 1]

[*, 2000, -, -]

SEDS Research Laboratory School of EECS, Washington State University

SummaryInterpretation from NL to Zed

Clarifying ambiguitiesInterpretation from Zed to Statecharts

Clarifying misinterpreted Zed specificationIterative process

Boundary Testing, Fault Injection analysisReveals weak point(s) in the systemFault Tolerance validation

SEDS Research Laboratory School of EECS, Washington State University

ConclusionUsing this combination of FMs we could:

Clarify ambiguitiesValidate Correctness, Completeness, and ConsistencyValidate Fault tolerance features of the SRS

This approach enabled us to:Avoid the problems that result when incorrectly

specified artifacts force corrective rework.Minimize the risk of costly errors being propagated into

downstream activities

SEDS Research Laboratory School of EECS, Washington State University

Future StudyBuild concrete translation rules between the

methods

Find an effective algorithm to automate the process

Validate the algorithm for the different (domain/ application specific) critical software requirements

In depth comparative study with other approaches