copyright critical software s.a. 1998-2006 all rights reserved. val-cots validation of real time...

Copyright Critical Software S.A. 1998-2006 All Rights Reserved.

VAL-COTSValidation of Real Time COTS Products

Ricardo Barbosa, Henrique Madeira, Nuno Silva

Dependability & Embedded

© Copyright Critical Software S.A. 1998-2006 All Rights Reserved.

2

Overview - VALCOTS

Introduction

Objectives and Goals

Developed Techniques

Conclusions

Future Work


3

Introduction (1)

VAL-COTS VALCOTS is an acronym for “Validation of Real

Time COTS Products” COTS current trend

Commercial Off-the-shelf products are being used broadly in space applications

How to have full confidence in COTS components? (custom made components confidence can be controlled and assessed in-house)

... sometimes one may doubt of certain dependability characteristics of COTS


4

Introduction (2)

Why are COTS different from custom made software? Different requirements needs...

COTS are designed to be general purpose (requirements such as timing and safety might not be given high priority)

Wrong assumptions... Product engineers tend to assume that it is impossible

to predict all possible component violations for each function

Overlooked tests... Input tests are sometimes overlooked (insufficient) due

to performance or schedule constraints


5

Objectives and Goals (1)

Goals The project main goal is to develop techniques and

toolsets for the assessment of real time COTS products based on automated fault injection techniques

Fault Injection...? Fault injection techniques have shown to be an excelent

approach in the validation of COTS products Mentioned in Galileo Software Standard as an advisable

technique to be used for the validation of space systems

Benchmarks used fault injection techniques for validating real time constraints [DBench]

Can simulate effects of radiation on hardware (space environment)


6


Introduce the new concepts on fault injection and testing in NASA’s IV&V philosophy as an agency’s value added research and expertise

Improve NASA’s IV&V processes, tools and techniques on dynamic testing


7


Exchange of software metrics (on COTS and others)

Collaborative knowledge exchange on IV&V, dependability evaluations, dynamic testing and automated testing between CSW(European Space Industry) and NASA


8

Developed Techniques (1)First Technique

Three different techniques were developed during the project

First Technique is focused on COTS robustness and stress capabilities Based on application requirements, API fault

injection is performed to assess the COTS component in the presence of in and out of bounds values

This process is accelerated and tunned up by performing software automated fault injection in upstream components of the target COTS


9

mutex_change_priority (TCB *p_tcb,UINT oldpri,UINT *p_newpri)

Type Name

Test Values

INT -2147483648, -1, 0, 1, 2147483647

UINT 0, 1, 4294967295

BOOL true, false

FN -2147483648, -1, 0, 1, 2147483647

__xception__ = 0; m_p_tcb = mutex_change_priority ( p_tcb, oldpri, __xception__);



Developed Techniques (2)First Technique

Automatic Test Cases Generation


10

Developed Techniques (3)Second Technique

The second technique is double phased: (1) COTS component dependability is assessed

individually at an earlier development stage (2) Software fault injection is performed inside

the actual component to assess the component (and system) within the full integrated architecture, conducted at a latter developement stage

Results may lead to architectural changes or component wrapping, or even COTS component replacement


11

Developed Techniques (4)Second Technique

DESIGN

ENGINEERING PHASE

SOFTWARE REQUIREMENTS

ENGINEERING PHASE

SYSTEM ENGINEERING

PHASE

SRR PDR

Verification

∙ Select COTS candidates for use within the system; ∙ Analyse each COTS for robustness; ∙ Tune wrappers for improving component robustness; ∙ Reselect best suitable component after wrapping;

OPERATIONS & MAINTENANCE

PHASE

ACCEPTANCE PHASE

VALIDATION PHASE

QR AR

Validation

∙ Assess COTS component behaviour with integrated system; ∙ Assess COTS behaviour when subjected to faults; ∙ Assess fault propagation within the system; ∙ Validate COTS fault tolerance mechanisms;

Earlier Stages of SDP Latter Stages of SDP


12

Developed Techniques (5)Third Technique

A third technique was developed based on the second one

To complement the technique, risk assessment calculations are also introduced to estimate failure probability of the component within the system

Risk assessment calculations are based on software metrics from component and system complexity (e.g. cyclomatic complexity)


13

Conclusions

The case studies conducted have shown that the applicaiton of any of the techniques lead to architectural and component improvements

Thus, robustness and dependability improvements Results identify that COTS components have

intrinsic faults that are hard to find with traditional testing

Usefulness of fault injection Xception based automated testing as proven to

be very usefull particularlly considering fault injection definition and injection automation

Some difficulties where encountered while evaluating injection run results as it is important to improve results analysis automation


14

Future Work

VAL-COTS II Improvement of the developed techniques Further testing to validate the techniques

(hopefully on an on-going project) Adjust techniques for use in certification

processes (follow specific standards like ISO 25051 or DO-178B)

Improve the techniques based on the newly developed software fault injection G-SWFIT

G-SWFIT is a technique developed at Universidade de Coimbra for emulating real faults in fault injection

Using this technique, no source code is needed for performing fault injection, only binary files


15

The end!

Thank You!! Any questions?

[email protected]

[email protected]@criticalsoftware.com

copyright critical software s.a. 1998-2006 all rights reserved. val-cots validation of real time...

Documents