fy 2004 initiative: risk assessment of software architectures

1

Research Heaven,West Virginia

FY 2004 Initiative: Risk Assessment of Software Architectures

Hany Ammar, Katerina Goseva-Popstojanova,Ajith Guedem, Kalaivani Appukkutty,

Walid AbdelMoez, and Ahmad Hassan

LANE Department of Computer Science and Electrical EngineeringWest Virginia University

Less risk, soonerWVU UI:

Risk Assessment of Software Architectures

2


Outline

Introduction Severity Analysis Methodology Requirement Risk Analysis Methodology Relevance to NASA Case studies Conclusion

3


Introduction

• Risk is a combination of: The probability of failure and the consequence of the failure (Severity)

• Severity analysis is conducted on Unified Modeling Language (UML) models to quantify the severity of failure

• Severity is estimated based on cost of failure and Hazard analysis techniques:

– Functional Failure Analysis (FFA)– Failure Mode and Effects Analysis (FMEA)– Fault Tree Analysis (FTA)

• Requirement Risk is assessed based on: Probability of failure (Normalized Dynamic Complexity) and Severity

4


Outline


5


Proposed Severity Analysis Methodology

• INPUT: DYNAMIC UML SPECIFICATIONS

– System level scenarios related to use cases, annotated with the following:

– Guide words (Omission, Commission, Late, etc)– Cost of hazard

– Sequence diagram that show the component/connector interactions annotated with the following:

– Component states, probability that a component fails in a given failure mode, and the cost of failure in each state

– Messages transmitted between components, probability that connectors fail to transmit a message, and cost of failure of each message

• OUTPUT: COMPONENTS/CONNECTORS and SCENARIO SEVERITY

6

FMEA

2

Proposed severity analysis methodology

FTA

3(List of scenario level Hazards)

(List of Component/Connector Failure Modes)

Scenario Severity

Component/Connector Severity

O b ject : C las sO b j ect : C l as s O b jec t : C las s

UMLSpecs

Use CaseDiagrams,System levelScenarioDiagrams

SequenceDiagramsComponent/Connectorinteractions

Component/connector

Cost-Severity Graph

Scenario levelCost-Severity

Graph

5Cost of Failure Graph

4FFA

1

ScenarioSeverity

And

Components/

ConnectorsSeverity

7


Proposed Severity Analysis Methodology

1. Identify system hazards: states of the system that can contribute to accidents and mishaps

Perform FFA using system scenario diagrams as an input

2. Identify component/connector failure modes Perform FMEA using UML Sequence Diagrams as an input

3. Construct a detailed cause-and-effect model, to record how failures propagate from component/connector level to the system level

FTA is used to combine the outputs from FFA and FMEA

4. Develop the Cost of Failure Graph to estimate severity of each component/connector in a given scenario, or severity level of the scenario

5. Estimate severity of component/connector and system scenario using cost of failure graph and cost-severity graph

The final result is a table of component/connector severity and system scenario severity.

8


Step 1: FFA Table

Event Name

Class of failure

Failure Effects on System Cost of Failure

($)

Comments

Command Value A Fault in processing command routine

Heart is continuously triggered but device is still monitored by physician, need immediate fix or disable.

1000 The component received the command misinterpret it

VSense Omission Timer not set correctly

No pace is generated for the heart, patient could die

100000 Some component’s timers does not work well

Pace Commission Pacing hardware device malfunctioning

Heart is always paced while patient condition requires only pacing the heart when no pulse is detected. Heart operation is irregular

100000 Some component’s sensor failed to sense the heart.

9


Step 2: Example (Pacemaker AVI Scenario)

Sequence diagram of the AVI scenario

Ventr icular: VTCommunicat ionGnome: CG

Atr ial: AR

T oOn

T oOn

T oAVIT oAVI

VRefract DoneRef T imeOut

Got VSenseVSense

A Pace Start

RefractoryT imeOut

Pace

Pace T imeOut

A Pace Done

Heart

Refractory

Idle

Wait ing

Wait ing

Pacing

Refractory

VRefractDone

Off

Rfractory

Wait ing

Wait ing

Pacing

Refract ory

Scnse T imeOut

Command

Programmer

V a lu e , $1 0 0 0$ 10 0 0 , 0 .0 9

$ 10 0 0 , 0 .0 5

$1 0 0 0 , 0 .0 5

$1 0 0 0 , 0 .0 5

$1 0 0 0 , 0 .0 5

$ 1 00 0 0 0 , 0 .1

$1 0 0 0 0 0 , 0 .0 5

O m is s io n , $ 1 0 0 0

$ 10 0 0 0 0 , 0 . 0 3

$ 1 00 0 0 0 , 0 .0 5

$ 10 0 0 , 0 .0 1

$ 1 0 0 0 0 0 , 0 .0 5

V a lu e , $1 0 0 0

$ 10 0 0 0 0 , 0 . 05

$ 1 0 0 0 , 0 .0 3

$1 0 0 0 00 , 0 .0 3

$1 0 0 0 00 ,0 .1

$ 10 0 0 0 0 , 0 . 05

C o m m is s io n , $ 1 0 00

$ 10 0 0 0 0 , 0 . 03

$ 10 0 0 , 0 .0 2 $ 10 0 0 , 0 .0 5

$1 0 0 0 00 , 0 . 1

$ 1 0 0 0 , 0 .1

$ 10 0 0 , 0 .0 3

10


Step 2: FMEA Table

Component Failure Mode Effect on the system

Cause of failure Cost of failure

mode ($)AR ToOn Value Error

(AR stuck in Idle state)

The component will not work and there is no pace of the heart

The component miss interpret signal received from CG

1000

- AR stuck in Refractory State

The component will stay in Refractory state and there is no pace

Connector VT-AR sends a wrong message, or component AR failes to understand the message

1000

- The component receive GotVSense but there is no pace(AR stuck in Waiting state)

The component will stay in waiting state and there is no pace

The component sensor does not work

100000

...

11


Step 3: Fault Tree Analysis (FTA)

Step 1:Top event hazard identified by applying FFA

Step 2: Component/Connector failure modes identified through

FMEA

Step3: Combining the results of steps 1 and 2 to build a cause-effect

model by applying FTA

T o p E v en tC o m m is s io n "P a ce "

VT Co mp o n e n tF a ilu re

A R Co mp o n e n tF a ilu re

C o n n ecto r V T -A RF ai lu re

C o n n ecto r A R -V TF ai lu re

N o p ac e S tar t , 0 . 1

VR ef r ac t E r ro r , 0 . 0 5G o t VS en s e Er ro r , 0 .1

S tu c k in R ef r ac to r y S ta te , 0 .0 5S tu c k in W ait in g S ta te , 0 .0 5S en s e T im eO u t E r ro r , 0 .1

S tu c k in R ef r ac to r y S ta te , 0 . 0 5

S en s e T im eO u t E r ro r , 0 .1

R ef T im eO u tE r ro r , 0 . 0 5

0 .2

0 .2

0 .1

0 .1 5

0 .6 5

Commission “Pace” Fault Tree

12


Step 4: Cost of Failure Graph forAR Component

System Level hazards (Fault Trees Top Event)

Commission “Pace”

“Command” Value ErrorP(“Command” Value Error)=0.02

P(Commission “Pace”) = 0.65

Failure Modes probabilitiesof AR Component

Consequence (Cost) of AR Failure Modes

AR “stuck in Refractory” State P(“stuck in Refractory”) = 0.05AR Cost

Of Failure Sense TimeOut ErrorP(“Sense TimeOut Error”) = 0.1

PaceTimeOut ErrorP(“PaceTimeOut Error”) = 0.03

AR “stuck in Pace” StateP(“stuck in Pace”) = 0.03

AR failed to handle ToOnP(“failed to handle ToOn”) =.09

AR “ stuck in Waiting” StateP(“stuck in Waiting”) = 0.05

$ 1000(Regular Care)

$ 1000(Regular Care)

$ 100000(Intensive Care)




Sense TimeOut ErrorP(“Sense TimeOut” Error) = 0.1

P(Omission “VSense") = 0.5

Omission “VSense”

13


Step 5: Severity of components/connectors

Cost-Severity Graph

14


Step 5: Component/connector severity and scenario severity

Component/connector name

Severity

Connector CG-AR 0.50

Connector CG-VT 0.50

Connector AR-VT 0.94

Connector VT-AR 0.95

Component CG 0.05

Component AR 0.96

Component VT 0.95

Severity of components/connectors in the AVI scenario

Scenario Severity

AVI Scenario

0.94

AVI scenario severity

15


Outline


16

Proposed Requirement Risk Matrix

Requirements Scenarios Failure Modes

FM1 FM2 … FMnR1 S1 Rf

S2R2 S3…Rm

Risk factor of scenario S1 in Failure mode FM2

• Requirements are mapped to UML use case and scenarios

• A failure mode refers to the way in which a scenario fails to achieve its requirement

• Failure mode of a scenario is determined based on the severity methodology

• According to Dr. Martin Feather’s DDP Process, “The Requirements matrix maps the impacts of each failure mode (should it occur) on each requirement.”

Impact Matrix (DDP Process)

17


Calculating Risk Factor

• The dynamic complexity is calculated at the scenario level

• The risk factor of a particular scenario (a), in a particular failure mode (b) is:

Rfab = Normalized Dynamic Complexityab X Severityab

• Severity of a scenario in a specific failure mode is estimated using the previously discussed severity analysis methodology [Slide 6]

18


Calculating Complexity

• We estimate complexity as:

Complexity = CC * Msg

– CC : McCabe’s cyclomatic complexity of a scenario for each failure mode

– Msg: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario

According to Fenton et al, “The product of cyclomatic complexity (CC) and sigFF was shown to be a good predictor of fault proneness. ..The combined metrics appear to be better than both sigFF and Cyclomatic complexity on their own and also better than the size metric”

19


Proposed Requirement Risk Analysis Methodology

STEP 1: Map the requirements to UML sequence diagramsFor each Sequence diagram

STEP 2: Construct the control flow graph of the scenario from the sequence diagramSTEP 3: Identify the different failure modes on the sequence diagram and the control flow graphFor each failure mode

STEP 4: Assess the severity of the failure mode using Function Failure Analysis (FFA)STEP 5: Determine the cyclomatic complexity of the sub-control flow graph STEP 6: For the failure mode, measure the number of messages exchanged between the components in the sequence diagram.STEP 7: Calculate the complexity of the scenario for the failure mode as: Cyclomatic complexity * Number of messagesSTEP 8: Calculate the Risk factor of the scenario for the failure mode as: Complexity * Severity

End For each failure modeEnd for each sequence diagram

20

FM1Msg = 4

FM3Msg = 9

FM2Msg = 7

Atrial-Ventricular Inhibit Scenario: Control Flow Graph (Obtained from

Sequence Diagram)

Requirement R2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit Scenario

Failure Modes and Msg Values marked in the sequence diagram

21


Calculating Risk• Calculating Complexity

• Severity index of failure modes FM1, FM2 and FM3 of AVI scenario is 0.95

• Risk is calculated as the product of Complexity and Severity index

Failure Modes

Cyclomatic Complexity

(CC)

Number of Messages

(Msg)

Complexity (CC * Msg)

Normalized Complexity

FM1 3 4 12 0.111

FM2 6 7 42 0.389

FM3 6 9 54 0.5

Requirements ScenariosFailure Modes

FM1 FM2 FM3

R2 AVI 0.11 0.37 0.475

22


Requirement Risk Matrix for Pace Maker


FM1 FM2 FM3 FM4 FM5

R1 Programming 0.017 0.233R2 Atrial-Ventricular

Inhibit0.11 0.37 0.475

R3 Atrial-AtrialInhibit

0.151 0.283 0.566

23


Outline


24


Severity Analysis of EOS• Severity Analysis Methodology is applied on NASA’s Earth

Observing System (EOS)

Event Name Actor sends or receives this event

Status From IDB (external database) to system

Uplink Command From system to Space craft

Preplanned emergency scenario

25


FFA Table for EOS

SEQUENCE DIAGRAM

EVENT FAILURE EFFECTS COST SEVERITY

Preplanned Emergency

System retrieves the Instrument status from IDB database

IDB takes longer time to return the status

Emergency cannot be handled at proper time

Mission Loss

Catastrophic

System uplinks commands to the space craft

Commands don’t reach the space craft at proper time

Command groups cannot be up linked in time

Mission Loss

Catastrophic

26


Requirement Risk Matrix for HCS case study


FM1 FM2 FM3 FM4 FM5

R1 Retry both Pumps 0.029 0.366 0.289

R2 Single LT Mode 0.059 0.662

27


Outline


28


Conclusion• We have developed a methodology to assess severity of

failures of components, connectors, and scenarios based on UML models

• We have developed a methodology for assessing requirements based risk using normalized dynamic complexity and severity of failures

• These methodologies were applied on NASA case studies

• Our future work will address the validation of risk methodology using software artifacts and data from the Metrics Data Program

fy 2004 initiative: risk assessment of software architectures

Documents