iv&v facility 1 fy 2002 initiative iv&v of uml hany ammar, katerina goseva-popstojanova, v....

1

IV&V Facility

FY 2002 Initiative IV&V of UML

Hany Ammar, Katerina Goseva-Popstojanova,V. Cortelessa, Ajith Guedem, Diaa Eldin Nassar, Walid AbdelMoez, Ahmad

Hassan, and Rania Elnaggar

LANE Department of Computer Science and Electrical EngineeringWest Virginia University

Ali Mili, Bo Yu

College of Computing ScienceNew Jersey Institute of Technology

Less risk, sooner-

A Catch Phrase by Coach Menzies

WVU UI: Architectural-level Risk Assessment

2

IV&V Facility

Outline

• Objectives

• What we can do

• Why UML

• UML & NASA

• Project Overview

• Architecture-Based Risk Analysis

• The Risk Assessment Methodology

• Performance – based risk

• Accomplishments

• Future Work • Publications

3

IV&V Facility

Objectives

• Automated techniques V&V of dynamic specifications– Performance and timing

analysis – Fault-injection based

analysis,• Less risk, sooner

– Risk assessment• Technologies:

– UML– Architectures– Risk assessment

methodology• Benefits:

– Find & rank critical • use cases, scenarios, • components, connectors

Before bad software

After bad software

The ARIANE 5explosion

4

IV&V Facility

What We Can Do

• Identify and rank critical components based on risk factors and severity classes• How?- details follow

Scitcs

Fritcs

Pfmc_

LT

Pfmc_

MT

Sch

State

_Man

Op_C_Q

App_C

_QN3_

1

N3_2

Rpcm

_LT

Rpcm

_MT

Single_LT

Single_MT

Dual_MT_Failed

Dual_LT_Failed

Dual

Retry_MT_Pump

Retry_LT_Pump

Retrt_Both_Pumps

Monitoring

0

0.1

0.2

0.3

0.4

Risk Factor

Components

Scenarios

Minor

Major

Critical

Catastrophic

Not contributing

5

IV&V Facility

Why UML

• Unified modeling language– Rational software

– The three amigos: Booch Rumbaugh, Jacobson.

• International standard in system specification

An international standardIn system specification

6

IV&V Facility

UML & NASA

• Increasing use at NASA• Informal (very) survey

– Google search:

– “rational rose nasa”

– 10,000 hits

– 3 definite projects, just in first ten

• We use a case study

based on the UML specs of a component of the

International Space Station

7

IV&V Facility

Project Overview

FY01• Developed of an automated simulation environment for UML dynamic specification,

suggested an observer component to detect errors • Conducted performance and timing analysis of the NASA case study

FY02• Develop a fault injection methodology

Define a fault-model for components at the specification level • Develop a methodology for architecture-based risk analysis

Determine critical use case ListDetermine critical component/connector list

(based on recent paper by Yacoub & Ammar on IEEE Trans. on Software Engineering, June 02)

FY03• Develop a methodology for performance-based/reliability-based risk assessment• Validation of the risk analysis methodology on several NASA projects

8

IV&V Facility

Architecture-Based Risk Analysis

• Develop architecture-based approach for risk assessment– Overall system/subsystem– Different use cases– Key scenarios associated with use cases

• Heavily used scenarios• Scenarios that are used infrequently but perform critical functions

• Develop components and connectors risk factors– Define components risk factors as

Normalized dynamic complexity * Severity – Estimate dynamic complexity measure based UML sequence diagrams and state charts– Estimate severity measure based FEMA and hazard analysis– Consistent with the NASA definition of risk

Probability of an undesired event * Consequences if that event should occur

– Define connectors risk factor as Normalized dynamic coupling * Severity

9

IV&V Facility

Risk Assessment Methodology

• For each use case– For each scenario

• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor

• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor

• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list

– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list

• Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope

10

IV&V Facility






– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios

list • Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case

list • Determine critical component / connector list in the system scope

11

IV&V Facility

NASA CASE STUDY Use Case Diagram

Mode_setting

Single_MT

<<uses>><<uses>>

Single_LT

<<uses>><<uses>>

Dual_LT_Fai led

<<uses>><<uses>>

Dual

<<uses>><<uses>>

Dual_MT_Failed

<<uses>><<uses>>

MT_Pump_Retry

LT_Pump_Retry

Retry_Both_Pumps

SFCA_MT

SFCA_LTPPA_MTPPA_LT

Warnig_for_Total_fai lure

Failure_Recovery

<<uses>><<uses>>

<<uses>><<uses>>

<<uses>><<uses>>

<<uses>><<uses>>

<<uses>><<uses>> <<uses>><<uses>>

<<uses>><<uses>>

<<uses>><<uses>>

Operator

Monitoring

12

IV&V Facility


• For each use case

– For each scenario• For each component

– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor




list • Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case


13

IV&V Facility

Both Pumps Retry- scenario

RPCM_LT / rPCMR2 : RPCM

RPCM_MT / rPCMR1 : RPCM

/ pFMC_MTR1 : PFMC_MT

/ pFMC_LTR1 : PFMC_LT

/ fRITCSR1 : FRITCS

/ sCITCSR1 : SCITCS

Switch_CloseFailed Switch_CloseFailedDual_Mode_OOOODual

1: LT_Fai led1: LT_Fai led

1: MT_Failed1: MT_Failed

2: Pump_Retry(Retry)2: Pump_Retry(Retry)

PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O

3: Open_Switch (void)3: Open_Switch (void)

4: Open_Switch (void)4: Open_Switch (void)

DualSwitch_Open

PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O Switch_Open

5: Close_Switch (void)5: Close_Switch (void)

Switch_ClosePPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O

6: Pump_Retry (void)6: Pump_Retry (void)

PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O Pump_Retry

7: Retry_Success (void)7: Retry_Success (void)

8: Pump_Retry_Success (void)8: Pump_Retry_Success (void)

Dual_Mode_OFOO

8: MT_Operating (void)8: MT_Operating (void)

Dual

Dual_Mode_OFOO Operating

9: Pump_Retry (Pump_Retry_Data{Retry_Type 1,Failure_Type 6})9: Pump_Retry (Pump_Retry_Data{Retry_Type 1,Failure_Type 6})

10: Pump_Retry (void)10: Pump_Retry (void)

Dual_Mode_OFOO Pump_Retry

11: Retry_Success (void)11: Retry_Success (void)

12: Pump_Retry_Success (void)12: Pump_Retry_Success (void)

Dual Dual_Mode_OOOO

12: LT_Operating (void)12: LT_Operating (void)

OperatingDual_Mode_OOOO

14

IV&V Facility







• Calculate system level risk

• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope

15

IV&V Facility

Component Dynamic Complexity

Dynamic Complexity of a component Oi in scenario x is defined as :

Where

CCx(oi ) =ti – ci + 2

Is the cyclomatic complexity of a component Oi in scenario x

Cx(oi ) : A finite set of states for a component Oi

for a scenario x , ci is the cardinality of this set.

Tx(oi ) : A finite set of transitions from one state to another for Oi component in a scenario x,

ti is the cardinality of this set.

Ox : is the set of components collaborating during the execution of a scenario

( ( ))iDOC o

1

( )( )

( )x

x ix i O

x kk

CC oDOC o

CC o

16

IV&V Facility

Component Severity (FEMA)

Component Name Failure Mode Cause of Failure Effect of Failure Criticality of effects

SCITCS Failed to Synchronize with the FRITCS

Error in interpreting FRITCS message

Unable to follow up with the FRITCS recovery action

Major

FRITCS Failed to react to the failure of both pumps

Error in interpreting the message of the pumps or it is in wrong state

Unable to take the required failure recovery procedure of both pumps fail.

Catastrophic

PFMC_LT Failed to report the right status of the LT PUMP

LT PUMP sensor is malfunctioning

Unable to monitor or set the pump correctly

Critical

PFMC_MT Failed to report the right status of the MT PUMP

MT PUMP sensor is malfunctioning

Unable to monitor or set the pump correctly

Critical

RPCM_LT Failed to respond to the FRITCS commands

LT switch controller is malfunctioning

Unable to set the LT switch in the appropriate position

Major

RPCM_MT Failed to respond to the FRITCS commands

MT switch controller is malfunctioning

Unable to set the MT switch in the appropriate position

Major

17

IV&V Facility









18

IV&V Facility

Dynamic coupling for connector between component Oi and component Oj

{ ( , )| , }( , ) x i j i j x i j

x

MT o o o o O o o

x i j MTEOC o o

Connector Dynamic Coupling

MTx(Oi ,Oj) : is the set of messages sent from component Oi to component Oj during the execution of scenario x

MTx : is the set of total messages exchanged between all components during the execution of scenario x .

is the percentage of the number of messages sent from Oi to OJ with respect to the total number of messages exchanged during the scenario x

19

IV&V Facility

Connector Dynamic Coupling

• Dynamic Coupling for Connector between component C1 and component C2 is

EOC(C1,C2) =2/5=0.4

M1

M2

M3

M4

M5

C1C2 C3

20

IV&V Facility

Connector Severity (FEMA)

Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effects

SCITCS-FRITCS Failure to keep the SCITCS and FRITCS synchronized

Wrong message or message has been lost

The FRITCS won’t be able to take appropriate action in case of a pump or valve failure

Minor

FRITCS-SCITCS Failure to keep the SCITCS and FRITCS synchronized


The SCITCS won’t be able to take appropriate action in a mode setting operation

Major

FRITCS-PFMC Unable to retry the pump

Incorrect interpretation of the sent message to the pump

The failure recovery may fail though this pump has not been retried

Critical

FRITCS-RPCM Failed to set the switch in the required setting

Incorrect interpretation of the sent message to the pump

The switch will not be in the position required to make the recovery procedure

Major

PFMC-FRITCS Failed to deliver the right pump state to FRITCS


The whole failure recovery scenario will not be initiated at all as there the case of both pumps fail won’t be detected

Catastrophic

RPCM-FRITCS Failed to report the current status of the switch to FRITCS


The Failure recovery is not responsible for reacting to different switch positions so it won’t be affected much with it

Minor

21

IV&V Facility









22

IV&V Facility

The Markov Chain Model for Both Pumps Retry- scenario

FAILURE STATES OF VARIOUS SEVERITIES

PFMC_LT

Minor

Major

Critical

Catastrophic

FRITCS

SCITCS

RPCM_MT

RPCM_LT

PFMC_MT

S

T

23

IV&V Facility






– Calculate use case level risk factors

– Rank the scenarios based on risk factors, Determine critical scenarios list

• Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case


24

IV&V Facility

Distribution of risk factors of each scenario over the severity classes

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Minor

Marginal

Crtical

Catastrophic

25

IV&V Facility







list


• Rank the use cases based on risk factors, Determine critical use case list

• Determine critical component / connector list in the system scope

26

IV&V Facility

Overall System risk distribution over the severity classes

MINORMARGINALCRITICALCATASTROPHIC

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

MINOR

MARGINAL

CRITICAL

CATASTROPHIC

MINOR MARGINAL CRTICAL CATASTROPHIC

0.3014 0.0103 0.2192 0.2879

The overall system risk factor is: 0.8189

27

IV&V Facility

Sensitivity analysis of components

00.10.20.30.40.50.60.70.80.910.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Component risk factor

Sc

en

ari

o

ris

k

fa

cto

r

LTMTFRITCSRPLTRPMTSCITCS

28

IV&V Facility








• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system

scope

29

IV&V Facility

Determine Critical Component/Connector List

Scitcs

Fritcs

Pfmc_

LT

Pfmc_

MT

Sch

State

_Man

Op_C_Q

App_C

_QN3_

1

N3_2

Rpcm

_LT

Rpcm

_MT

Single_LT

Single_MT

Dual_MT_Failed

Dual_LT_Failed

Dual

Retry_MT_Pump

Retry_LT_Pump

Retrt_Both_Pumps

Monitoring

0

0.1

0.2

0.3

0.4

Risk Factor

Components

Scenarios

Minor Major Critical CatastrophicNot contributing

30

IV&V Facility

• Performance failure is the inability of the system to meet its performance objective(s)

• Define components performance-based risk as Normalized component demand factor * Severity

Performance – based risk

X1 X2 X3

T11 T21

T22 T31

T23

T12

T32

D12

D21

D22

D23

D31

D32

D11 = 11 1

11 2

11 3

T r

T r

T r

d

d

d

kijrTd is demand for resource kr

(e.g, CPU, disk, etc.) in state Tij

(state j of component i )

Scaling vector ][ keSC the resource demands accordingly to the corresponding service times of the resources

scales

31

IV&V Facility

•Total demand of component xi in a scenario Sk is1

i k

l

X S ijj

D D

•Normalized demand factor of component xi in Scenario Sk

DFi = ( . SCT) / ( . SCT)i kX SD

kSD

where m is total number of components and l total number of states for a given component in a given scenario

• Overall demand of a scenario Sk is1 1

k

m l

S iji j

D D

Performance – based risk

32

IV&V Facility

Accomplishments

• Developed analytical techniques and a methodology for Architecture-Based Risk Analysis

• A lightweight approach based on static analysis of dynamic specifications is developed and automated

• A tool will be presented in the Tools session

• Applied the methodology and tool to the NASA case study

33

IV&V Facility

Future Work

• The main thrust of our future work will be in the development of a cohesive methodology for performance- based /reliability- based risk assessment

• Compare risk factors based on other Complexity and coupling metrics obtained from static analysis of UML dynamic specs.– COSMIC-Full Function Point measurement maybe a good complexity

predictor.

– COCOMO II’s effort prediction may be another good complexity predictor

• Validation of methodology using several NASA case studies

34

IV&V Facility

Publications

1. Sherif M. Yacoub, Hany H. Ammar , “A Methodology for Architecture-Level Reliability Risk Analysis,” IEEE Transactions on Software Engineering, June 2002, pp. 529-547

2. H. H. Ammar, T. Nikzadeh, and J. B. Dugan "Risk Assessment of Software Systems Specifications," IEEE Transactions on Reliability, September 2001

3. Hany H. Ammar, Sherif M. Yacoub, Alaa Ibrahim, “A Fault Model for Fault Injection Analysis of Dynamic UML Specifications,” International Symposium on software Reliability Engineering, IEEE Computer Society, November 2001

4. Rania M. Elnaggar, Vittorio Cortellessa, Hany Ammar, “A UML-based Architectural Model for Timing and Performance Analyses of GSM Radio Subsystem” , 5th World Multi-Conference on Systems, Cybernetics and Informatics, July. 2001, Received Best Paper Award

5. Ahmed Hassan, Walid M. Abdelmoez, Rania M. Elnaggar, Hany H. Ammar, “An Approach to Measure the Quality of Software Designs from UML Specifications,” 5th World Multi-Conference on Systems, Cybernetics and Informatics and the 7th international conference on information systems, analysis and synthesis ISAS July. 2001.

6. Hany H. Ammar, Vittorio Cortellessa, Alaa Ibrahim “Modeling Resources in a UML-based Simulative Environment”, ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'2001), Beirut, Lebanon, 26-29 June 2001

7. A. Ibrahim, Sherif M. Yacoub, Hany H. Ammar, “Architectural-Level Risk Analysis for UML Dynamic Specifications,” Proceedings of the 9th International Conference on Software Quality Management (SQM2001), Loughborough University, England, April 18-20, 2001, pp. 179-190

iv&v facility 1 fy 2002 initiative iv&v of uml hany ammar, katerina goseva-popstojanova, v....

Documents