research heaven, west virginia fy2003 initiative: hany ammar, mark shereshevsky, walid abdelmoez,...
TRANSCRIPT
Research Heaven,West Virginia
FY2003 Initiative:
Hany Ammar, Mark Shereshevsky,Walid AbdelMoez, Rajesh Gunnalan, and Ahmad Hassan
LANE Department of Computer Science and Electrical EngineeringWest Virginia University
The OSMA Software Assurance Symposium July 30, 2003Lakeview, WV
WVU UI: WVU UI: Quantitative Relations Between Static and Dynamic
Software Metrics
NASA OSMA SAS’03 2
Research Heaven,West Virginia
Two main views of software systems: The static view of the structure and composition,
The dynamic view during execution
• Static and dynamic properties of software have been widely studied in the literature.
NASA OSMA SAS’03 3
Research Heaven,West Virginia
So how good is static analysis?
• IV&V testing is mostly focused of static analysis– Software rarely executed at Fairmont.
• Why?– NASA software supports NASA hardware
– So, some software can’t run without its associated hardware
• Traditional view:– Dynamic execution more informative than
static analysis
• This research:– To what extent are static measure
surrogate for dynamic measures?
– Compare insights gained from static and dynamic measures
NASA OSMA SAS’03 4
Research Heaven,West Virginia
This talk
• Propose three hypotheses of possible correlations between selected static and dynamic metrics– Hypothesis I: Static coupling
metrics correlate with error propagation in software architectures.
– Hypothesis II: Static error propagation correlates with dynamic error propagation
– Hypothesis III: Change proneness” correlates with dynamic coupling of components
• Experiments to measure – static metrics
– dynamic metrics
– for selected case studies
• Statistical analysis of the data:– computing correlations
– developing linear or non-linear regression models.
• In summary:– Hypothesis I: not quite
supported– Hypothesis II: supported– Hypothesis III: rejected
NASA OSMA SAS’03 5
Research Heaven,West Virginia
Project Overview
FY03
• Conduct empirical study of the relationship between the static and dynamic metrics.
• Measure Static Metrics • Measure related Dynamic Metrics • Conduct statistical Analysis of the
Data
FY04
• Establish Relationships between Static and Dynamic Metrics
Research Heaven,West Virginia
Tools
NASA OSMA SAS’03 7
Research Heaven,West VirginiaTools for Static Metrics
• We can automatically collect several static metrics (OO metrics,Complexity metrics and Size/Volume metrics).
• We are currently using Understand for Java and C++ by Scientific Toolworks for collecting static metrics.
NASA OSMA SAS’03 8
Research Heaven,West Virginia
Tools for Dynamic Metrics
• Rational Rose-Realtime simulation facility is used to simulate UML models
• At the code level, Rational Purifyplus produces dynamic sequence diagrams at run-time
• JProbe profiler by Quest Software serves a similar purpose for Java code but does not produce sequence diagrams.
Research Heaven,West Virginia
Hypothesis I
Static coupling metrics correlate with error propagation in software architectures.
NASA OSMA SAS’03 10
Research Heaven,West Virginia
Study of Static Coupling Metrics and Error Propagation
• Correlate the experimental data on – error propagation – static information coupling (connector based).
• Correlate:– experimental error propagation – and static NAS (number of associations) coupling measure
(connector Based).
• Correlate:– experimental error propagation – and CBO (connector based).
NASA OSMA SAS’03 11
Research Heaven,West Virginia
NASA OSMA SAS’03 12
Research Heaven,West Virginia
Table Highlighting Non-zero Values Information Flow Vs Ep
From To1 1 5 0.1331 2.3219 10672 1 6 0.128 2.3219 10553 1 3 0.0557 2 5924 3 1 0.0161 1.585 5595 7 1 0.75 2.3219 646 1 4 0.4912 2 577 2 1 0.7838 2.8074 378 1 2 0.5 2.585 369 4 2 0.6429 2 28
10 4 1 0.7083 2 2411 3 2 0.7917 1.585 2412 2 8 0.4286 2 713 10 1 1 1 414 6 2 1 1 415 5 2 1 1 4
EntryConnector EP
(Dynamic)# Fault
InjectionsInformation Flow(Stat)
NASA OSMA SAS’03 13
Research Heaven,West Virginia
Table Highlighting Non-zero Values CBO Vs Ep
From To1 1 5 0.1331 16 10672 1 6 0.128 16 10553 1 3 0.0557 15 5924 3 1 0.0161 3 5595 7 1 0.75 5 646 1 4 0.4912 15 577 2 1 0.7838 31 378 1 2 0.5 6 369 4 2 0.6429 4 28
10 4 1 0.7083 3 2411 3 2 0.7917 4 2412 2 8 0.4286 14 713 10 1 1 3 414 6 2 1 2 415 5 2 1 2 4
EntryConnector EP
(Dynamic)# Fault
InjectionsCBO
NASA OSMA SAS’03 14
Research Heaven,West Virginia
Correlation Values
• Correlation between Information Flow Coupling and Error Propagation is 0.545657
• Correlation between CBO and Error Propagation is 0.130478
• Correlation between NAS and Error Propagation is 0.0• The correlations are only for the values where the # fault
injections >25.
NASA OSMA SAS’03 15
Research Heaven,West Virginia
Discussion of Results
• Information flow coupling metric shows a higher correlation with dynamic error propagation than CBO.
• On further analysis the R2 correlation between Information Flow and EP was 0.297742.
• This low correlation value could be partly due to the small size of our sample (9 data points)
• This result is not sufficient to validate the Hypothesis I
Research Heaven,West Virginia
Hypothesis II:
Static error propagation correlates with dynamic error propagation
NASA OSMA SAS’03 17
Research Heaven,West Virginia
Static and Dynamic Error Propagation
• We will use the experimental data on error propagation and the static error propagation measure(connector Based).
• The static error propagation measure is based on the relation developed in the software architecture metrics USIP using the information flow between components.
NASA OSMA SAS’03 18
Research Heaven,West Virginia
Experimental (Dynamic) Error Propagation Matrix of HCS
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C1 1 0.11 0.15 0.3 0.76 0.73 0 0 0 0C2 0.37 1 0 0 0 0 0 0.52 0 0C3 0.03 0.32 1 0 0 0 0 0 0 0C4 0.03 0.19 0 1 0 0 0 0 0 0C5 0 0.67 0 0 1 0 0 0 0 0C6 0 0.28 0 0 0 1 0 0 0 0C7 0.32 0 0 0 0 0 1 0 0 0C8 0 0 0 0 0 0 0 1 0 0C9 0 0 0 0 0 0 0 0 1 0
C10 0 0 0 0 0 0 0 0 0 1
B
A
NASA OSMA SAS’03 19
Research Heaven,West Virginia
Static Error Propagation Matrix of HCS
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C1 1 0.11 0.38 0.44 0.58 0.59 0 0 0 0C2 0.18 1 0 0 0 0 0 0.5 0 0C3 0.11 0.26 1 0 0 0 0 0 0 0C4 0.17 0.15 0 1 0 0 0 0 0 0C5 0 0.57 0 0 1 0 0 0 0 0C6 0 0.18 0 0 0 1 0 0 0 0C7 0.23 0 0 0 0 0 1 0 0 0C8 0 0 0 0 0 0 0 1 0 0C9 0 0 0 0 0 0 0 0 1 0
C10 0.08 0 0 0 0 0 0 0 0 1
B
A
NASA OSMA SAS’03 20
Research Heaven,West Virginia
Correlation Results
• There is a strong correlation between static error propagation and dynamic error propagation.
• Correlation = 0.875098• R2 = 0.765797• This supports Hypothesis II and our conjecture that
some static metrics correlate with related dynamic metrics.
Research Heaven,West Virginia
Hypothesis III:
“Change proneness” correlates with dynamic coupling of components
NASA OSMA SAS’03 22
Research Heaven,West Virginia
Change Proneness and Change Propagation
• Hypothesis III “change proneness correlates with dynamic coupling of components”– from the study of Erik Arisholm June 2002, – Dynamic Coupling Measures for Object-Oriented Software,
– Eighth IEEE Symposium on Software Metrics
• Correlate – dynamic import coupling, – dynamic export coupling metrics – with change proneness measured – based on the software architecture metrics USIP using interface
change propagation probabilities.
NASA OSMA SAS’03 23
Research Heaven,West Virginia
Dynamic Coupling Metrics
• The dynamic coupling metrics we focus on are as follows1. Dynamic Export Coupling
2. Dynamic Import Coupling
3. Total Dynamic Coupling sum of both import and export coupling.
NASA OSMA SAS’03 24
Research Heaven,West Virginia
Experiment
• The experiment was conducted on the SharpTool case study.with 32 components.
• Dynamic coupling was obtained by counting the object interactions dynamically at run time. During the run all functionalities of the tool were executed.
NASA OSMA SAS’03 25
Research Heaven,West Virginia
Experiment Results
Export Import Proneness18 14 0.2
39466 0 0.3757317 20 07182 7179 3.4
75 2 10 0 10 0 1
78 0 2.523 97 0.510 48 08 3 19 25 0.9591 1 03 5 0
23 78 0.85718 14 0.734
123 151 0.56 9 0.151
92 0 0.677 0 25 0 19 5 01 0 0
374 32893 1.541 95 053 3 0.5411 44 0.603
153 15208 0.75181 215 013 5 0.42 158 0.8889 60 1.142
NASA OSMA SAS’03 26
Research Heaven,West Virginia
Discussion of Results
• Zero correlation between – change proneness – and dynamic coupling.
• We also computed CBO and found zero correlation between change proneness and CBO
• This negates Hypothesis III.• The study in the paper by Erik Arisholm was done
using actual project changes data .• In our experiment we used static change proneness data
using our interface change propagation probabilities defined in the architecture metrics USIP
Research Heaven,West Virginia
NASA OSMA SAS’03 28
Research Heaven,West Virginia
So how good is static analysis?
• Hypothesis I: Static coupling metrics correlate with error propagation in software architectures.– Not Sure
• Hypothesis II: Static error propagation correlates with dynamic error propagation– Yes
• Hypothesis III: Change proneness” correlates with dynamic coupling of components– No
• And so, what does all this mean? – IV&V can not run code – And still offer value added to NASA
systems