global sensitivity analysis of predictor models in software engineering
DESCRIPTION
Stefan WagnerTRANSCRIPT
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 1 / 27
Global Sensitivity Analysis of Predictor
Models in Software Engineering
Stefan WagnerSoftware & Systems EngineeringTechnische Universitat Munchen
May 20, 2007
Introduction
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 2 / 27
Predictor Models in Software Engineering
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 3 / 27
■ Predictor models describe situations in software engineering■ Various types, aims, . . .■ Examples
◆ COCOMO (costs)◆ Musa-Okumoto (reliability)◆ Fault-proneness
Problem
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 4 / 27
■ The models themselves can be complex■ Their use needs a lot of effort■ How can I analyse those models themeselves?■ How can I simplify them?■ How do I improve their predictive power?
Example: Fischer-Wagner Model
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 5 / 27
■ Reliability model used at Siemens COM■ Two parameters estimated by failure data■ Failure probability of a fault: pa = p1 · d
(a−1)
■ Geometrical distribution : Fa(t) = 1 − (1 − pa)t
■ Cummulated failures up to t:µ(t) =
∑∞
a=1 1 − (1 − p1 · d(a−1))
■ Input parameters
◆ p1: Highest failure probability of a fault◆ d: Complexity of the system ]0; 1[◆ t: Execution time (incidents)◆ inf : approximation of ∞
■ Output: µ(t)
More details: S. Wagner, H. Fischer. Ada-Europe, 2006
Questions About the Model
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 6 / 27
■ How do the input parameters influence the output?■ Which parameter(s) do I have to estimate best to get a good
prediction?■ Are there insignificant parameters that I can remove?
Overview
IntroductionPredictor Models inSoftwareEngineering
ProblemExample:Fischer-WagnerModelQuestions About theModel
Overview
Global SensitivityAnalysis
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 7 / 27
Introduction
Global Sensitivity Analysis
Approach for SE
Summary
Global Sensitivity Analysis
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 8 / 27
Definition
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 9 / 27
■ Saltelli (2000): “Sensitivity analysis studies the relationshipsbetween information flowing in and out of the model.”
■ How do the input parameters influence the output?■ How can this influence by quantified?■ Global properties
◆ Inclusion of influence of shape and scale◆ Multidimensional averaging
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
x1
x2
x3
xn
Yf(x)
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Method
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 10 / 27
Input Model Output
Yf(x)
x1
x2
x3
xn
(Based on slide of S. Tarantola)
Variance-based Methods
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 11 / 27
■ Global analysis usually variance-based■ “model-free”■ Analysis on the basis of sensitivity indices
◆ Main or first-order effect
◆ Higher-order effects
◆ Total effect
■ Decomposition in main effects and interactions
y = f(x) = f0 +k∑
i=1
fi(xi) +∑
i
∑
j>i
fij(xi, xj)
+ . . . + f1,2,...,k(x1, x2, . . . , xk)
Variance-based Methods
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 11 / 27
■ Global analysis usually variance-based■ “model-free”■ Analysis on the basis of sensitivity indices
◆ Main or first-order effect
◆ Higher-order effects
◆ Total effect
■ Decomposition in main effects and interactions
For example k = 3:
f(x) = f0 + f1(x1) + f2(x2) + f3(x3) + f12(x1, x2)
+f13(x1, x3) + f23(x2, x3) + f123(x1, x2, x3)
Main Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 12 / 27
Decomposition of the variance V of f(x)
Var(y) = V =k∑
i=1
Vi +∑
i
∑
j
Vij
∑
i
∑
j
∑
k
Vijk + . . . + V1,2,...k
First-order sensitivity coefficient
Si =Vi
V
(Factors Priorisation)
Higher-Order Effects
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 13 / 27
Higher-order indices analog
Si1...io = Vi1...io/V
With k = 4:2. order: S12, S13, S14, S23, S24, S34
3. order: S123, S124, S134, S234
4. order: S1234
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
S123 + S124 + S134 + S234 + S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
S123 + S124 + S134 + S234 + S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
S123 + S124 + S134 + S234 + S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23 + S24 + S34 +
S123 + S124 + S134 + S234 + S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +
S123 + S124 + S134 + S234 + S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +
S123 + S124 + S134 + S234//////+ S1234 = 1
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +
S123 + S124 + S134 + S234//////+ S1234 = 1
ST1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234
(Factors Fixing)
Total Effect
Introduction
Global SensitivityAnalysis
Definition
MethodVariance-basedMethods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 14 / 27
■ Sum of the first- and higher-order effects of xi
■ Removal of the non-relevant parts
S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +
S123 + S124 + S134 + S234//////+ S1234 = 1
ST1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234
ST2 = S2 + S12 + S23 + S24 + S123 + S124 + S234 + S1234
(Factors Fixing)
Approach for SE
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 15 / 27
Overview of the Approach
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 16 / 27
Inputdistributions
.045
.003
.002
.375
Sensitivityindices
Determiningdistributions
Visualisingusing
scatterplots
Globalsensitivityanalyses
Scatterplots
Determining Distributions
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 17 / 27
■ Distribution of the values of the input parameteters■ Derived from
◆ scientific literature◆ physical boundaries◆ expert opinion◆ surveys◆ experiments
Example: Distributions
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 18 / 27
■ Based on expert opinion■ General parameter
◆ p1 uniformly distributed between 0 and 1◆ d uniformly distributed between 0.9 and 1
■ Dependent on application
◆ inf e.g.. uniformly distributed between 50 and 500◆ For t e.g. t ∼ N (1000, 200)
Visualising Using Scatterplots
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 19 / 27
■ Sampling using the distributions■ Pairwise relationship between factors■ Detection of errors in the model■ Indications of influences
p1 and µ
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 20 / 27
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0
100
200
300
400
500
600
700
800
900
1
p1
µ
d and µ
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 21 / 27
0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 0
100
200
300
400
500
600
700
800
900
µ
d
t and µ
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 22 / 27
99200 99400 99600 99800 100000 100200 100400 100600 100800 0
100
200
300
400
500
600
700
800
900
µ
t
inf and µ
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 23 / 27
900
800
700
600
500
400
300
200
100
0 400 450 500 550 600 650 700 750 800 850 900
µ
inf
Applying Global SA
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 24 / 27
■ Various methods for calculating indices■ FAST (Fourier Amplitude Sensitivity Test) gives quantitative
results■ Uses sampled data■ Tool-support: Simlab
Indizes
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 25 / 27
Main Effect
t 6.88e-5p1 0.0090d 0.9026inf 0.0158
Indizes
Introduction
Global SensitivityAnalysis
Approach for SE
Overview of theApproach
DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 25 / 27
Main Effect
t 6.88e-5p1 0.0090d 0.9026inf 0.0158
Total Effect
t 0.005788p1 0.022365d 0.975155inf 0.086735
Summary
Introduction
Global SensitivityAnalysis
Approach for SE
Summary
Conclusions
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 26 / 27
Conclusions
Introduction
Global SensitivityAnalysis
Approach for SE
Summary
Conclusions
Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 27 / 27
■ Sensitivity analysis useful tool for predictor models■ SA can help to
◆ find errors in the models◆ simplify the models◆ identify interactions between input parameters◆ identify parameters that should be investigated more◆ get more robust predictions
■ Experience with
◆ reliability model◆ QA economics model◆ process model◆ expert system for IT tools
■ Good tool-support available (Simlab:http://simlab.jrc.cec.eu.int/)