metamodelling and sensitivity analysis of models with
TRANSCRIPT
IMACS seminar on MCM 28/08/2011 – 02/09/2011
Metamodelling and Sensitivity Analysis of Metamodelling and Sensitivity Analysis of Models with Dependent Variables
Sergei Kucherenko,Miguel Munoz Zuniga,
Nilay ShahImperial College London, London, SW7 2AZ, UK
Outline
ANOVA decomposition
Global Sensitivity Analysis and Sobol’ indices
D d t i bl C lDependent variables. Copula
Metamodelling based on Quasi Random Sampling - HighMetamodelling based on Quasi Random Sampling High Dimensional Model Representation
Two methods for dealing with dependent inputs
T t ltTest results
ANOVA decomposition and Sobol’ Sensitivity Indices (SI)
Consider a function f(x) x is a vector of input variables
2( ) [0 ,1]
( ) [0 1]
n
n
f x L
x x x x
∈
= ∈f(x) is integrable
ANOVA decomposition is unique if variables are independent
1 2( , , ..., ) [0 ,1]kx x x x= ∈
( ) ( ) ( )0 1,2,..., 1 21
( ) , ... , , ..., ,k
i i ij i j k ki i j i
f x f f x f x x f x x x= >
= + + + +∑ ∑∑
ANOVA decomposition is unique if variables are independent
1 1 1 1
1
1 1
... ... ...0 0
( , , ..., ) 0, , 1 , 0,s s i s l i ik k l
i i j i
i i i i i i i i k lf x x dx k k s f f dx dx i i
= >
= ∀ ≤ ≤ → = ∀ ≠∫ ∫0 0
Variance decomposition: 2 2 2 2, 1,2,...,i iji i j nD D D D= + +∑ ∑ …
kijlij
k
i SSSS 21...1 ++++= ∑∑∑Sobol’ SI:
3
klji
ijlji
iji
i ,...,2,11
∑∑∑<<<=
Sobol’ Sensitivity Indices (SI)
Definition:1 1
2 2... ... /
s si i i iS D D=
- partial variances( )1 1 1 1
12 2... ...
0
,..., ,...,s si i i i i is i isD f x x dx x= ∫
12
∫ - total variance
Sensitivity indices for subsets of variables:
( )( )220
0
D f x f dx= −∫( )x y z=Sensitivity indices for subsets of variables: ( ),x y z=
( )1
2 2, ,
1s
m
y i is i i
D D= ⟨ ⟨ ∈Κ
=∑ ∑ …
Total variance for a subset:( )11 ... ss i i= ⟨ ⟨ ∈Κ
( )2 2 2toty zD D D= −
Corresponding global sensitivity indices:
2 2/S D D ( )2 2/tot totS D D/ ,y yS D D= ( ) / .y yS D D=
How to use How to use SobolSobol’ Sensitivity ’ Sensitivity Indices ?Indices ?
0 1toty yS S≤ ≤ ≤
tot SS t f ll i t ti b t d ( )ytoty SS −
iS totiS
accounts for all interactions between y and z, x=(y,z).
Important indices in practice are and
( )0totiS f x= → does not depend on ;
only depends on ;
ix( )1iS f x= → ix
corresponds to the absence of interactions between
and other variables
( )i itotii SS =
ix
If then function has an additive structure:
Fixing unessential variables
∑=
=n
siS
1,1 ( ) ( )0 i i
i
f x f f x= +∑Fixing unessential variables
If does not depend on so it can be fixed( )1totzS f x<< → z
( ) ( )f f complexity reduction, from to variables( ) ( )0,f x f y z≈ → znn −n
Evaluation of Sobol’ Sensitivity Indices.Independent Variablesdepe de a ab es
Straightforward use of Anova decomposition requires 2n integral evaluations – not practical !2 integral evaluations not practical !
There are efficient direct formulas for evaluation of Sobol’ Sensitivity indices ( Sobol’ 2001 S K 2002 2009) :
11 [ ( )[ ( ') ( ' ')] ' 'S f y z f y z f y z dydy dzdz= ∫
indices ( Sobol 2001, S.K. 2002,2009) :
2 0
1 22
[ ( , )[ ( , ) ( , )] ,
1 [ ( , ) ( ', )] ',
y
tot
S f y z f y z f y z dydy dzdzD
S f y z f y z dydzdz
= −
= −
∫
∫2 0
12 2 200
[ ( , ) ( , )] ,2
( , )
yS f y z f y z dydzdzD
D f y z dydz f= −
∫
∫0∫
Evaluation of SI is reduced to high-dimensional integration using MC/QMC methods
Effective dimensions
superposition seThe effective dimension of ( ) in is the smallest integer s
Let u be a cardinality of a set of variables .
uch thanse
tf x
d
u
0
is the smallest integer s
uch that(1 ), 1
S
S
uu d
dS ε ε
< <≥ − <<∑
It means that ( ) is almost a sum of f x -dimensional functions.Sd___________________________________________________________
is
The effective dimension of ( ) in the smallest integer such that
truncation senseT
f xd
( )
{1,2,..., }(1 ),
E l
1T
n
uu d
f d d
S ε ε⊆
≥ − <<
∑
∑
( )1
1,Example
does not depend on the o
:
ri
i TS
S
nf x x d d
d=
→ = ==∑der in which the input variables are sampled,
can be reduced- depends on the order by reodering variables important property:
T T
TS
d dd d≤
→
Classification of functions
F.E .: Majority of problems in finance either have low effective dimensions or effective dimensions can be reduced by using special y g ptechniques ( Brownian Bridge), hence QMC is always more efficient than MC regardless of n
Global Sensitivity Analysis of the standard and Brownian Bridge discretizations. Asian call option
For the standard discretization1/2
10C( , ) max[0,( exp[( ) ( )]
2
nn i
rTi j
TK T e S r t un
− −⎡ ⎤σ
= − +σ Φ⎢ ⎥⎢ ⎥⎣ ⎦
∑∏∫
.
11
1
2
)] ... n jiH
n
n
K du du==⎢ ⎥⎣ ⎦
−
∫
.
1 2( ), ( , ,..., ) uncertaint parameterskY f x x x x x= = −
Apply global SA to a payoff function :1/2
n
n iT⎡ ⎤21
011
( ) max(0, exp[( ) ( )]2
n i
i jji
Tf x S r t xn
−
==
⎡ ⎤σ= − +σ Φ⎢ ⎥
⎢ ⎥⎣ ⎦∑∏
Asian call. Convergence curves
Asian Call with geometric averaging 252 observations Asian Call with geometric averaging. 252 observations S=100, K=105, r=0.05, s=0.2, T=1.0, C=5.56
1
10
1/ 21 K⎛ ⎞
0.1
1
(RM
SE)
2
1
1 ( )K
kN
k
I IK
ε=
⎛ ⎞= −⎜ ⎟⎝ ⎠∑
0.01
Log(
QMC, Brownian BridgeQMC, Standard ApproximationMC Brownian Bridgeα
0.00110 100 1000 10000
MC, Brownian Bridge Trendline -QMC, BB, 1/N 0̂.82Trendline - QMC, Stand., 1/N 0̂.56Trendline - MC, 1/N 0̂.5
~ , 0 1N αε α− < <
Log-log plot of the root mean square error
Log(N_path)
versus the number of paths. Brownian bridge – much faster convergence with QMC methods: ~1/N0.8 even in high dimensions. Why ?
Global Sensitivity Analysis of the standard and Brownian Bridge discretizations . Asian call optiong p
Apply global SA to payoff function i i i({Z })=max( ({Z })- ,0), {Z }, 1,AP S K i n=
1Standard Approximation
Brownian Bridge
0 01
0.1
otal
0.001
0.01
S_to
0.00011 6 11 16 21 26 31
time step number
Log of total sensitivity indices versus time step number i.
S d d di i i Si l l l d i h i B i
time step number
Standard discretization - Si_total slowly decrease with i. Brownian bridge - Si_total of the first few variables are much larger than those of the subsequent variables.
Global Sensitivity Analysis of two algorithms at different n (time steps)g p
The effective dimensions
Standard approximation: dT ≈ ndS > 2
Brownian Bridge approximation: dT ≈ 2d 2dS ≈ 2
Evaluation of Evaluation of SobolSobol’ Sensitivity ’ Sensitivity Indices Indices Dependent Variablesepe de a ab es
[ ( ( , ) | )] [ ( ( , ) | )]y z y zD D E f y z y E D f y z y= +
[ ( ( , ) | )]y zy
D E f y z yS
D=
first order effects[ ( ( , ) | )]
1 z yT E D f y z zS S
→
= − =1
total order effects
y zS SD
→
⎡ ⎤⎡ ⎤' ' ' 20
1 ( ) ( , ) ( , | ) ( , ) ( , | )s n s n s
yR R R
S p y dy f y z p y z y dz f y z p y z y dz fD − −
⎡ ⎤⎡ ⎤= −⎢ ⎥⎢ ⎥
⎢ ⎥⎣ ⎦⎣ ⎦∫ ∫ ∫
1yS ' 2 ' '1 [ ( , ) ( , )] ( ) ( , | ) ( , | )
2Evaluation of SI requires sampling from multivariate probability distributions
n s
T
R
f y z f y z p z p y z z p y z z dydy dzD +
= −∫Evaluation of SI requires sampling from multivariate probability distributions( ( ), ( , | ), etc)p y p y z y
Copula. Uniform distrubutions
1,..., , [0,1], 1,..., , - correlation matrix . n i uu u u u i n= ∈ = Σ
1 11 1
Gaussian copula function: ( ,..., ; ) ( ( ),..., ( ); ) .n u n nC u u F F u F u− −Σ = Σ( ) - n-variate cumulative normal distribution function (NDF)( ) u
n
i
FF
ξξ − nivariate NDF.1 - inverse NDF F −
(independent uniform) (independent normal) (dependent normal, ) (dependent uniform, )uu
u ξξ
→→ Σ → Σ(It requires mapping ) ( )W
u
T uuΣ →
=Σ
l th i t f ti We1
can also use the inverse transformation ( )u T u−=
Copula. Arbitrary distributions
In a general case of arbitrary distributed r.v. X with and cumulative marginal distribution functions G (X )
x
i i
Σ
1 11 1 1 1 1
g ( )Gaussian copula function:
( ( ) ( ); ) ( ( ( )) ( ( )); )
i i
C G X G X F F G X F G X− −Σ = Σ1 1 1 1 1( ( ),..., ( ); ) ( ( ( )),..., ( ( )); )
n n x n nC G X G X F F G X F G XΣ Σ
(inu dependent uniform) (independent normal)(dependent normal ) X(dependent r v )
ξξ
→→ Σ → Σ(dependent normal, ) X(dependent r.v, )(Require mapping )
( )
x
x
X T u
ξ→ Σ → Σ
Σ →Σ
= ( )We can also apply the inverse transformation
X T u=
1( )u T X− ( )u T X=
Evaluation of Sobol’ Sensitivity Indices. Dependent Inputs
1 1
Main effect SI:
1 ⎡ ⎡⎢ ⎢∫ ∫ 1 11 ( ) ( ( ( )), ( ( ))) ( , | )
s n sy s s s n s n s n s
R R
S y dy f G F y G F z y z y dzD −
− −− − −= Φ Φ⎢ ⎢
⎢⎢ ⎣⎣⎤
⎤
∫ ∫
∫ 1 1 ' ' ' 20 ( ( ( )), ( ( ))) ( , | ) ,
Total order effect SI:n s
s s n s n s n sR
f G F y G F z y z y dz f−
− −− − −
⎤⎤Φ −⎥ ⎦⎥⎦
∫
1
Total order effect SI:1 [ ( (
2Ty sS f G F
D−= 1 ' 1 2( )), ( ))) ( ( ( )), ( ( )))]
n ss n s s s n s n s
R
y F z f G F y G F z+
− −− − −− ⋅∫
' ' ( ) ( , | ) ( , | ) .Here
R
n s s sz y z z y z z dydy dz−⋅Φ Φ Φ
1
1 1 11 1
1( ) ( )2
( ( )) ( ( ( ),..., ( ( )),
1( )T
s s s s
x x
G F y G F x G F xµ µ−
− − −
− − Σ −
=
Φ 2/ 2
( )(2 ) | |n n
x eπ
Φ =Σ
Dependent Input Case.Test example (Gaussian Hyperplane)
• Model:
Test example (Gaussian Hyperplane)
Inputs are Gaussians
( )1 2 3 1 2 3, ,Y f X X X X X X= = + +
Gaussians
• Correlation matrix: 1 0 00 1 ρσ⎛ ⎞⎜ ⎟Σ = ⎜ ⎟⎜ ⎟
1 1 ;TS S
• Sensitivity indices :0 1ρσ⎜ ⎟⎜ ⎟⎝ ⎠
( )
1 12 2
2 2
, ;2 2 2 2
1 1 ;T
S S
S S
σ ρσ σ ρσ
ρσ ρ
= =+ + + +
+ −= =
( ) ( )2 22 2
2 2 2
, ;2 2 2 2
1T
S S
S S
σ ρσ σ ρσ
σ ρσ ρ
= =+ + + +
−+= =3 32 2, .
2 2 2 2S S
σ ρσ σ ρσ= =
+ + + +
Evolution of the first and total order indicesat different values of correlation ratio ρat different values of correlation ratio ρ
Quasi MC sample size N = 213 , σ=2.0
0 80.9
1
ndex
0.50.60.70.8
Sens
itivi
ty I
S1S2S3ST1
0.10.20.30.4
Ana
lytic
al S ST1
ST2ST3
00.1
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
Correlation coefficient between x 2 and x 3
A
2
2, 2,3 if 0 or Ti iS S i σρ ρ≤ = ≥ ≤ − 2
2 3
, ,1
0, 0 if 1
i i
T TS S
ρ ρσ
ρ+
→ → →
Random Sampling - High Dimensional Model Representation
1 10 ...( ) ( ) ( ,..., )s s
d s
i i i if x h x f f x x≈ = +∑ ∑For many problems only low
[ ]11 ...
21( , ) ( ) ( )
ss i i
f h f x h x dxD
δ
= < <
= −∫
For many problems only low order terms in the ANOVA decomposition are important.
1 1
1
...1 ...
( , ) 1 ( ,..., )s s
s
d s
i i i is i i
f h S x xδ= < <
= −∑ ∑
( )h x is a metamodel (HDMR), Rabitz et al: d=2
( ) ( )0( ) ,n
i i ij i jh x f f x f x x= + +∑ ∑∑1i i j i= >
HDMR: polynomial approximation
( ) ( )( ) ( )n
f h f f f+ +∑ ∑∑( ) ( )01
( ) ( ) ,i i ij i ji i j i
f x h x f f x f x x= >
≈ = + +∑ ∑∑
• Orthonormal polynomial expansion:'
1 1 1
( ) ( ) , ( , ) ( , )k l l
i iji i r r i ij i j pq pq i j
r p q
f x x f x x x xα φ β φ= = =
≈ ≈∑ ∑∑Orthonormal polynomial expansion:
( ) ( ) ( ) ( ) ( )n n
i ijr r i pq p i q jH H
f x x dx f x x x dxα φ β φ φ= =∫ ∫
• Orthonormal polynomial coefficients:
pq p q jH H∫ ∫• Orthonormal polynomials:
1 2( ) ( ) nkf x x x x x H= ∈for
Use modified Legendre polynomials defined in Hn
1 2( ) , ( , , ..., )kf x x x x x H∈for
Dependent Input Case
Problem:Problem:
ANOVA decomposition is unique only if the inputs are independent
Possible Solutions:
A. Transform dependent inputs to independent and use the methodology for an independent case.
B. Construct an orthonormal basis directly without a tranformationto independent inputsto independent inputs.
First method for the dependent Input Case: transformation to independenttransformation to independent
ModelInputs Output(s)
( )X X f ( )Y f X X
Any
( )1, , dX X… f ( )1, , dY f X X= …
( )11, , df T U U−⎡ ⎤= ⎣ ⎦…
Any dependent
RandomInputsp
Model1T − T
( )1, , dU U…Uniform
Independent Random Inputs
Model(independent)
1f T −
Random Inputs
First method for the dependent Input Case: Polynomial chaos expansionPolynomial chaos expansion
Inputs Output(s)
( )
Model(independent)
( )11, , dY f T U U−⎡ ⎤= ⎣ ⎦…( )1, , dU U…
Uniform Independent
1f T −
( )a U Uφ+∞
=∑Independent Random Inputs
≈1
1( ,..., )i i d
ia U Uφ
=
=∑
Metamodel1
1
ˆ ˆ ( ,..., )M
i i di
Y a U Uφ=
=∑
Legendre
ˆ( , )a φ
gpolynomials1
1 11
1ˆ ( ,..., ) ( ,..., )N
j j j ji d i d
j
a f T U U U UN
φ−
=
= ∑Coefficients are estimated by MC/QMC
Second method for the dependent Input Case: direct polynomial chaos expansiondirect polynomial chaos expansion
( )1, , dY f X X= …Model: Marginal pdf
( ) ( )( ) ( )di
X
dXdXdi XX
XpXpXp
XX ,...,...
),...,( 1
2/11
11 Φ⎟⎟
⎠
⎞⎜⎜⎝
⎛=ΨOrthonormal
Basis:
∑+∞
Ψ= 1 ),...,( dii XXaYExact polynomial E i
joint pdfStandard orthogonal polynomials∑
=1i
∑ Ψ=M
XXaY )(ˆˆ
Expansion:
Metamodel: ∑=
Ψ=i
dii XXaY1
1 ),...,(
∑n1
Metamodel:
Weight estimation ∑=
Ψ=j
jd
ji
jd
ji XXXXf
na
111 ),...,(),...,(1ˆ
Weight estimation by MC:
Dependent Input Case.Test example (Gaussian Hyperplane)Test example (Gaussian Hyperplane)
Inp ts a e • Model:
( )Y f X X X X X X
Inputs are Gaussians
• Correlation matrix:
( )1 2 3 1 2 3, ,Y f X X X X X X= = + +
1 0 00 1 ρ⎛ ⎞⎜ ⎟Σ = ⎜ ⎟
• Error estimate:
0 1ρ
ρ⎜ ⎟⎜ ⎟⎝ ⎠
[ ]21( , ) ( ) ( )f h f x h x dxδ = −∫[ ]( , ) ( ) ( )f h f x h x dxD
δ ∫
Test example (Hyperplane)First Method.First Method.
Only first order terms in the ANOVA polynomial expansion are present
[1; 1; 1]
Number of QMC sampling N = 1024
l ( ) 10δ
0,0.5,0.9ρ =
2log ( ) 10δ ≈ −
Test example (Hyperplane)First Method.First Method.
0ρ = 0.5ρ =0ρ = 0.5ρ
0.9ρ =
Test example (Hyperplane)Second Method.
Number of Quasi MC sampling points N = 1024
Second Method.
9.0=ρ (2nd orders for all l f i bl )
Number of Quasi MC sampling points N 1024
10
20
30
couples of variables)
4
6
8
6
8
10
12
-10
0
10
PC
app
roxi
mat
ion
4
-2
0
2
PC
app
roxi
mat
ion
-2
0
2
4
6P
C a
ppro
xim
atio
n
O l 1 t d O l 1 t d 2 d d 1 t t 3 d d
-8 -6 -4 -2 0 2 4 6 8-30
-20
Original data -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
Original data-8 -6 -4 -2 0 2 4 6 8
-8
-6
-4
Original data
( ) 13log =δ ( ) 64log =δ
Only 1st orders Only 1st and 2nd orders 1st to 3rd orders
( ) 23log =δ[1; 3; 3] [1; 1; 1; 8; 8] [1; 1; 1; 8; 8; 4; 4; 4]
( ) 1.3log2 −=δ ( ) 6.4log2 −=δ( ) 2.3log2 −=δ
Test example: Ishigami function
( ) ( ) ( ) ( ) ( )2 4sin 7sin 0 1 sinY f X X X X X X Xπ π π π= = + +
• Model:Inputs are uniformly distributed between -1 and 1
( ) ( ) ( ) ( ) ( )1 2 3 1 2 3 1, , sin 7sin 0.1 sinY f X X X X X X Xπ π π π= = + +
• Correlation matrix:
1 00 1 0
ρ⎛ ⎞⎜ ⎟Σ = ⎜ ⎟⎜ ⎟0 1ρ⎜ ⎟⎜ ⎟⎝ ⎠
• Error estimate :
[ ]21( ) ( ) ( )f h f x h x dxδ = ∫[ ]( , ) ( ) ( )f h f x h x dxD
δ = −∫
Numerical Results (Ishigami)First method.First method.
•N= 1024
0
• Optimal Polynomial Orders= [5; 8; 3; 5; 5]
• Optimal Polynomial Orders= [3; 8; 0; 5; 5]
0ρ = 0.9ρ =
2log ( ) 6.53δ = −2log ( ) 6.92δ = −
Ishigami functionSecond method.Second method.
9.0=ρNumber of Qausi MC sampling points N = 1024
Optimal Polynomial Orders=
[3; 6; 2; 4; 4; 2; 2; 2] 25
30
( ) 15.2log2 −=δ[3; 6; 2; 4; 4; 2; 2; 2]
15
20
tion
0
5
10
PC
app
roxi
mat
-10
-5
0
-10 -5 0 5 10 15 20-15
Original data
Ishigami functionFirst method. Y=f(x1*, x2, x3*)First method. Y f(x1 , x2, x3 )
N=1024
• Orders = [3; 3; 3; 5; 5]
• Orders = [5; 5; 5; 5; 5]
• Orders = [7; 7; 7; 5; 5] Orders = [7; 7; 7; 5; 5]
• Optimal Orders = [5; 8; 3; 5; 5]
0,0.9ρ =
Numerical Results (Ishigami)
0ρ = 0.9ρ =
Conclusions
A new method for estimation variance based sensitivity indices for models with dependent variablesmodels with dependent variables.
A Gaussian copula based approach allows to construct arbitrary multivariate probability distributions.
Two methods for metamodelling of models with dependent inputsTwo methods for metamodelling of models with dependent inputs based on polynomial chaos expansion :
1) Th fi t th d th t th d d b t1) The first method assumes that the dependence betweenthe variables is given by Gaussian copula in which case it is more efficient than the second method
2) The second method can be used when the dependence betweenthe variables is given by any copulathe variables is given by any copula