![Page 1: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/1.jpg)
Surrogate minimisation in high dimensions
Fabrizia Guglielmetti (ESO)
In collaboration with: Torsten Enßlin (MPA), Henrik Junklewitz (AIfA), Theo Steininger (MPA)
Bayes Forum Seminar - 24.02.2017 (1)
![Page 2: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/2.jpg)
Outline
❖ Motivation!• Performance enhancement of optimisation schemes !
❖ Global optimisation as a Bayesian decision problem!• Two connected statistical layers!
❖ Surrogates basic principles !
❖ Application of Kriging Surrogate within NIFTY (Selig, M. et al., 2013, A&A, 554, 26) framework
(2)Bayes Forum Seminar - 24.02.2017
![Page 3: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/3.jpg)
Introduction
(3)Bayes Forum Seminar - 24.02.2017
!
❖ Complex computer codes are essentials!
!
!
!
!
!
!
![Page 4: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/4.jpg)
(3)Bayes Forum Seminar - 24.02.2017
A 48-hour computer simulation of Typhoon Mawar using the Weather Research and Forecasting model! credit Wikipedia
Introduction
![Page 5: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/5.jpg)
(3)Bayes Forum Seminar - 24.02.2017
See, e.g., Cox D.D, Park J.S., Singer C.E. (2001) Preuss, R. & von Toussaint, U. (2016)
Introduction
![Page 6: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/6.jpg)
(3)Bayes Forum Seminar - 24.02.2017
Introduction
![Page 7: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/7.jpg)
(3)Bayes Forum Seminar - 24.02.2017
Introduction
![Page 8: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/8.jpg)
(3)Bayes Forum Seminar - 24.02.2017
Introduction
![Page 9: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/9.jpg)
(3)Bayes Forum Seminar - 24.02.2017
Introduction
![Page 10: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/10.jpg)
(4)Bayes Forum Seminar - 24.02.2017
!
❖ Complex computer codes are essentials!
!
!
!
!
!
!
Computer Codex z(x)
Simulator
Introduction
![Page 11: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/11.jpg)
(4)Bayes Forum Seminar - 24.02.2017
!
❖ Complex computer codes are essentials:!
!
!
!
!
!
!
Computer Codex z(x)
!
Minimisation of Complex Objective function. For reasonable data size, current estimation techniques:!
a. SLOW if full objective function is accounted!
b. FAST if do not account for full objective function!
Simulator
Introduction
![Page 12: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/12.jpg)
(4)Bayes Forum Seminar - 24.02.2017
!
❖ Complex computer codes are essentials:!
!
!
!
!
!
!
Computer Codex z(x)
!
• statistical representation of x !
• expresses knowledge about z(x) at any given x!
• built using prior information and training set of model runs
Emulator
Introduction
![Page 13: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/13.jpg)
(4)Bayes Forum Seminar - 24.02.2017
!
❖ Complex computer codes are essentials:!
!
!
!
!
!
!
Computer Codex z(x)
Kriging surrogate sampler is used to emulate the original Complex Objective function
Emulator
Introduction
![Page 14: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/14.jpg)
Surrogates (or metamodels, emulators, …)
(5)Bayes Forum Seminar - 24.02.2017
![Page 15: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/15.jpg)
• A map is a surrogate that predicts terrain elevation
• Prediction of the map allows one to locate the summit without climbing it
• Searching for the highest peak is a form of global optimisation
Surrogates (or metamodels, emulators, …)
(5)Bayes Forum Seminar - 24.02.2017
![Page 16: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/16.jpg)
… in the same fashion we can use metamodels to find the optimal signal configuration
Surrogates (or metamodels, emulators, …)
(5)Bayes Forum Seminar - 24.02.2017
![Page 17: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/17.jpg)
Bayesian inference, parametric model❖ Data: x, y!
❖ Model: !
!
Gaussian likelihood:!
!
Prior over the param:!
!
Posterior param distr: !
!
Make predictions: !
Bayes Forum Seminar - 24.02.2017
y = fk(x) + ✏
p(y|x,k,Mi) /Y
j
exp
�� 1
2
(yj � fk(xj))2/�2
�
p(k|Mi)
p(k|x,y,Mi) =p(y|x,k,Mi)p(k|Mi)
p(y|x,Mi)
p(y⇤|x⇤,x,y,Mi) =
Zp(y⇤|k, x⇤
,Mi)p(k|x,y,Mi)dk
(6)
![Page 18: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/18.jpg)
Bayesian inference, parametric model❖ Data: x, y!
❖ Model: !
!
Gaussian likelihood:!
!
Prior over the param:!
!
Posterior param distr: !
!
Make predictions: !
Bayes Forum Seminar - 24.02.2017
y = fk(x) + ✏
p(y|x,k,Mi) /Y
j
exp
�� 1
2
(yj � fk(xj))2/�2
�
p(k|Mi)
p(k|x,y,Mi) =p(y|x,k,Mi)p(k|Mi)
p(y|x,Mi)
p(y⇤|x⇤,x,y,Mi) =
Zp(y⇤|k, x⇤
,Mi)p(k|x,y,Mi)dk
(6)
![Page 19: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/19.jpg)
Bayesian inference, parametric model❖ Data: x, y!
❖ Model: !
!
Gaussian likelihood:!
!
Prior over the param:!
!
Posterior param distr: !
!
Make predictions: !
Bayes Forum Seminar - 24.02.2017
y = fk(x) + ✏
p(y|x,k,Mi) /Y
j
exp
�� 1
2
(yj � fk(xj))2/�2
�
p(k|Mi)
p(k|x,y,Mi) =p(y|x,k,Mi)p(k|Mi)
p(y|x,Mi)
p(y⇤|x⇤,x,y,Mi) =
Zp(y⇤|k, x⇤
,Mi)p(k|x,y,Mi)dk
(6)
![Page 20: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/20.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
(7)Bayes Forum Seminar - 24.02.2017
![Page 21: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/21.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
Information Hamiltonian
Bayes Forum Seminar - 24.02.2017 (7)
![Page 22: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/22.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
Information Hamiltonian
Partition function
Bayes Forum Seminar - 24.02.2017 (7)
![Page 23: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/23.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
d = (d1, d2, . . . , dn)T n 2 N
IFT exploits known or inferred correlation structures of the field of interest over some domain to regularise the ill-posed inverse problem of determining an number of dof from a finite dataset
finite dataset
s = s(x) ⌦ = {x}1
Bayes Forum Seminar - 24.02.2017 (7)
![Page 24: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/24.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
most probable signal configuration
Bayes Forum Seminar - 24.02.2017 (7)
![Page 25: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/25.jpg)
IFT for signal inference❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields!
❖ Signal field (s) estimation: !
!
!
!
!
!
P (s|d) = P (d|s)P (s)
P (d)⌘ e�H(d,s)
Zd
hsi = argminhs|diH(d, s)
Bayes Forum Seminar - 24.02.2017
most probable signal configuration
(7)
![Page 26: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/26.jpg)
Minimization of (complex) energy function
Expensive computer code
(8)Bayes Forum Seminar - 24.02.2017
![Page 27: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/27.jpg)
Minimization of (complex) energy function
Expensive computer code
Optimizer
Bayes Forum Seminar - 24.02.2017 (8)
![Page 28: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/28.jpg)
Minimization of (complex) energy function
Expensive computer code
Optimizer
Find optimal image reconstruction
Bayes Forum Seminar - 24.02.2017 (8)
![Page 29: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/29.jpg)
Minimization of (complex) energy function
Expensive computer code
Response Surface
Model (RSM)
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 30: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/30.jpg)
Minimization of (complex) energy function
Expensive computer code
Response Surface
Model (RSM)
Used to increase the speed
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 31: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/31.jpg)
Minimization of (complex) energy function
Expensive computer code
Used to increase the speed
Cheap-to-run surrogate model of the original model function
is used and combined with the full objective function!
!(same model is used throughout the
optimisation process)
Response Surface
Model (RSM)
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 32: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/32.jpg)
Minimization of (complex) energy function
Expensive computer code
Kriging (Wiener, 1938; Krige, 1951; Matheron, 1962; Sacks et al, 1989;
Cressie, 1989, 1993; Jones et al, 1989; Kleijnen, 2009; Razavi et al; 2012)
Response Surface
Model (RSM)
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 33: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/33.jpg)
Minimization of (complex) energy function
Expensive computer code
Kriging (Kitanidis, 1986; Handcock & Stein, 1993; Mira & Sanchez,2002; More & Halvorsen, 1989; Morris, 1993; Kennedy &
O’Hagan, 2000; Bayarri & Berger, 2007; Preuss et al. 2016)
Response Surface
Model (RSM)
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 34: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/34.jpg)
Minimization of (complex) energy function
Expensive computer code
Response Surface
Model (RSM)
Find optimal image reconstruction
Optimizer/ Adjoint simulation
Partially converged simulations
Bayes Forum Seminar - 24.02.2017 (8)
![Page 35: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/35.jpg)
Minimization of (complex) energy function
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Bayes Forum Seminar - 24.02.2017 (8)
![Page 36: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/36.jpg)
Minimization of (complex) energy function
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Gradients take correct direction early in
convergence
Bayes Forum Seminar - 24.02.2017 (8)
![Page 37: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/37.jpg)
Minimization of (complex) energy function
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Proper shape of RSM is provided
Bayes Forum Seminar - 24.02.2017 (8)
![Page 38: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/38.jpg)
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Max shape control in image variations
afforded to the optimiser for a
minimum number of variables
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Minimization of (complex) energy function
Bayes Forum Seminar - 24.02.2017
![Page 39: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/39.jpg)
Minimization of (complex) energy function
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Divert optimization away from infeasible
regions
Max shape control in image variations
afforded to the optimiser for a
minimum number of variables
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Bayes Forum Seminar - 24.02.2017
![Page 40: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/40.jpg)
Minimization of (complex) energy function
Find optimal image reconstruction
Expensive computer code
Optimizer/ Adjoint simulation
Response Surface
Model (RSM)
Divert optimization away from infeasible
regions
Max shape control in image variations
afforded to the optimiser for a
minimum number of variables
Partially converged simulations
Position of individual surface points are moved in the direction of the gradients predicted by the
adjoint simulation
Bayes Forum Seminar - 24.02.2017
![Page 41: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/41.jpg)
Surrogate recipe
Target function
z=f(x)
The simulator
(9)Bayes Forum Seminar - 24.02.2017
![Page 42: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/42.jpg)
Surrogate recipe
Target function
Quantity of interest
z=f(x)
Bayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 43: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/43.jpg)
Surrogate recipe
Target function
Input x
Quantity of interest
z=f(x)
Bayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 44: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/44.jpg)
Surrogate recipe
Target function
Input x
Output z
Quantity of interest
z=f(x)
Bayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 45: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/45.jpg)
Surrogate recipe
Target function
Input x
Output z
Quantity of interest
z=f(x)
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 46: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/46.jpg)
Surrogate recipe
Target function
Input x
Output z
Quantity of interest
z=f(x)
Response
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 47: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/47.jpg)
Surrogate recipe
Target function
Input x
Output z
Quantity of interest
z=f(x)
Computationally intensive
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The simulator
(9)
![Page 48: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/48.jpg)
Surrogate recipe
Target function
x,z multi-dimensional
z=f(x)
The aim is to learn about f(x) …
Input x
Output z
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 49: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/49.jpg)
Surrogate recipe
Target function
x,z multi-dimensional
z=f(x)
… assume we can effort only n evaluations of f(x)
Input x
Output z
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 50: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/50.jpg)
Surrogate recipe
Target function
Input x
Output z
x,z multi-dimensional
z=f(x)
Bayes Forum Seminar - 24.02.2017
… assume we can effort only n evaluations of f(x)
The emulator(9)
![Page 51: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/51.jpg)
Surrogate recipe
Target function
Input x
Output z
x,z multi-dimensional
z=f(x)
Build a sampling plan with a set of experimental inputs:X = {x(1), . . . ,x(n�1)}T
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 52: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/52.jpg)
Surrogate recipe
Target function
Input x
Output zf(x) field samples
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The emulator
Build a sampling plan with a set of experimental inputs:X = {x(1), . . . ,x(n�1)}T
(9)
![Page 53: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/53.jpg)
Target function
Input x
Output z
Use measurement apparatus (numerical solver) to get the observations: i.e. calculate the responses y = {y(1), . . . ,y(n�1)}T
y
Surrogate recipe
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 54: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/54.jpg)
Surrogate recipe
Target function
Input x
Output z
y
f̂(x)
Fit a surrogate model to the data: from the observations (x,y) we make a prediction of the response
f̂
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 55: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/55.jpg)
Surrogate recipe
Target function
Input x
Output z
y
f̂(x)
Assume stands in for we can find as close as to the true minimum of
f̂ f x
0
f̂(x0)
x,z multi-dimensionalBayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 56: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/56.jpg)
Surrogate recipe enhanced
Target function
Input x
Output z
x,z multi-dimensional
y
f̂(x)
Improve inference by including observations of the GRADIENT
reduce computational cost of higher-dimensional optimisation problems
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 57: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/57.jpg)
Surrogate recipe enhanced
Weights are assigned to sample points according to! a data driven weighting function: sample points
closer to the new sample point to be predicted receive more weight
KRIGING WEIGHTING SCHEME
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 58: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/58.jpg)
Surrogate recipe
Target function
Input x
Output z
y
f̂(x)
Validate against the true expensive : computation of ff̂(x0) f(x0)
Bayes Forum Seminar - 24.02.2017
The emulator(9)
![Page 59: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/59.jpg)
!
1. Multimodal and multidimensional landscape!
2. Surrogate function has to emulate well the real function, at least in the location of the optima!
Surrogate challenges
(10)Bayes Forum Seminar - 24.02.2017
![Page 60: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/60.jpg)
Kriging surrogate - Gaussian processEmpirical modelling in high dimensional data:!
• y(x) assumed to underlie the data {x(n)
, tn} : adaption of model to data corresponds to inference of the function given the data!
Generalisation of a (multivariate) Gaussian distribution to a function space of infinite dimension!
Specified by mean and covariance functions!• mean is a function of x (the zero function)!• cov is a function C(x,x’), expected covariance between the values of the function y at the
points x and x’ !
!!!y(x) lives in the infinite-dimensional space of all continuous functions of x
(11)Bayes Forum Seminar - 24.02.2017
P (y(x)|µ(x), A) = 1
Z
exp
⇥� 1
2
�y(x)� µ(x)
�TA
�y(x)� µ(x)
�⇤
y(x) ⇠ GP
�µ(x), C(x, x0)
�
![Page 61: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/61.jpg)
Kriging surrogate
❖ Gaussian process regression, Wiener-Kolmogorov prediction !❖ Kriging is employed within Bayesian formalism!❖ Assumptions: !
• prior distribution and covariance of unknown function f(x)!• observed data (x) are normally distributed !
-> responses y(x) Gaussian process !
!
P (y(x)|ti,Xi, I) =P (ti|y(x),Xi, I)P (y(x))
P (ti)
(12)Bayes Forum Seminar - 24.02.2017
![Page 62: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/62.jpg)
Kriging surrogate
❖ Gaussian process regression, Wiener-Kolmogorov prediction!❖ Kriging is employed within Bayesian formalism!❖ Assumptions: !
• prior distribution and covariance of unknown function f(x)!• observed data (x) are normally distributed !
-> responses y(x) Gaussian process !
!
P (y(x)|ti,Xi, I) =P (ti|y(x),Xi, I)P (y(x))
P (ti)
Posterior pdf of the emulated signal
Bayes Forum Seminar - 24.02.2017 (12)
![Page 63: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/63.jpg)
Bayes Forum Seminar - 24.02.2017
Kriging surrogate
❖ Gaussian process regression, Wiener-Kolmogorov prediction!❖ Kriging is employed within Bayesian formalism!❖ Assumptions: !
• prior distribution and covariance of unknown function f(x)!• observed data (x) are normally distributed !
-> responses y(x) Gaussian process !
!
P (y(x)|ti,Xi, I) =P (ti|y(x),Xi, I)P (y(x))
P (ti)tn = {ti}ni=1
(12)
![Page 64: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/64.jpg)
Bayes Forum Seminar - 24.02.2017
Kriging surrogate
❖ Gaussian process regression, Wiener-Kolmogorov prediction !❖ Kriging is employed within Bayesian formalism!❖ Assumptions: !
• prior distribution and covariance of unknown function f(x)!• observed data (x) are normally distributed !
-> responses y(x) Gaussian process !
!
P (y(x)|ti,Xi, I) =P (ti|y(x),Xi, I)P (y(x))
P (ti)tn = {ti}ni=1
target values (12)
![Page 65: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/65.jpg)
Kriging surrogate
❖ Gaussian process regression, Wiener-Kolmogorov prediction !❖ Kriging is employed within Bayesian formalism!❖ Assumptions: !
• prior distribution and covariance of unknown function f(x)!• observed data (x) are normally distributed !
-> responses y(x) Gaussian process !
!
P (y(x)|ti,Xi, I) =P (ti|y(x),Xi, I)P (y(x))
P (ti)⌘ e�H(y(x),ti)
Zti
Bayes Forum Seminar - 24.02.2017 (12)
![Page 66: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/66.jpg)
Minimization challenge
d = Rs+ n
(13)Bayes Forum Seminar - 24.02.2017
![Page 67: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/67.jpg)
Minimization challenge
G(s, S) = 1p|2⇡S|
exp(�1
2
s
†S
�1s)
d = Rs+ n
Bayes Forum Seminar - 24.02.2017 (13)
![Page 68: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/68.jpg)
Minimization challenge
G(s, S) = 1p|2⇡S|
exp(�1
2
s
†S
�1s)
H(d|s) = � lnP (d|s) = 1
2(d�Rs)†N�1(d�Rs)
d = Rs+ n
Bayes Forum Seminar - 24.02.2017 (13)
![Page 69: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/69.jpg)
Minimization challenge
H(d, s) = �lnP(d, s) b= 1
2s†S�1s + H(d|s)
H(d|s) = � lnP (d|s) = 1
2(d�Rs)†N�1(d�Rs)
d = Rs+ nG(s, S) = 1p
|2⇡S|exp(�1
2
s
†S
�1s)
Bayes Forum Seminar - 24.02.2017 (13)
![Page 70: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/70.jpg)
Minimization challenge
m = argmins
H(d, s)
H and dH/ds are calculated at several positions to slide towards the minimum -> computationally expensive
H(d, s) = �lnP(d, s) b= 1
2s†S�1s + H(d|s)
H(d|s) = � lnP (d|s) = 1
2(d�Rs)†N�1(d�Rs)
d = Rs+ nG(s, S) = 1p
|2⇡S|exp(�1
2
s
†S
�1s)
Bayes Forum Seminar - 24.02.2017 (13)
![Page 71: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/71.jpg)
Surrogate minimization
x = S
�1/2s
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ n
Bayes Forum Seminar - 24.02.2017 (13)
![Page 72: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/72.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
Bayes Forum Seminar - 24.02.2017 (13)
![Page 73: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/73.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
Bayes Forum Seminar - 24.02.2017 (13)
![Page 74: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/74.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
Bayes Forum Seminar - 24.02.2017 (13)
![Page 75: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/75.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
I
n
= (xi
, E
i
= E(xi
), vi
= �@
x
E(x)|xi)
n
i=1
Bayes Forum Seminar - 24.02.2017 (13)
![Page 76: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/76.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
I
n
= (xi
, E
i
= E(xi
), vi
= �@
x
E(x)|xi)
n
i=1
FIn
(x) =nX
i=1
wi(x)
8<
:Eie
�v
†i
(x�x
i
)
E
i if v†i (x� xi) > 0
Ei � v
†i (x� xi) if v†i (x� xi) 0
Bayes Forum Seminar - 24.02.2017 (13)
![Page 77: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/77.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
I
n
= (xi
, E
i
= E(xi
), vi
= �@
x
E(x)|xi)
n
i=1
FIn
(x) =nX
i=1
wi(x)
8<
:Eie
�v
†i
(x�x
i
)
E
i if v†i (x� xi) > 0
Ei � v
†i (x� xi) if v†i (x� xi) 0
Bayes Forum Seminar - 24.02.2017 (13)
![Page 78: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/78.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
I
n
= (xi
, E
i
= E(xi
), vi
= �@
x
E(x)|xi)
n
i=1
H(d|s) = 1
2(d�Rs)†N�1(d�Rs)
FIn
(x) =nX
i=1
wi(x)
8<
:Eie
�v
†i
(x�x
i
)
E
i if v†i (x� xi) > 0
Ei � v
†i (x� xi) if v†i (x� xi) 0
Bayes Forum Seminar - 24.02.2017 (13)
![Page 79: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/79.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
E(x) ⇡ FIn(x)
I
n
= (xi
, E
i
= E(xi
), vi
= �@
x
E(x)|xi)
n
i=1
FIn
(x) =nX
i=1
wi(x)
8<
:Eie
�v
†i
(x�x
i
)
E
i if v†i (x� xi) > 0
Ei � v
†i (x� xi) if v†i (x� xi) 0
wi(x) =(|x� xi|2 + ✏
2)�↵2
Pnj=1(|x� xj |2 + ✏
2)�↵2
(1)
Bayes Forum Seminar - 24.02.2017 (13)
![Page 80: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/80.jpg)
Surrogate minimization
H(x) = H(d, s = S1/2x)b=1
2x†x+H(d|s = S
12x)| {z }
E(x)
d = Rs+ nx = S
�1/2s
H(x)b=1
2x†x+ FIn(x) = HIn(x)
@x
HI
n
(x) = x�nX
i=1
wi
(x)
8<
:↵AE
i
e�v
†i
(x�x
i
)
E
i + vi
e� v
†i
(x�x
i
)
E
i if v†i
(x� xi
) > 0
↵A⇥E
i
� v†i
(x� xi
)⇤+ v
i
if v†i
(x� xi
) 0
A =⇥ x� x
i
(|x� xi
|2 + ✏2)↵�X
j
wj
(x)x� x
j
(|x� xj
|2 + ✏2)↵⇤
where
Bayes Forum Seminar - 24.02.2017 (13)
![Page 81: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/81.jpg)
Application
Simulated sky signal employing NIFTY package!!
1.6E+4 dimensions
NIFTY= Numerical Information Field Theory
(M.Selig, T. Enßlin et al.)
(14)Bayes Forum Seminar - 24.02.2017
![Page 82: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/82.jpg)
Application
Simulated dataset employing NIFTY package!!
d=R(s) + n
NIFTY= Numerical Information Field Theory
(M.Selig, T. Enßlin et al.)
Bayes Forum Seminar - 24.02.2017 (14)
![Page 83: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/83.jpg)
Application
Simulated dataset employing NIFTY package!!
d=R(s) + n
find optimal solution for s given d
NIFTY= Numerical Information Field Theory
(M.Selig, T. Enßlin et al.)
Bayes Forum Seminar - 24.02.2017 (14)
![Page 84: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/84.jpg)
Application
Simulated dataset employing NIFTY package!!
d=R(s) + n
find optimal solution for s given d
assumption: s is random field !following some statistics and !being constrained by the data
NIFTY= Numerical Information Field Theory
(M.Selig, T. Enßlin et al.)
Bayes Forum Seminar - 24.02.2017 (14)
![Page 85: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/85.jpg)
Application -Results
Bayes Forum Seminar - 24.02.2017 (14)
![Page 86: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/86.jpg)
Application -Results
Bayes Forum Seminar - 24.02.2017 (14)
![Page 87: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/87.jpg)
Application -Results
Surrogate solution!coincide with !
Wiener filter solution
(15)Bayes Forum Seminar - 24.02.2017
![Page 88: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/88.jpg)
Application -Results
(15)Bayes Forum Seminar - 24.02.2017
![Page 89: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/89.jpg)
Application -Results
(15)Bayes Forum Seminar - 24.02.2017
![Page 90: Surrogate minimisation in high dimensionsaws/BFSeminar24022017Fabrizia.pdf · Outline Motivation! • Performance enhancement of optimisation schemes ! Global optimisation as a Bayesian](https://reader035.vdocuments.us/reader035/viewer/2022081600/60539fd5f0cce735565d16fe/html5/thumbnails/90.jpg)
Speed up of computer run of minimisation of Complex Energy function is desired !
A Kriging Surrogate sampler is developed to emulate the behaviour of the Complex Energy Function!• powerful way to perform Bayesian inference about functions in high-
dimensional space!• non-intrusive approach, but still an approximation!• reduces number of function evaluations!• includes sensitivity analysis !
Application in high dimensions of Kriging Surrogate within NIFTY framework is shown !• can speed up to a factor of ~100 other optimisation schemes
Conclusion & Summary
(16)Bayes Forum Seminar - 24.02.2017