calibration of computer simulators using emulators
TRANSCRIPT
![Page 1: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/1.jpg)
Calibration of Computer Simulators using Emulators
![Page 2: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/2.jpg)
Recap –Emulators
• We are concerned with complex, non-linear simulators
• In this session we will look at calibration of such simulators
• We will heavily depend on emulators • An emulator is a Gaussian process (or second order
process) that interpolates the simulator output• Emulators are fast
EGU short course - session 4 2
![Page 3: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/3.jpg)
Calibration
• Simulator users often want to tune the simulator using observations of the real system
• Adjust the input parameters so that the simulator output matches observations as well as possible
• Two very important points1. Calibration will reduce uncertainty about x but will not
eliminate it2. It is necessary to understand how the simulator relates to
reality• Model discrepancy
EGU short course - session 4 3
![Page 4: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/4.jpg)
Calibration and Assimilation
• Calibration is concerned with the values of the inputs that are consistent with the data.
• Assimilation is concerned with producing the best forecast/hindcast
• Calibration changes the simulator inputs• Assimilation changes the simulator state variables
EGU short course - session 4 4
![Page 5: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/5.jpg)
Model discrepancy
• Simulator output y = f(x) will not equal the real system value z
• Even with best/correct inputs x• Model discrepancy is the difference z – f(x)• As discussed in Session 1, model discrepancy is due to
• Wrong or incomplete science• Programming errors, rounding errors• Inaccuracy in numerically solving systems of equations
• Ignoring model discrepancy leads to poor calibration• Over-fitting of parameter estimates• Over-confidence in the fitted values
EGU short course - session 4 5
![Page 6: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/6.jpg)
History matching
• History matching (a term taken from the petroleum industry) means finding sets of inputs that given simulator outputs that are ‘compatible’ with data
• Calibration means finding a best value (or a distribution) for the inputs given the data
EGU short course - session 4 6
![Page 7: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/7.jpg)
Implausibility
• Define a measure of implausibility (Imp)
• If the implausibility is greater then ±3 those values of the inputs are deemed implausible
• Because this is a function of the emulator not the original simulator runs we calculate it everywhere in input space
EGU short course - session 4 7
![Page 8: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/8.jpg)
Waves of Implausibility
• Wave 1: Apply the implausibility measure. Mark part of input space as implausible
• Wave 2: Add extra points in the not implausible region and rebuild the emulator. Repeat the implausibility measure
• Wave 3+: Repeat until the implausible region ceases to grow
EGU short course - session 4
8
![Page 9: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/9.jpg)
A 1-d example
EGU short course - session 4 9
![Page 10: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/10.jpg)
EGU short course - session 4 10
![Page 11: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/11.jpg)
EGU short course - session 4 11
![Page 12: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/12.jpg)
EGU short course - session 4 12
![Page 13: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/13.jpg)
EGU short course - session 4 13
![Page 14: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/14.jpg)
EGU short course - session 4 14
![Page 15: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/15.jpg)
EGU short course - session 4 15
![Page 16: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/16.jpg)
Example -Galform
EGU short course - session 4 16
![Page 17: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/17.jpg)
Example - Galform• Galform is a simulator of Galaxy formation• It has 17 inputs• The amount of not implausible space in each wave is
• None of the original 1000 member LHC was an acceptable fit to the data
EGU short course - session 4 17
Wave 1 14.9%
Wave 2 5.9%
Wave 3 1.6%
Wave 4 0.26%
Wave 5 0.036%
![Page 18: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/18.jpg)
Calibration
• In history matching we were simply looking for regions of input space that were not implausible given the data.
• In calibration we want to find the ‘best input’ x (and it associated uncertainty)
EGU short course - session 4 18
![Page 19: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/19.jpg)
Kennedy and O’Hagan(2001)• ζ is the real system• z=ζ+ε is data on the real system (ε~N(0,σ2))• y=f(x) is the simulator output• d=ζ-y is the model discrepancy
• ζ=f(x)+d• Build an emulator for f and simultaneously model
the discrepancy as a GP• ζ=f*(x)+d*
• z=f*(x)+d*+εEGU short course - session 4 19
![Page 20: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/20.jpg)
Kennedy and O’Hagan (2001) -2
• We can now perform an uncertainty analysis• This shows how much we have learned about the
simulator inputs from the data• The mean/mode of the posteriors give us our
estimate of the best value for the inputs
EGU short course - session 4 20
![Page 21: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/21.jpg)
Model Discrepancy Revisited
• We have seen that we can use the model discrepancy to calibrate/history match the simulator
• We can also look at the discrepancy between different simulators
• This is particularly interesting if we have hierarchies of simulators
EGU short course - session 4 21
![Page 22: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/22.jpg)
Hierarchies of Simulators
• Often we have hierarchies of simulators• Usually the resolution is increasing but additional
processes could be added
EGU short course - session 4 22
![Page 23: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/23.jpg)
Hierarchies of Simulators
• Rather than emulate each simulator separately• Emulate simulator 1 and then emulate the difference
between each level• Need to have some runs at common inputs• Need few runs of expensive complex simulators
EGU short course - session 4 23
![Page 24: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/24.jpg)
Reified Simulators
EGU short course - session 4 24
![Page 25: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/25.jpg)
Reified Simulators
EGU short course - session 4 25
![Page 26: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/26.jpg)
Reified Simulators
EGU short course - session 4 26
![Page 27: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/27.jpg)
Reified Simulators
• Reified simulators are ‘imaginary’ simulators that we impose between our simulators and reality
• They are the ‘best’ simulator we could produce• Model discrepancy is split into two:
1. The discrepancy between the current simulator and the reified simulator
2. The discrepancy between the reified simulator and reality
• Reification does not reduce the discrepancy. It might make it easier to elicit.
EGU short course - session 4 27
![Page 28: Calibration of Computer Simulators using Emulators](https://reader036.vdocuments.us/reader036/viewer/2022062423/56649e4f5503460f94b46be7/html5/thumbnails/28.jpg)
Overview
• Emulators are useful tools in the calibration of complex simulators
• Two methods have been described:• History Matching – ruling out implausible regions of input
space• Calibration – Finding ‘best fit’ input values
• Reification may be useful in eliciting the relationship between simulators and reality
EGU short course - session 4 28