reliable and objective assessment of the forecast quality ... · 1 reliable and objective...
TRANSCRIPT
![Page 1: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/1.jpg)
1
Reliable and objective assessment
of the forecast quality
of meta-models
Thomas Most
Dynardo GmbH
13. Weimarer Optimierungs und
Stochastiktage 23.-24. Juni 2016
![Page 2: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/2.jpg)
2T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
Automatic workflow
with a minimum of solver runs to:
• identify the important parameters for each response
• Generate best possible metamodel (MOP) for each response
• understand and reduce the optimization task
• check solver and extraction noise
Sensitivity AnalysisUnderstand the most important input variables!
© Dynardo GmbH
![Page 3: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/3.jpg)
3T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
Latin Hypercube Sampling
![Page 4: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/4.jpg)
4T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
Deterministic DoE
• Complex scheme required to detect multivariate dependencies
• Exponential growth with dimension
• Full factorial:
• Koshal linear:
Advanced Latin Hypercube Sampling
• Reduced sample size for statistical estimates
compared to plain Monte Carlo
• Reduces unwanted input correlation
How to Generate a Design of Experiments
© Dynardo GmbH
![Page 5: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/5.jpg)
5T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Reduction of correlation errors by simplified Iman &
Connover algorithm (applicable for nSamples > nPar)
• Start designs are not considered
Advanced Latin Hypercube Sampling (oSL 4.x)
© Dynardo GmbH
• 10 variables• 50 samples• Correlation errors
up to 0.04
![Page 6: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/6.jpg)
6T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Improved Iman & Connover algorithm + optimization
(applicable even for nSamples < nPar)
• Start designs are considered
Advanced Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
• 10 variables• 50 samples• Correlation errors
less than 0.001
![Page 7: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/7.jpg)
7T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Discrete parameter states are now
considered in the minimization of
the correlation errors
Advanced Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
ALHS for discrete parameters
![Page 8: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/8.jpg)
8T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Spatial uniform distribution of LHS optimized by discrepancy criterion
Space-filling Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
![Page 9: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/9.jpg)
9T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Spatial uniform distribution of LHS optimized by discrepancy criterion
• Start design are considered
Space-filling Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
![Page 10: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/10.jpg)
10T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Optimal space-filling distribution in full space
• Reduced linear and quadratic correlations (efficient up to 10 variables)
Space-filling Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
![Page 11: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/11.jpg)
11T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Attention: reduced subspace is not space-filling anymore!
Recommended application up to 10 variables or
for adding samples in subspace
2 of 2 parameters 2 of 5 parameters
Space-filling Latin Hypercube Sampling (oSL 5.2)
© Dynardo GmbH
![Page 12: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/12.jpg)
12T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
Coefficient of Prognosis
![Page 13: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/13.jpg)
13T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Approximation of response variables as
explicit function of all input variables
• Approximation function can be used for
sensitivity analysis and/or optimization
• Global methods (Polynomial
regression, Neural Networks, …)
• Local methods (Spline interpolation,
Moving Least Squares, Radial Basis
Functions, Kriging, …)
• Approximation quality decreases with
increasing input dimension
• Successful application requires
objective measures of the
prognosis quality
© Dynardo GmbH
Response Surface Method
![Page 14: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/14.jpg)
14T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• CoD (Coefficient of Determination or R²)
• CoP (Coefficient of Prognosis)
with
• Total variation:
• Unexplained variation:
CoP versus CoD
© Dynardo GmbH
![Page 15: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/15.jpg)
15T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Coefficient of Determination quantifies merely the Goodness of Fit.
• Interpolation models (e.g. MLS, Kriging) can reach CoD of 1.00
• But perfect fit does not mean perfect forecast quality!
Measure Goodness of Fit = Coefficient of Determination (CoD)
CoD = 1.00
![Page 16: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/16.jpg)
16T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Split the data in two sets (optimal spatial distribution)
• Build up the meta-model for data set 1 = regression point set
• Evaluate forecast quality on data set 2 = independent test point set
Measure forecast quality = Coefficient of Prognosis (CoP)
CoD_1 = 1.00CoD_1 = 1.00CoP_2 = 0.80
![Page 17: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/17.jpg)
17T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Build up the meta-model for data set 2 = regression point set
• Evaluate forecast quality on data set 1 = independent test point set
CoD_2 = 1.00CoP_1 = 0.59
Measure forecast quality = Coefficient of Prognosis (CoP)
![Page 18: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/18.jpg)
18T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Coefficient of Prognosis sums up the
errors from both cross validation sets:
• CoP is an objective measure of forecast quality!
CoD = 1.00CoP = 0.74
Measure forecast quality = Coefficient of Prognosis (CoP)
![Page 19: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/19.jpg)
19T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• CoD (R²) tells how good the
regression model fits through the
sample points, no real prediction
quality
• CoD is too optimistic for a small
number of support points
• CoD=1 for interpolation models
(e.g. Kriging, …)
CoP versus CoD
• CoP measures the true forecast quality of a regression model using an
independent set of test data (cross validation)
• Suitable for any kind of meta-model
• Remark: Leave-one-out may be too optimistic as well!
Labeled as Predictive Coefficient of Determination in oSL post-processing
© Dynardo GmbH
![Page 20: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/20.jpg)
20T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
Metamodel of Optimal Prognosis (MOP)
• Objective measure of prognosis quality
• Determination of relevant parameter subspace
• Determination of optimal approximation model
• Approximation of solver output by fast
surrogate model without over-fitting
• Evaluation of variable sensitivities
© Dynardo GmbH
![Page 21: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/21.jpg)
21T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• MOP approach yields the best model regarding forecast quality (CoP)
• Smoothing of noisy responses as added benefit
• A difference >3% between CoD and CoP indicates that the model
forecast quality can be further increased by adding more support points.
Metamodel of Optimal Prognosis (MOP)
CoD = 0.87CoP = 0.77
![Page 22: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/22.jpg)
22T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
Example
![Page 23: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/23.jpg)
23T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
Analytical Nonlinear Function
• 20 input variables
• Additive linear and nonlinear terms, one coupling term and one noise term with normal distribution
© Dynardo GmbH
![Page 24: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/24.jpg)
24T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Anisotropic Kriging in automatically detected four-dimensional subspace
• CoD (100 support points): 98,9% (noise is smoothed)
• CoP (cross validation): 97,0%, predicted RMSE 0,81
• Verification data set (500 samples): 97,8%, verified RMSE 0,70
Metamodel of Optimal Prognosis
Prediction (CoP) Verification
![Page 25: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/25.jpg)
25T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
© Dynardo GmbH
• Genetic Aggregation in full space (weighted sum of polynomials,
Kriging, support vector regression and neural networks)
• CoD (100 support points): 100% (noise is interpolated!)
• Predicted CoD: 92,9%, predicted RMSE 1,27
• Verification data set (500 samples): 83,4%, verified RMSE 1,94
“Competitor”
![Page 26: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/26.jpg)
26T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Smooth approximation function
of MOP enables reliable
optimization on the response
surface
• Interpolating the noise makes
the optimization task much
more difficult and yields to
nonsufficient results
© Dynardo GmbH
Optimization on the Meta-Model
Reference MOP Competitor
Response -12,41 +/- noise -10,94 -9,46
X1 3,14 3,14 1,59
X2 -3,14 -3,14 -3,14
X3 -1,57 -1,56 -1,39
X4 -0,50 -0,36 -0,38
![Page 27: Reliable and objective assessment of the forecast quality ... · 1 Reliable and objective assessment of the forecast quality of meta-models Thomas Most Dynardo GmbH 13. Weimarer Optimierungs](https://reader030.vdocuments.us/reader030/viewer/2022041215/5e03d8a0319b180f6a497aa2/html5/thumbnails/27.jpg)
27T. Most: Reliable assessment of the forecast quality of meta-models
13. Weimarer Optimierungs und Stochastiktage 23.-24. Juni 2016
• Global refinement is now
possible with Advanced
(correlation optimized) and
Space-filling Latin Hypercube
Sampling
• Coefficient of Prognosis is a
reliable measure of forecast
quality
• Making meta-models more and more flexible tends to over-fitting with
more flexible results in
low forecast quality!
Further developments: Local refinement of MOP
• Local approximation quality
• Single-objective criteria
• Multi-objective criteria
© Dynardo GmbH
Summary