multi-model ensembling for seasonal-to-interannual prediction: from simple to complex
DESCRIPTION
Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex. Lisa Goddard and Simon Mason International Research Institute for Climate & Society The Earth Institute of Columbia University. Benefit of Using Multiple Models RPSS for 2m Temperature (JFM 1950-1995). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/1.jpg)
Multi-Model Ensembling for Seasonal-to-Interannual Prediction:
From Simple to Complex
Lisa Goddard and Simon MasonInternational Research Institute for Climate & Society
The Earth Institute of Columbia University
![Page 2: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/2.jpg)
Benefit of Combining ModelsBenefit of Using Multiple Models
RPSS for 2m Temperature (JFM 1950-1995)
Combining models reduces deficiencies of individual models
![Page 3: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/3.jpg)
Refining:
(1) RAW MODEL PROBABILITIES (simple) Tercile thresholds determined by model history -- Counting
(2) RECALIBRATED PDF PROBABILITIES (less simple)- Contingency table recalibration (CT): categorical probabilitiesdetermined by category of ensemble mean- Uncertainty in forecast PDFs based on ensemble mean MSE
Varying Complexity in Building MM
![Page 4: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/4.jpg)
Combining: (1) POOLED MM ENSEMBLE (simple) Each model weighted equally
(2) PERFORMANCE-BASED MM ENSEMBLE (less simple)- Bayesian: determine optimal weights for AGCMs & climatology
by maximizing likelihood
- Multiple linear regression (MLR): obtain probabilities from prediction error variance using first few moments of ensemble distributions
- Canonical Variate (CV): maximize discrimination between categoriesusing first few moments of ensemble distributions
Varying Complexity in Building MM
![Page 5: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/5.jpg)
Bayesian Model Combination
Climo Fcst“Prior”
GCM Fcst“Evidence”
t=1 2 3 4…
Combine “prior” and “evidence” to produce weighted “posterior” forecast probabilities, by maximizing the likelihood.
…
…
Combining: Based on model performance
![Page 6: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/6.jpg)
The canonical variates are defined to maximize the ratio of the between-category to within-categoryvariance.
Canonical Variate Analysis
Combining: Based on obs. relative to model
![Page 7: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/7.jpg)
NINO3.4
![Page 8: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/8.jpg)
DEMETER DataRetrospective seasonal climate forecasts generated using fully-coupled ocean-atmosphere models are available from the DEMETER project. Models are initialized using ERA40 atmospheric conditions and an ocean-assimilation. Initializations are available for February, May, August, and November, with lead-times of up to 5 months (e.g., for forecasts initialized in February, forecasts are available for February – July). For ECMWF, Meteo-France, and Met Office, forecasts for at least the 43-year period 1959-2001 are available, each with 9 ensemble members. The total number of ensemble forecasts used is 43×4×6=1032.
![Page 9: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/9.jpg)
Equal Weighting
Probabilistic forecasts were obtained by counting the number of ensemble members beyond the outer quartiles, and then averaging across the three models.
The pooled ensemble is thus an equally-weighted combination of predictions uncorrected for model skill (although corrected for model drift).
Reliability is good for all three categories.
![Page 10: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/10.jpg)
Canonical Variate Analysis
![Page 11: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/11.jpg)
Multiple Linear Regression
![Page 12: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/12.jpg)
Conclusions IVirtually no difference in performance between the various methods considered for objectively combining the DEMETER forecasts of NINO3.4 SSTs. None of the procedures improve noticeably upon simple averaging of the uncalibrated probabilities. However, the underlying skill level is very high. Sophisticated combination schemes may be more effective in situations of weaker skill (??). Attempting to combine all the models at once is clearly unsuccessful; the most effective strategy may be to recalibrate the models individually and then to calculate an equally-weighted average.
![Page 13: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/13.jpg)
Terrestrial Climate
![Page 14: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/14.jpg)
Data
AGCMs: Simulations for 1950-2000* CCM3 (NCAR) – 24 runs* ECHAM4.5 (MPI) – 24 runs* ECPC (Scripps) – 10 runs* GFDL AM2p12b – 10 runs* NCEP-MRF9 (NCEP/QDNR) – 10 runs* NSIPP1 (NASA-GSFC) – 9 runs
Observations:2m Air Temperature and Precipitationfrom CRU-UEA (v2.0)
![Page 15: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/15.jpg)
Effect of Probability TreatmentJFM 2m air temperature over land
![Page 16: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/16.jpg)
Effect of Probability Treatment
100 PDF RAW
RAW
REL RELSSREL
100 PDF RAW
RAW
RES RESSSRES
![Page 17: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/17.jpg)
Effect of Combination Method
![Page 18: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/18.jpg)
Effect of Combination MethodRAW Probabilities
100 PDF RAW
RAW
REL RELSSREL
100 PDF RAW
RAW
RES RESSSRES
![Page 19: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/19.jpg)
Effect of Combination MethodPDF Probabilities
100 PDF RAW
RAW
REL RELSSREL
100 PDF RAW
RAW
RES RESSSRES
![Page 20: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/20.jpg)
Conclusions II
• Reliability of N models pooled together, with uncalibrated PDFs, is better than any individual AGCM.
• Gaussian (PDF) recalibration gives some improvement, but Bayesian recalibration gives the greatest benefit.
• Reliability is typically gained at the expense of resolution.
![Page 21: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/21.jpg)
ISSUES
• Number of Models
• Length of training period
• When simple is complex enough?
![Page 22: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/22.jpg)
Effect of # of Models3 vs 6 AGCMS; 45-year training period
(Robertson et al., 2004, MWR)
Different approaches are more similar
with more models.
![Page 23: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/23.jpg)
RPSS for 2m TemperatureBayesian MM from Raw Probs. –
6 models, 45-yr training
Jan-Feb-Mar Jul-Aug-Sep
![Page 24: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/24.jpg)
RPSS for PrecipitationBayesian & Pooled MM from Raw Probs. –
6 models, 45-yr training
Jan-Feb-Mar Jul-Aug-Sep
![Page 25: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/25.jpg)
Reliability Diagrams* several methodsyield similar results overthe United States.
* MMs are remarkablyreliable over the US,even though the accuracyis not high.
![Page 26: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/26.jpg)
CONCLUSIONS III
•MM simulations over the US are remarkably reliable, even if their not terribly accurate.
•Simple pooling of the AGCMs, with uncalibrated probabilities, is equivalent to any of our techniques over the U.S.
• Doesn’t require long history, but largenumber of models (>5?) is desirable.
![Page 27: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/27.jpg)
GRAND CONCLUSIONS
• Overall, we find that recalibrating individual models gives better results than putting models together in complex combination alorithm.
• In comparing different recalibration/combination methods, we find that generally a gain in reliability is countered with a loss in resolution.
• More complicated approaches are not necessarily better. This needs to investigated for different forecast situations (i.e. variables, region, season).
![Page 28: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/28.jpg)
![Page 29: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/29.jpg)
Ranked Probability Skill ScoresTemperature
Jan-Feb-Mar (1950-1995)
![Page 30: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/30.jpg)
Ranked Probability Skill ScoresPrecipitation
Jul-Aug-Sep (1950-1999)
Comparing treatment of probabilityComparing treatment of probability- Even with 6 models, have regions- Even with 6 models, have regionsof large negative RPSSof large negative RPSS Suggests common model errors Suggests common model errors
- Recalibration reduces, but does notRecalibration reduces, but does noteliminate, large errorseliminate, large errors
- Some improvement of positive skillSome improvement of positive skillRecal-RawRecal-Raw
![Page 31: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/31.jpg)
Ranked Probability Skill ScoresPrecipitation
Jul-Aug-Sep (1950-1999)
Comparing combination methods- Performance-based combination eliminates large errors
- More improvement of positive skill
- More cases of negative skill turned topositive skill
![Page 32: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/32.jpg)
Canonical Variate Analysis
A number of statistical techniques involve calculating linear combinations (weighted sums) of variables. The weights are defined to achieve specific objectives:
• PCA – weighted sums maximize variance
• CCA – weighted sums maximize correlation
• CVA – weighted sums maximize discrimination
![Page 33: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/33.jpg)
Canonical Variate AnalysisLet X be a set of centred explanatory variables, with variance-covariance matrix Sxx. Let Y be a set of (non-centred) indicator variables that define the group membership of a set of observations, with cross-products matrix Syy. Solve the eigenproblem:
1 1 2xx xy yy xy r S S S S I a 0,
which is identical to the CCA problem, but with Y as the set of indicator variables rather than continuous values on dependent variables. The loadings, A, maximize the ratio of between-group variance to total variance, represented by the canonical correlation, r.
![Page 34: Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex](https://reader035.vdocuments.us/reader035/viewer/2022062814/568167b3550346895ddd008e/html5/thumbnails/34.jpg)
Effect of Probability Treatment
2 2
1
: ( , )1 ( )
1
i X i X
n
X i X ii
On