non-parametric postprocessing of ensemble forecasts for ... · non-parametric postprocessing of...
TRANSCRIPT
Non-parametric postprocessing of ensemble forecasts for extremeand rare events: a focus on daily rainfall using weighted scoring
rules for verification
Maxime Taillardat1,2,3
O. Mestre4,M. Zamo4,P. Naveau2 andA-L. Fougères3
1CNRM/Météo-France 2LSCE 3ICJ4Météo-France
April 25, 2016
The QRF technique Ensemble forecast verification Results Prospects
Ensemble forecast
source : P. Naveau
Maxime Taillardat 1/20
The QRF technique Ensemble forecast verification Results Prospects
Ensemble forecast
source : P. Naveau
Maxime Taillardat 2/20
The QRF technique Ensemble forecast verification Results Prospects
Motivations for statistical post-processing
◮ Ensembles are subject to model biases and underdispersion for
surface weather variables. (Hamill and Colucci 1997 ...)
◮ A simple bias correction is not sufficient.
◮ The skill added by post-processing is not reduced byimprovements in ensemble developments. (Hemri et al. 2014)
Our goal
Most of recent developments are based on parametric techniques.
◮ We want to focus on non-parametric/data-driven techniques.
◮ We want to deal with “tricky” weather variables (precipitation
accumulation).
Maxime Taillardat 3/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Find in the model climate situations which are the closest
(according to a metric) of a given prediction◮ Substitute this prediction by the “analogs” observations
◮ Bayesian model averaging (Raftery et al. 2005)
◮ Ensemble model output statistics (Gneiting et al. 2005)
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Bayesian model averaging (Raftery et al. 2005)
Forecasted proba =K∑
k=1
[proba from forecaster k
× posterior of forecaster k being correct]
◮ Ensemble model output statistics (Gneiting et al. 2005)
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Main calibration techniques
Most popular techniques :
◮ Analog Method (Hamill and Whitaker 2006)
◮ Bayesian model averaging (Raftery et al. 2005)
◮ Ensemble model output statistics (Gneiting et al. 2005)Under Gaussianity, the EMOS predictive mean is a bias-corrected
weighted average of the ensemble member forecasts. The EMOS
predictive variance is a linear function of the ensemble variance.
Maxime Taillardat 4/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 5/20
The QRF technique Ensemble forecast verification Results Prospects
Quantile Regression Forests (QRF)
◮ Meinshausen 2006 (package R “quantregForest”)
◮ Quantile Regression : estimation of the conditional median or
any other quantile of the response variable given a set ofpredictors (Koenker and Bassett Jr 1978)
◮ Random Forests : aggregating predictions from binary decision
trees (CART) (Breiman 2001)
◮ Non-parametric : elimination of any assumption on the variable
subject to calibration
Maxime Taillardat 6/20
The QRF technique Ensemble forecast verification Results Prospects
Quantile Regression Forests (QRF)
◮ Meinshausen 2006 (package R “quantregForest”)
◮ Quantile Regression : estimation of the conditional median or
any other quantile of the response variable given a set ofpredictors (Koenker and Bassett Jr 1978)
◮ Random Forests : aggregating predictions from binary decision
trees (CART) (Breiman 2001)
◮ Non-parametric : elimination of any assumption on the variable
subject to calibration
Maxime Taillardat 6/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
◮ Let s be the threshold of a predictor Xi , s must create the mostpossible homogeneous branches in terms of variance :
∆R(s, b) = maxs∈Σ[R(b) − (R(bl ) + R(br ))]
where
R(t) =∑
X∈b
(yi − y(b))2
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
From CART to QRF
◮ Binary decision tree
A
B C
◮ Unstable trees (low bias but very high variance) : One fits K trees
using K random samples with replacement of the training set
(bootstrap) : Tree Bagging◮ Strongly correlated trees : each split of each bagged tree is built
on a random subset of the predictors in Σ : Random Forests◮ For each final leaf of each tree one does not compute the mean
of the predictand’s values but instead their empirical CDF :
Quantile Regression
F̂x (y) = P̂(Y ≤ y |X = x) =
n∑
i=1
πi (x)I(Yi ≤ y)
Maxime Taillardat 7/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS
QRF
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Comparison between EMOS and QRF
◮ Raw ensemble : EMOS◮ Calibrated ensemble :
QRF◮ Calibrated ensemble :
Maxime Taillardat 8/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 9/20
The QRF technique Ensemble forecast verification Results Prospects
Paradigm of verification, scoring rules
“The paradigm of maximizing the sharpness of the predictivedistributions subject to calibration” (Gneiting et al. 2006)
◮ A proper score : the CRPS (Murphy 1969 ; Gneiting and Raftery2007 ; Naveau et al. 2015 ; MT 2016)
CRPS(F , y) =
∫
∞
−∞
(F (x)− 1{x ≥ y})2dx
= EF |X − y | −1
2EF |X − X ′|
= y + 2[
F (y)EF (X − y |X > y)− EF (XF (X))]
= EF |X − y |+ EF (X) − 2EF (XF (X))
◮ Test of equal predictive accuracy : the Diebold-Mariano test type
(1995)◮ The CRPSS (Skill score)
CRPSS(A,B) = 1 −CRPSA
CRPSB
Maxime Taillardat 10/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 11/20
The QRF technique Ensemble forecast verification Results Prospects
Data and model fitting
◮ 4 yr of PEARP (ARPEGE 35-member Ensemble PredictionSystem) data from 2011 to 2014 on 87 French SYNOP stations
for 24h lead time, initialization 18H UTC
◮ For EMOS precipitation : GEV and Censored/Shifted Gammadistributions are selected (as in Hemri et al. 2014 ; Scheuerer,
Baran 2015) on a overall CRPS minimization criterion. (GPD isrejected)
◮ For Analog Method : Mahalanobis metric is kept.
◮ Two sets of predictors for QRF technique :
◮ Only with predictors concerning the variable of interest (QRF_O)◮ Like QRF_O with also the first, ninth and fifth decile of other
variable PEARP distributions (QRF_M)
Maxime Taillardat 12/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall
Maxime Taillardat 13/20
The QRF technique Ensemble forecast verification Results Prospects
Assessing performance for extreme and rare events
◮ A weighted score : the wCRPS (Gneiting and Ranjan 2012 ;
Naveau et al. 2015 ; MT 2016)
CRPS(F , y) =
∫
∞
−∞
w(x)(F (x) − 1{x ≥ y})2dx
= W (y) + 2[
F (y)EF (W (X) − W (y)|X > y)− EF (W (X)F (X))]
= EF |W (X)− W (y)|+ EF (W (X)) − 2EF (W (X)F (X))
where W =∫
w and 0 <∫
wf < ∞
◮ The weight function cannot depend on the observation : it leads
to improper scores.
Maxime Taillardat 14/20
The QRF technique Ensemble forecast verification Results Prospects
Weight functions used
0 5 10 15 20 25
05
10
15
20
25
Weight functions
rainfall
W
W1
W2
W3
Weight functions◮
w1(x) = 1 −f (x)
f (0.2)
f is the PDF of the climatology
◮
w2(x) = 1{x ≥ 20}
◮
w3(x) = 2w1(x)W1(x)
Maxime Taillardat 15/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall with weighted scoring rules
Maxime Taillardat 16/20
The QRF technique Ensemble forecast verification Results Prospects
Daily rainfall with weighted scoring rules
w4(x) = 1{x ≤ 15}
Maxime Taillardat 17/20
The QRF technique Ensemble forecast verification Results Prospects
Plan
1 The QRF technique
2 Ensemble forecast verification
3 Results
4 Prospects
Maxime Taillardat 18/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even betterperformance than EMOS unless for very high thresholds :
This is normal : if ie. an event occurs 5 times per year, we have to
get a 7-year training sample in order to build a sound 35-memberensemble with data-driven techniques.
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yrreforecast has been used)
◮ Combination of QRF and GPD CDF fitting (see tomorrow)
◮ Deal with other parameters (TCC : good preliminary results)
◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik
2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even better
performance than EMOS unless for very high thresholds :
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yr
reforecast has been used)
◮ Combination of QRF and GPD CDF fitting (see tomorrow)
◮ Deal with other parameters (TCC : good preliminary results)
◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even better
performance than EMOS unless for very high thresholds :
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yr
reforecast has been used)
◮ Combination of QRF and GPD CDF fitting (see tomorrow)
◮ Deal with other parameters (TCC : good preliminary results)
◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even better
performance than EMOS unless for very high thresholds :
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yr
reforecast has been used)
◮ Combination of QRF and GPD CDF fitting (see tomorrow)
◮ Deal with other parameters (TCC : good preliminary results)
◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
Prospects
◮ QRF technique gives at least the same or even betterperformance than EMOS unless for very high thresholds :
◮ Reforecast work (Hamill and Whitaker 2006 : a 25-yr
reforecast has been used)◮ Combination of QRF and GPD CDF fitting (see tomorrow)◮ Deal with other parameters (TCC : good preliminary results)◮ Recovering spatio-temporal trajectories (eg. ECC Schefzik
2013)
Maxime Taillardat 19/20
The QRF technique Ensemble forecast verification Results Prospects
References
◮ Taillardat, M., O. Mestre, M. Zamo, and P. Naveau, 2016 :Calibrated Ensemble Forecasts using Quantile Regression
Forests and Ensemble Model Output Statistics. Mon. Wea. Rev.doi :10.1175/MWR-D-15-0260.1, in press.
Maxime Taillardat 20/20
References Références
References I
Baran, S., and S. Lerch, 2015 : Log-normal distribution based
ensemble model output statistics models for probabilisticwind-speed forecasting. Quarterly Journal of the Royal
Meteorological Society.
Ben Bouallègue, Z., 2013 : Calibrated short-range ensemble
precipitation forecasts using extended logistic regression withinteraction terms. Weather and Forecasting, 28 (2), 515–524.
Breiman, L., 1996 : Bagging predictors. Machine learning, 24 (2),
123–140.
Breiman, L., 2001 : Random forests. Machine learning, 45 (1), 5–32.
Breiman, L., J. Friedman, C. J. Stone, and R. A. Olshen, 1984 :Classification and regression trees. CRC press.
Maxime Taillardat 9/20
References Références
References II
Courtier, P., C. Freydier, J. Geleyn, F. Rabier, and M. Rochas, 1991 :
The arpege project at meteo-france. ECMWF seminarproceedings, Vol. 2, 193–231.
Descamps, L., C. Labadie, A. Joly, E. Bazile, P. Arbogast, and
P. Cébron, 2014 : PEARP, the Météo-France short-range ensemble
prediction system. Quarterly Journal of the Royal MeteorologicalSociety.
Feldmann, K., M. Scheuerer, and T. L. Thorarinsdottir, 2014 : Spatialpostprocessing of ensemble forecasts for temperature using
nonhomogeneous gaussian regression. arXiv preprintarXiv :1407.0058.
Friederichs, P., and A. Hense, 2007 : Statistical downscaling ofextreme precipitation events using censored quantile regression.
Monthly weather review, 135 (6), 2365–2378.
Maxime Taillardat 10/20
References Références
References III
Friederichs, P., and T. L. Thorarinsdottir, 2012 : Forecast verification
for extreme value distributions with an application to probabilisticpeak wind prediction. Environmetrics, 23 (7), 579–594.
Gneiting, T., F. Balabdaoui, and A. E. Raftery, 2007 : Probabilistic
forecasts, calibration and sharpness. Journal of the RoyalStatistical Society : Series B (Statistical Methodology), 69 (2),
243–268.
Gneiting, T., and M. Katzfuss, 2014 : Probabilistic forecasting. AnnualReview of Statistics and Its Application, 1, 125–151.
Gneiting, T., and A. E. Raftery, 2007 : Strictly proper scoring rules,prediction, and estimation. Journal of the American Statistical
Association, 102 (477), 359–378.
Maxime Taillardat 11/20
References Références
References IV
Gneiting, T., A. E. Raftery, A. H. Westveld III, and T. Goldman, 2005 :
Calibrated probabilistic forecasting using ensemble model outputstatistics and minimum crps estimation. Monthly Weather Review,
133 (5), 1098–1118.
Hamill, T. M., 2001 : Interpretation of rank histograms for verifyingensemble forecasts. Monthly Weather Review, 129 (3), 550–560.
Hamill, T. M., R. Hagedorn, and J. S. Whitaker, 2008 : Probabilisticforecast calibration using ecmwf and gfs ensemble reforecasts.
part ii : Precipitation. Monthly weather review, 136 (7), 2620–2632.
Hamill, T. M., and J. S. Whitaker, 2006 : Probabilistic quantitativeprecipitation forecasts based on reforecast analogs : Theory and
application. Monthly Weather Review, 134 (11), 3209–3229.
Maxime Taillardat 12/20
References Références
References V
Hemri, S., M. Scheuerer, F. Pappenberger, K. Bogner, and T. Haiden,
2014 : Trends in the predictive performance of raw ensembleweather forecasts. Geophysical Research Letters, 41 (24),
9197–9205.
Hersbach, H., 2000 : Decomposition of the continuous rankedprobability score for ensemble prediction systems. Weather and
Forecasting, 15 (5), 559–570.
Meinshausen, N., 2006 : Quantile regression forests. The Journal ofMachine Learning Research, 7, 983–999.
Pinson, P., 2012 : Adaptive calibration of (u, v)-wind ensembleforecasts. Quarterly Journal of the Royal Meteorological Society,
138 (666), 1273–1284.
Maxime Taillardat 13/20
References Références
References VI
Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005 :
Using bayesian model averaging to calibrate forecast ensembles.Monthly Weather Review, 133 (5), 1155–1174.
Schefzik, R., T. L. Thorarinsdottir, T. Gneiting, and Coauthors, 2013 :
Uncertainty quantification in complex simulation models usingensemble copula coupling. Statistical Science, 28 (4), 616–640.
Scheuerer, M., 2014 : Probabilistic quantitative precipitation
forecasting using ensemble model output statistics. QuarterlyJournal of the Royal Meteorological Society, 140 (680), 1086–1096.
Schuhen, N., T. L. Thorarinsdottir, and T. Gneiting, 2012 : Ensemblemodel output statistics for wind vectors. Monthly Weather Review,
140 (10), 3204–3219.
Maxime Taillardat 14/20
References Références
References VII
Sloughter, J. M., T. Gneiting, and A. E. Raftery, 2010 : Probabilistic
wind speed forecasting using ensembles and bayesian modelaveraging. Journal of the american statistical association,
105 (489), 25–35.
Sloughter, J. M. L., A. E. Raftery, T. Gneiting, and C. Fraley, 2007 :Probabilistic quantitative precipitation forecasting using bayesian
model averaging. Monthly Weather Review, 135 (9), 3209–3220.
Thorarinsdottir, T. L., and T. Gneiting, 2010 : Probabilistic forecasts of
wind speed : ensemble model output statistics by usingheteroscedastic censored regression. Journal of the Royal
Statistical Society : Series A (Statistics in Society), 173 (2),371–388.
Maxime Taillardat 15/20
References Références
References VIII
Weijs, S. V., R. Van Nooijen, and N. Van De Giesen, 2010 :
Kullback-leibler divergence as a forecast skill score with classicreliability-resolution-uncertainty decomposition. Monthly Weather
Review, 138 (9), 3387–3399.
Wilks, D. S., 1995 : Statistical methods in the atmospheric sciences.Academic press, 467 pp.
Zamo, M., O. Mestre, P. Arbogast, and O. Pannekoucke, 2014 : A
benchmark of statistical regression methods for short-termforecasting of photovoltaic electricity production. part ii :
Probabilistic forecast of daily production. Solar Energy, 105,
804–816.
Maxime Taillardat 16/20
References Références
Results on surface temperature
Maxime Taillardat 17/20
References Références
Results on surface temperature
Maxime Taillardat 18/20
References Références
Interest of QRF for forecasters
Maxime Taillardat 19/20
References Références
Interest of QRF for forecasters
Maxime Taillardat 19/20
References Références
QRF can have a meteorological interpretation
Maxime Taillardat 20/20
References Références
QRF can have a meteorological interpretation
Maxime Taillardat 20/20