multi-model short-range ensemble forecasting at spanish met agency (aemet) j. a. garcía-moya, a....

Multi-model Short-Range Ensemble Forecasting at

Spanish Met Agency (AEMET)

J. A. García-Moya, A. Callado, P. Escriba, C. Santos, D. Santos, J. Simarro

Spanish Met Service AEMET

Training Workshop onNOWCASTING TECHNIQUESBuenos Aires, August 2013

15 August 2013 T-Note Workshop 2

OutlineIntroductionEPS for short-range forecastSREPS system at AEMETVerification exercise

Against analysisAgainst synoptic observationsAgainst climate network observations

Probabilistic scoresSynoptic variables10 m surface wind speed6h accumulated precipitation24h accumulated precipitation

Time-Lagged Super-ensembleComparison with ECMWF EPSMulti-model predictabilityConclusions


Meteorological Framework

Main Weather Forecast issues are related with Short-Range forecast of extreme events.Convective precipitation is the most dangerous weather event in the Mediterranean.


Geographical Framework

Western Mediterranean is a close sea rounded by high mountains.In autumn sea is warmer than air.Several cases of more than 200 mm/few hours occurs every year.Some fast cyclogenesis like “tropical cyclones” also appears from time to time (“medicanes”).


Geographical Framework


Ensemble for Short Range

Surface parameters are the most important ones for weather forecast.Forecast of extreme events (convective precip, gales,…) is probabilistic.Short Range Ensemble prediction can help to forecast these events.Forecast risk (Palmer, ECMWF Seminar 2002) is the goal for both Medium- and, also, Short-Range Prediction.


Errors of short-range forecast

Due to model formulation.Due to simplifications in parameterisation schemes.Due to uncertainty in the initial state.Special for LAMs, due to errors in lateral boundary conditions.Due to uncertainties in soil fields (soil temperature and soil water content, …).


SREPS I

Multi-model approach (Hou & Kalnay 2001).Stochastic physics (Buizza et al. 1999).Multi-boundaries:

From few global deterministic models.From global model EPS (ECMWF).SLAF technique (Ebisuzaki & Kalnay 1991).


SREPS II

Different assimilation techniques:Optimal Interpolation.Variational (3D or 4D).

Perturbed analysis:Singular vectors (ECMWF, Palmer et al. 1997).Breeding (NCEP, Toth & Kalnay 1997).Scaled Lagged Average Forecasting (SLAF, Ebisuzaki & Kalnay 1991).EnKF, ETKF, LETKF, PO


Multi-model

Hirlam (http://hirlam.org).

HRM from DWD (German Weather Service).

MM5 (http://box.mmm.ucar.edu/mm5/).

UM from UKMO (UK Weather Service).

LM (COSMO Model) from COSMO consortium (http://www.cosmo-model.org).

WRF (NOAA – NCEP) – work in progress


Multi-Boundaries

From different global deterministic models:ECMWFUM from UKMO (UK Weather Service)

GFS from NCEPGME from DWD (German Weather Service)

CMC from SMC (Canadian Weather Service)


SREPS at AEMET

Mummub: Multi-model Multi-boundaries72 hours forecast two times a day (00 & 12 UTC).Characteristics:

5 models.5 boundary conditions.2 latest ensembles (HH & HH-12).

25 member ensemble every 12 hoursTime-lagged Super-Ensemble of 50 members every 12 hours.


Post-processing

Integration areas 0.25 latxlon, 40 levelsInterpolation to a common area

~ North Atlantic + EuropeGrid 380x184, 0.25º

SoftwareEnhanced PC + LinuxECMWF Metview + Local developments

OutputsDeterministicEnsemble probabilistic


Monitoring in real time

Intranet web serverDeterministic outputs

Models X BCs tables Maps for each couple (model,BCs)

Ensemble probabilistic outputsProbability maps: 6h accumulated precipitation, 10m wind speed, 24h 2m temperature trendEnsemble mean & Spread mapsEPSgrams (work in progress)

Verification: Deterministic & ProbabilisticAgainst ECMWF analysisAgainst observations


Monit: all models X bcs

Whole Area

Zoom over Spain


Monit: Spread – Ensemble mean maps

Spread at key

mesoscale areas


Case Study 06/10/2006 at 00 UTC

More than 15 mm/6 hours


Verification

Verification exercise, April-June 2006:

Calibration: with synoptic variables Z500, T500, PmslResponse to binary events: reliability and resolution of surface variables 10m surface wind, 6h and 24h accumulated precipitation


Interpolation


Obs verification - Probabilistic scoresEnsemble calibration:

Synoptic variables:Z500, T500, Pmsl

Scores:Rank histogramsSpread-skill

Response to binary events:Surface variables:

10m surface wind (10,15,20m/s thresholds)6h accumulated precipitation (1,5,10,20mm thresholds)24h accumulated precipitation (1,5,10,20mm thresholds)

Scores:Reliability, sharpness (H+24, H+48)ROC, Relative Value (H+24, H+48)BSS, ROCA with forecast length


Summary of Probabilistic Verification


Synoptic parameters

Using ECMWF Analysis as reference:MSLP

Over all FC lengths H+00 .. H+72:Spread-skill

H+72:Rank histograms

Using Synoptic observations as reference:MSLP

Over all FC lengths H+00 .. H+72:Spread-skill

H+72:Rank histograms


MSLP-ECMWF


MSLP - Obs


Binary events

Binary events X = {0,1} at every pointAccumulated precipitation in 24 hours >= 5mmUseful to decompose the forecast in thresholdsPerformance computed using contingency tables (CT’s)


Contingency tables It is the best way to characterize a binary event

fc(X)={1,0}ob(X)={1,0}

ob

1 0

fc1 a b a+b

0 c d a+d

a+c

b+d

a+b+c+d = N


Contingency tables: scores

Several scores can computed from CT’s

Base Rate s ( a + c ) / n

HitRate HIR a / ( a + c )

False Alarm Rate

FAR b / ( b + d )

False Alarm Ratio

FARatio

b / ( a + b )

Proportion Correct PC ( a + d ) / n

…

ob

1 0

fc1 a b a+b

0 c d a+d

a+c

b+d

a+b+c+d = N


Example: Every pdf threshold has its own CT

ob

1 0

P(X) >= 0/n1 a0 b0

0 c0 d0

P(X) >= 1/n1 a1 b1

0 c1 d1

P(X) >= 2/n1 a2 b2

0 c2 d2

…P(X) >= n/n

1 an bn

0 cn dn

HIR0 FAR0

HIR1 FAR1

HIR2 FAR2

HIRn FARn


ReliabilityObservation frequency conditioned to forecast probabilityReliability diagrams

( i/N , Δai/(Δai+Δbi) ) i = 0…NNear diagonal ~ reliable

Sharpness histograms( i/N , Δai+Δbi ) i = 0…NU shape ~ sharp (discriminating binary event)


Precipitation

Using ECMWF Deterministic Model as reference:

6 hours accumulation – 24 hours forecast length24 hours accumulation – 54 hours forecast lengthThresholds 1, 5, 10 y 20 mm

Using Synoptic observations as reference:6 hours accumulation – 24 hours forecast length24 hours accumulation – 54 hours forecast lengthThresholds 1, 5, 10 y 20 mm


ECMWF

Reliab. - 24 h Acc. Precip H+54 (1,5,10,20) mm



ObservationsReliab. - 6 h Acc. Precip H+24 (1,5,10,20) mm



Resolution

Based on Signal Detection TheoryROC (Relative Operating Characteristics)

( FARi , HIRi ) i = 0…N

ROCArea ~ Resolution or binary event discrimination

Forecast conditioned by observation


ROC curves – 24 h Acc Precip (1, 5, 10 & 20 mm)

ECMWF Observations


ECMWF - AnalysisR

OC

10 m wind H+24 (10,15,20) m/s

Reliab

ilit

y


Observations10 m wind H+24 (10,15,20) m/s

Reliab

ilit

yR

OC


Joint


Reliability & Sharpness

Good reliability according tothresholds (base rate)forecast length

NoUnder-samplin

g


Resolution

Good resolutionROC AreasBSSs

Good RV curves0

1

0.5

1


Time-Lagged Super-ensemble

How much predictability can be added by a time-lagged super-ensemble?40 members super-ensemble (SE-SREPS) with the last two runs of SREPS ( HH & HH-12).Verifications against observationsCheap in terms of computer resourcesJust a different post-process


Spread-skill


Rank histogram


Reliability diagrams


Comparison with ECMWF EPS

Period: April to June 2006European Synop obs: H+72.

Mslp / v 10m / Precipitation

European climate precipitation network: H+54 (longest SREPS period matching observations).

24 hours accumulated precipitation (from early morning to early morning).


Climate Obs – 24 h. Precip (1, 5, 10 & 20 mm)



Brier Skill Score (BSS) Decomposition – 106 realizations

Precip (mm)

Climatolog. Frequency

Uncertainty

MumMub ECMWF 21 ECWMF 51

1 0.24 0.18 0.35 0.28 0.28

5 0.11 0.09 0.27 0.22 0.22

10 0.05 0.05 0.18 0.18 0.19

20 0.01 0.01 0.01 0.07 0.08

Precip (mm)


1 0.06 0.13 0.14

5 0.03 0.08 0.08

10 0.03 0.03 0.03

20 0.07 0.02 0.02Precip (mm)


1 0.59 0.59 0.59

5 0.70 0.70 0.70

10 0.79 0.79 0.78

20 0.91 0.91 0.90


24 hours accumulated precipitation from European climate network upscaled to 0.25 deg. Resolution - BSS


Why Multi-model?

Better representation of model errors (SAMEX, Hou & Kalnay 2001, and DEMETER).Consistent set of perturbations of initial state and boundaries.Better results than any single model ensemble (SAMEX, DEMETER, Arribas et al., 2005).


Single model Ensembles (4 members each)


BMA is a statistical method for postprocessing ensembles based

on standard method for combining predictive distributions from

different sources.

The BMA predictive PDF of any quantity of interest is a weighted

average of PDFs centred on the individual bias-corrected

forecasts, where the weights are equal to posterior probabilities

of the models generating the forecasts and reflect the models’

relative contributions to predictive skill over the training period.

Raftery et al, 2005

BMA INTRO


BMA FUNDAMENTALS

F2

F1

F4

F5

F3

BMA predictive PDF


BMA calibration using TEMP and SYNOP observations over whole area :

500 hPa Temperature (T500)500 hPa Geopotencial (Z500) 10m Wind speed (S10m)

3, 5 and 10 days of training period

3 months of calibration (April, May and June of 2006) for Z500 and T500 and 1 month for S10m (April 2006)

24, 48 and 72 hours forecast for Z500 and T500 and

24 and 72 hours forecast for S10m

BMA EXPERIMENTS


RESULTS

MULTIMODEL

BMA 3 T. DAYS

BMA 5 T. DAYS

BMA 10 T. DAYS


HRM T500 td 3 H+72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T HAV

HEC

HGM

WEIGHT TIME SERIES

HIRLAM T500 td 3 H+72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATEW

EIG

HT

IAV

IEC

IGM

IUK

LM T500 td 3 H+72

0,000

0,100

0,200

0,300

0,400

0,500

0,600

0,700

0,800

0,900

1,000

DATES

WE

IGH

TS LAV

LEC

LGM

LUK

MM5 td 3 H+72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

MAV

MEC

MGM

MUK

UM T500 td 3 H+72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

UAV

UEC

UGM

UUK

HRM HIRLAM LOKAL MODEL

MM5 UM


HIRLAM td 3 H+48

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK


0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

WEIGHT TIME SERIES

HIRLAM td 3 H+24

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

HIRLAM



0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

WEIGHT TIME SERIES

HIRLAM td 5 H+72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

HIRLAM td 10 h+ 72

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

HIRLAM


WEIGHT TIME SERIES

HIRLAM td 3 H+24

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WE

IGH

T

IAV

IEC

IGM

IUK

Z500 td 3 H + 24

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

DATE

WEI

GH

T

IAV

IEC

IGM

IUK

HIRLAM


Summary I

A Multi-model-Multi-boundaries Short Range Ensemble Prediction System (MMSREPS), has been developed at the AEMET-SpainWe show here 3months verification results (April-June 21006), against both synoptic and climate observations and ECMWF analysis and model:

Response to binary events: reliability and resolution of surface variables 10m surface wind, 6h and 24h accumulated precipitation

These results look promising:Verification against ECMWF analysis and model shows very good resultsVerification against observations shows quite good results

Ensemble is a bit under-dispersive Good response to binary events (synoptic and climate observations)


Summary II

A Time-Lagged Super-ensemble with 40 members had shown better performance than the Multi-model SREPS alone.Multi-model technique gives much better spread than any other single-model techniqueBias correction and calibration through a Bayesian Model Averaging scheme is under development (first results show better performance).


Questions

[email protected]

multi-model short-range ensemble forecasting at spanish met agency (aemet) j. a. garcía-moya, a....

Documents