© Crown copyright Page 1

Sarah Beare (nee John)

Neill Bowler, Marie Dando, Anette Van der Wal, Rob Darvell

Performance of the MOGREPS Regional Ensemble

© Crown copyright Page 2

Outline

Overview of MOGREPS Configuration

Overview of probabilistic verification

Results

Summary and future work

© Crown copyright Page 3

MOGREPS-Regional

Regional Ensemble

Regional ensemble over N. Atlantic and Europe (NAE)

23 perturbed members + control

IC Perturbations from regional ETKF

LBC from global ensemble

Stochastic physics

- random parameters

24km resolution, 38 levels

Run at 6Z & 18Z to T+54

© Crown copyright Page 4

MOGREPS Operational System diagram

New global analysis

Perturbations mixed and scaled by ETKF

Global ensemble forecast using stochastic physics

New NAE analysis

0Z 12Z 18Z06Z

New NAE analysis

LBCs

MOGREPS Verification

© Crown copyright Page 6

Verification

Aim to verify products issued from MOGREPS system

Two systems available

Focus on surface parameters and higher thresholds

Further results available in verification report recently published on met office website.

http://www.metoffice.gov.uk/research/nwp/publications/papers/technical_reports/index.html

© Crown copyright Page 7

Site-specific Verification system

- 79 sites across the UK and Europe.

- Probabilistic forecast data binned into 10% categories

- Ability to compare with ECMWF forecasts

- Ability to compare RAW model forecasts and post-processed data

- Can calculate and plot :

Reliability diagrams, Empirical ROC Curves, Brier Score and decompositions Brier Skill Score

- CRPS in the process of being added

© Crown copyright Page 8

Area Based Verification System

- Can verify against observations or analyses - Forecast probabilities not binned

- Can calculate and plot :

Reliability and attributes diagrams, Empirical and parametric ROC, Economic value, Brier Score and decompositions Brier Skill Score Spread and Skill based measures

- Would like to include the RPS, CRPS

- Can compare Global v Regional ensemble.

- THORPEX group introducing ECMWF EPS data for comparison purposes (no facility to account for different ensemble sizes).

© Crown copyright Page 9

Performance of ensemble and high-res (12km)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 12 24 36 48

Lead time (h)

RM

SE

(1

0m

win

d s

pe

ed

)

High-res

Control

Ensemble mean

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0 12 24 36 48

Lead time (h)

RM

SE

(s

tati

on

te

mp

era

ture

)

High-res

Control

Ensemble mean

Ensemble mean has lower RMSE than high-res for day 2 forecasts, indicating presence of substantial uncertainties

Verification for UK stations for Dec 2006 to Feb 2007. T+48 data only available after 6 Feb

10m w/s

2m Temp

© Crown copyright Page 10

Spread-skill relationship – wind speed T+30

Spread-skill for wind-speed binned into equal population bins by spread

Skill corrected for observation error

Blue – MOGREPS Pink – No SkillGreen – Perfect

Grad = 0.93

Grad = 0.05

Grad = 0.59

0

1

2

3

4

5

0 1 2 3 4 5

Average Standard Deviation (ms-1)

RM

SE

(m

s-1)

MOGREPS

No Skill

Perfect Spread

Grad = 0.60

Grad = -0.05

Grad = 0.98

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

Average Standard Deviation (ms-1)

RM

SE

(m

s-1

)MOGREPS

No Skill

Perfect Spread

DJF 06/07

JJA 06

© Crown copyright Page 11

Spread-skill relationship – temperature

Spread-skill for temperature binned into equal population bins by spread Skill corrected for

observation error

Blue – MOGREPS Pink – No SkillGreen – Perfect

DJF 06/07

JJA 06

Grad = 0.91

Grad = 0.63

Grad = 0.03

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

Average Standard Deviation (K)

RM

SE

(K

)

MOGREPS

No Skill

Perfect Spread

Grad = 0.96

Grad = 0.01

Grad = 0.08

0

1

2

3

4

5

0 1 2 3 4 5

Average Standard Deviation (K)

RM

SE

(K)

MOGREPS

No Skill

Perfect Spread

Temperature

© Crown copyright Page 13

Temp DJF06/07 (79 sites UK & Europe)

Temp < 0oC T+36 Temp < -2oC T+36

© Crown copyright Page 14

Brier Skill Score components - Temperature > 10oC 6 Nov 2006 - 28 Feb 2007

Temp>10C

79 sites UK & Europe 6 Nov 2006 – 28 Feb 2007

Brier Skill score

Reliability

Resolution

© Crown copyright Page 15

Reliability diagram for surface temperature

Reliability diagram for Temp>10C

79 sites UK & Europe

6 Nov 2006 – 28 Feb 2007

© Crown copyright Page 16

Temperature > 10oC KFMOS

Kalman-filter bias-corrected forecasts.

Reliability

Resolution

Brier Skill score

Precipitation

© Crown copyright Page 18

Brier Skill Score components for Precipitation

12-hour precip>0.5mm

79 sites UK & Europe 1 July 2006 – 31 March 2007

12-hour precip>5mm

© Crown copyright Page 19

Reliability diagram for precipitation

Reliability diagram for 12h precip>5mm

79 sites UK & Europe

1 July 2006 – 31 March 2007

© Crown copyright Page 20

6hr precip > 0.3mm against gridded analysis

•Reliability and sharpness diagram for T+36 forecast.

•6h precip > 0.3mm

•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.

•1st Jan 06 – 28th Feb 07

© Crown copyright Page 21

•Reliability and sharpness diagram for T+36 forecast.

•6h precip > 5mm

•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.

•1st Jan 06 – 28th Feb 07

6hr precip 5mm against gridded analysis

© Crown copyright Page 22

•Reliability and sharpness diagram for T+36 forecast.

•6h precip > 25mm

•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.

•1st Jan 06 – 28th Feb 07

6hr precip 25mm against gridded analysis

Wind speed

© Crown copyright Page 24

Reliability diagram for wind speed

Reliability diagram for wind speed >F5 at T+30

79 sites UK & Europe

6 Nov 2006 – 31 March 2007

© Crown copyright Page 25

Wind speed at least gale force 8

•Reliability and sharpness diagram for T+36 forecast.

•10m Wind > F8

•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07

© Crown copyright Page 26

Wind speed at least severe gale force 9

•Reliability and sharpness diagram for T+36 forecast.

•10m Wind > F9

•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07

© Crown copyright Page 27

Wind speed at least storm force 10

•Reliability and sharpness diagram for T+36 forecast.

•10m Wind > F10

•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07

© Crown copyright Page 28

Summary of verification

Ensemble mean beats control and hi-res from ~18h

Very good reliability for non-extreme events

Useful skill for some more severe events

T<-2C

Windspeed > F9

6h Precip > 5mm/12h

KFMOS bias correction effective in reducing many biases

Better forecasts from NAE than ECMWF, even after KFMOS for some

parameters

Combination of resolution and perturbation strategy

Useful spread-skill in many cases

Exception is temperature in summer

© Crown copyright Page 29

Future Plans

Fully operational early 2008

SKEB2 scheme including convection for improved stochastic physics

Vertical localization of ETKF to improve near-surface perturbation scaling

Perturbations to lower boundary (eg soil moisture, SST)

Enhanced resolution: Global 60km from 2009

Regional 16km from 2009 and 12 from 2011

Small convection-resolving ensemble from 2011 ~1.5km over UK

~6 members

Nested on members selected by clustering

© Crown copyright Page 30

Any Questions?

© Crown copyright Page 31

More Information….

Technical Reports

453 Comparison of error breeding, singular vectors,random perturbations and ensemble Kalman filter perturbation strategies on a simple model. N Bowler (2004)

452 Results of an initial stochastic physics scheme for the Met Office Unified Model. A. Arribas (2004)

503 The MOGREPS short-range ensemble prediction system: Verification report - Trial Performance of MOGREPS January 2006 - March 2007 N Bowler, M Dando, S Beare and K Mylne (2007)

497 The MOGREPS short-range ensemble prediction system Part 1: System Description

N Bowler, A Arribas, K Mylne and K Robertson, (2007)

© Crown copyright Page 32

NEW! Regional ETKF

Until end of May MOGREPS Perturbations for the NAE were interpolated from the global at T+6

Large scale perturbations Spread too large (especially at upper levels)

Since beginning of June perturbations to regional ensemble generated by a regional ETKF.

Smaller, more realistic perturbations Better representation of important short length scales But surface perturbations may be too small

© Crown copyright Page 33

Subjective Evaluation of Severe Weather

Forecaster comments/feedback Severe Gale Event – 02/09/06

…. the NAE MOGREPS output clearly captured the strength and gave a good forecast for potential severe gale gusts…. (Paul Davies, Chief Forecaster)

Intense rainfall - 20/07/07

… The risk of very high totals is confirmed by NAE MOGREPS members and also other models… (Steve Willington, Chief Forecaster)