bulk statistics on ensemble model forecasts for mdss demo 2003

Bulk Statistics on Ensemble Model Forecasts for MDSS

Demo 2003

Paul Schultz

NOAA Forecast Systems Laboratory

June 17, 2003

The MDSS ensemble modeling component

• What is it?– Several computer model forecasts to supplement the NWS model

forecast services

• Why are we doing this?– Better forecasts. Just seeing if you’re paying attention.

• How does it work?– By combining multiple (imperfect) forecasts of the (imperfectly

observed) atmosphere, we can make a single ensemble forecast that is better than any of the forecasts that went into it.

2

Ensemble modeling

• Did it work during the 2003 Demo?– Not as well as it can. It shows promise. It can be improved.

3

The ensemble for Demo 2003

• Three models, two LBC source models, total of six ensemble members– models: MM5, RAMS, WRF– LBC sources (from NCEP): AVN, Eta– 6-hour cycle– 27-hour forecasts– 12-km grid

5

Bulk statisticsState variables, 12-hr forecasts

Feb 1 – Apr 8, 2003

6

Temperature (K) Wind speed (m/s) Dewpoint (K)

MM5-AVN 3.1 -0.7 2.5 +0.8 5.6 +1.5

MM5-Eta 3.0 -0.5 2.5 +0.8 5.5 +1.6

RAMS-AVN 5.8 -1.1 2.6 +1.6 6.5 -0.9

RAMS-Eta 5.9 -1.1 2.6 +1.7 6.9 -1.0

WRF-AVN 3.1 -0.4 2.4 +1.1 5.7 +1.4

WRF-Eta 3.1 -0.4 2.4 +1.0 5.7 +1.3

Precipitation verification

MDSS 0-3 h QPF Equitable Skill Score 133 runs from 1 Feb - 26 Mar 2003

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.01 0.1 0.25 0.5

Precip Threshold (in)

Eq

uit

able

Ski

ll S

core

MM5

RAMS

WRF

7

MDSS 0-3 h QPF Bias Score 133 runs from 1 Feb - 26 Mar 2003

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0.01 0.1 0.25 0.5

Precip Threshold (in)

Fre

qu

ency

Bia

s

MM5

RAMS

WRF

A closer look

9 pm model runs, verifying only Iowa stations, entire expt

Improving the ensemble

• Remove unhelpful members– If we can’t fix RAMS problems, it’s gone

– Different LBC models don’t seem to help (?????)

8

Unhelpful members

MM5+Avn

WRF+Avn

MM5+Eta

WRF+Eta

The LBC models don’t add enough

diversity

Improving the ensemble

• Add good models– FSL/RUC a very good candidate for Demo 2004

• Change model configurations– WRF cloud/precip physics– Model cycle frequency, lead times, etc.

• Optimize use of available computing resources• Take advantage of what regional models do best

• Improve post-processing– Better PoP (probability of precip) estimates -- FSL– Better tuning procedures -- NCAR– Hope for “better” weather during tuning period

Reliability9

Percentage of expected FSL model runs

0

10

20

30

40

50

60

70

80

90

100

2/3 2/10 2/17 2/24 3/3 3/10 3/17 3/24 3/31 4/7

NCEP data problems Giant

snowstorm in Boulder

Planned power

outage at FSL

Reliability

• MM5 shows good reliability

• Others will improve with better scripting

10

Percentage of expected FSL model runs

0

10

20

30

40

50

60

70

80

90

100

MM5-AVN MM5-ETA RAMS-AVN RAMS-ETA WRF-AVN WRF-ETA

Photos from MDSS field trip

Bob Stradley and Ron Simmons

Downward-pointed radiometer mounted on rear-view mirror of

Jim Van Sickle’s truck

RWIS tower, I-35 south of Ames