Huiling Yuan1, Xiang Su1, Yuejian Zhu2, Yan Luo2, 3, Yuan Wang1
1. Key Laboratory of Mesoscale Severe Weather/Ministry of Education and School of Atmospheric Sciences, Nanjing University, China
2. Environmental Modeling Center/NCEP/NWS/NOAA, College Park, Maryland, USA
3. I.M. Systems Group, Inc., College Park, Maryland, USA
WWOSC 2014, Montreal, Canada, 21 August 2014
Evaluation of TIGGE ensemble predictions of Northern Hemisphere
summer precipitation during 2008-2012
Outline
Objectives The TIGGE Ensemble Prediction Systems (EPSs) data Evaluation results of TIGGE quantitative precipitation
forecasts (QPFs) and probabilistic QPF (PQPFs) Summary
the THORPEX Interactive Grand Global Ensemble (TIGGE)
Objectives
Evaluate the QPF and PQPF performance of TIGGE
EPSs
Assess the performance change before and after
major EPS upgrade
Su et al. 2014, JGR
Evaluation of TIGGE QPFs and PQPFs
Study period: 2008-2012 summer (June-August)
Spatial coverage: Northern Hemisphere (NH)
tropics (0-20°N) and midlatitude (20-49°N)
Accumulation period: 24-h precipitation (12 UTC-12 UTC)
Forecast lead time: 1-9 days
Horizontal resolution: 1° ×1° (interpolation)
4
Observation data: TRMM 3B42 V7 (gauge adjustment)
ECMWF portal http://tigge-portal.ecmwf.int/
Center Base time(UTC) members
Horizontal resolutionarchived
Fcstlength(day)
Initialperturb method
Modeluncertainty
Major EPS upgrade time
CMA(China)
00/12 14+1 0.56º×0.56º 0-10 BVs - -
CMC(Canada)
00/12 20+1 1.0º×1.0º 0-16 EnKF PTP + SKEBmulti-physics
17 Aug 2011
ECMWF 00/12 50+1 N320 (~0.28º)N160 (~0.56º)
0-1010-15
EDA-SVINI
SPPT + SPBS 9 Nov 2010
JMA 12 50+1 1.25º×1.25º 0-9 SVs SPPT 17 Dec 2010
NCEP 00/06/12/18
20+1 1.0º×1.0º 0-16 BV-ETR STTP 23 Feb 2010
UKMO 00/12 23+1 0.83º×0.56º 0-15 ETKF RP + SKEB 9 Mar 2010
1. The CMC EPS was upgraded to version 2.0.2 on 17 August 2011.2. The ECMWF EPS used a horizontal resolution of N200 (~0.45º) for 0-10 day forecasts and N128 (~0.7º) for 10-15 day forecasts before 26 January 2010. EVO-SVINI was used as the initial perturbation method before 24 Jun 2010. The SPBS method has been added on 9 November 2010.3. The JMA EPS began to use the SPPT method on 17 December 2010.4. The NCEP EPS was upgraded to version 8.0 and began to use the STTP method on 23 February 2010. In 14 February 2012, the NCEP EPS was upgraded to version 9.0.5. The UKMO EPS used a horizontal resolution of 1.25º×0.83º before 9 March 2010.
Referenced to the CMA EPS (frozen), the impacts of major model upgrades on the forecast performance are examined for other five EPSs.
Configurations of six TIGGE EPSs
Verification methods of QPFs and PQPFs
Area-weighted scores consider the latitude discrepancies Continuous scores: RMSE, Spatial correlation (SC), CRPSS, and
spread-skill relationship Dichotomous scores: Bias, ETS, POD, FAR, BSS,
attributes diagram (reliability curve and three decomposed terms of BS), ROC area and potential economic value (PEV)
6
where :wi=cos(lat)xi, yi are forecast and observation samples, N is the number of samples
Su et al. 2014, JGR
Precipitation climatology (day +3)
7
JMA day +1 ensemble mean QPFshave large moist biases in the NH
tropics
Cause: JMA employs moist SVs over the entire tropics and perturbs the specific humidity with a large amplitude (Yamaguchi and Majumdar, 2010)
TRMM
JMA day 1
JMA
NCEPCMA
CMC
ECMWF
UKMO
mm/day
Forecast error (RMSE)
Control:dotted (…..)
Ensemble mean:solid (——)
Ensemble is better than control, especially in longer lead times
EC ensemble best
EC control is better than CMA ensemble in short lead times
JMA control in NH midlatitude best
RMSE and frequency
9
Cause: JMA underestimates heavy rain
It is not appropriate to use only RMSE to evaluate QPFs
0 10 20 30 40 50 60 70 80 90 100-6
-5
-4
-3
-2
-1
0
Precipitation (mm day-1)
Log1
0 of
fre
quen
cyPDF of Day+3
TRMM OBSCMA
CMC
ECMWF
UKMO
NCEPJMA
The control QPFs of JMA have the smallest RMSE in the NH midlatitude
Discrimination diagrams
The discrimination ability decreases with the lead time
EPSs have weak ability to discriminate heavy events
In the NH tropics, CMA shows little discrimination ability among different rain events
10
Dichotomous scores of ensemble mean QPFs
ECMWF best CMA very poor in the NH tropics
11
Spread-skill relationship
The day +1 ensemble spread of JMA is the largest in the NH tropics
CMC has the largest spread and it grows with the lead time:
NH midlatitude: level with the ensemble mean error
NH tropics: slightly overdispersive
12Spread: dash (- - -), RMSE: solid (——)
PQPF error: CRPSS
ECMWF best
In the NH tropics, the day +1 CMA is even poorer than the day +9 ECMWF
Skill of CMC rapidly drops from day +2
13
PQPF skill: BSS and ROC area
Light rain:
CMC best
Heavy rain:
CMC and ECMWF better
CMA is very poor in the NH tropics
14
Attributes diagram (reliability curve, BSS, and BS terms)
NH Midlatitute
Potential economic value
Prob. thresholds of CMC are most reliable
ECMWF has the highest PEV
Performance changes due to major EPS upgrade
Spread: grey, RMSE: black
Changes of spread, Spread/RMSE ratio are significant for CMC and ECMWFChange of RMSE is only significant for CMC (increase)
Performance changes due to major EPS upgrade
Spread: grey, RMSE: black
Changes of spread, Spread/RMSE ratio are significant for UKMO, NCEP, JMAChange of RMSE is only significant for UKMO (decrease)
Score Center Before After Change
Spread CMC-CMA 1.1 4.4 3.3
ECMWF-CMA -0.1 -0.7 -0.6
UKMO-CMA -0.4 -0.2 0.2
NCEP-CMA -1.7 -0.8 0.9
JMA-CMA -1.6 -1.3 0.3
RMSE CMC-CMA -0.4 -0.1 0.3
ECMWF-CMA -0.7 -0.8 -0.1
UKMO-CMA -0.1 -0.3 -0.3
NCEP-CMA 0 -0.2 -0.2
JMA-CMA -0.5 -0.3 0.2
Spread CMC-CMA 0.20 0.59 0.39
/RMSE ECMWF-CMA 0.05 -0.02 -0.07
ratio UKMO-CMA -0.05 0 0.05NCEP-CMA -0.23 -0.10 0.13
JMA-CMA -0.19 -0.16 0.03
Changes due to major EPS upgrade
Use the frozen CMA as the reference to eliminate the interannual variability
CMC greatly increases spread and spread/RMSE ratio
Similar performance changes for other lead times
Changes due to major EPS upgrade of day +3 spread, RMSE, and Spread/RMSE ratio in the NH midlatitude (significant change with 95% confidence interval)
20
Skill changes
After EPS major upgrade:
CMC: decreased skill
ECMWF, UKMO,
NCEP: improved
JMA: no significant
change
Wednesday February 13, 2013 Major upgrade to the Global Ensemble Prediction System (GEPS)
version 3.0.0 at the Canadian Meteorological Centre
“Changes installed uniquely into the forecast component include: adjustments to how physics tendencies are perturbed for convective precipitation; the physics tendencies perturbations are applied at every level except the very last one; addition of diffusion into the advection procedure; perturbation of the bulk drag coefficient in the orographic blocking scheme; and fine tuning of the adjustment factor alpha of the stochastic kinetic energy backscattering scheme.”
http://collaboration.cmc.ec.gc.ca/cmc/CMOI/product_guide/docs/changes_e.html#20130213_geps
21
CMC EPS upgrade
Evaluation of the QPFs and PQPFs from six TIGGE EPSs in the NH midlatitude and tropics during the boreal summers of 2008-2012:
Ensemble mean QPFs:
CMA: large systematic biases, poor performance in the NH tropics
ECMWF: less errors and best skill
JMA: unusually large moist biases of day +1 QPFs in the NH tropics PQPFs:
CMC: relatively good for light precipitation and short lead times,
increased spread and larger errors for longer lead times;
better reliability and reliable probabilistic thresholds in PEV
ECMWF: best skill, except for light precipitation;
best discrimination ability and highest potential economic benefit
NCEP and UKMO: most sharp
22
Summary
The model upgrade in EPS cannot always guarantee skill improvements
The enlarged ensemble spread of CMC forecasts after the upgrade increases the QPF errors
Uncertainties and quality of verification data
How to fairly evaluate an EPS is essential for the development and upgrade of the EPSs
Comprehensive evaluation with multiple verification metrics
Provide general guidance for the postprocessing of the EPSs
Reliability and discrimination ability
23
Summary
E-mail: [email protected]
Thank you