Download - Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES
![Page 1: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/1.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Preliminary report on verification scores for malaria and the climate drivers DEMETER
to ENSEMBLES
Andy Morse, Cyril Caminade and Anne Jones
Department of Geography,University of Liverpool,
Liverpool,United Kingdom
![Page 2: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/2.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Running Order
• Introduction• Malaria verification – ENSEMBLES stream II –limited area• Climate verification –
ENSEMBLES stream II - all Africa•Summary and Way Forward
![Page 3: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/3.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Introduction
• Normally use BSS and ROCSS to work with probabilistic data
• Running RMSE, ACC and MSSS deterministically on ensemble mean
• Developed/modified scripts for skill scores in NCAR Command Language (NCL) – share & cross validate?
• Work not complete – will report via ENSEMBLES website
![Page 4: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/4.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Introduction – ROCSS malaria modelling
Tier-2 malaria runs - ROC Skill Scores Above Median Event
DEMETER driven LMM. Areas of high interannual variability were selected and persisted forecast skill was removed from the scores.
Jones, A. and Morse, A. (2007) CLIVAR Exchanges, 43
May 4-6JAS
Nov 4-6FMA
![Page 5: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/5.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Validation of Liverpool Malaria Model (LMM) runs
forced by ENSEMBLES stream2 seasonal forecasts
![Page 6: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/6.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
LMM dynamic malaria mode forced with daily Tx and pptn.
The domain: Botswana: [27.5°S-17.5°S] [17.5°E-30°E]
Table of experiments:
Expid name Project Institution/Model Period Start date (lead months)
Ensemble members
ecmf ENSEMBLES ECMWF IFS-HOPE
1960-2006
Nov 1 (14) 9
egrr ENSEMBLES UK Met-Office HadGem2
1970-2005
Nov 1 (14) 9
lfpw ENSEMBLES Meteo-France ARPEGE4/OPA
1960-2005
Nov 1 (14) 9
ingv ENSEMBLES INGV ECHAM5/OPA8.2
1960-2005
Nov 1 (7) 9
ifmk ENSEMBLES Kiel ECHAM5/OM1
1960-2005
Nov 1 (14) 9
![Page 7: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/7.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2Scores:
• ACC: anomaly correlation coefficients
• Addresses the question: How well did the forecast anomalies correspond to the observed anomalies?
• RMSE: Root mean square errors
• Answers the question: What is the average magnitude of the forecast errors?
• MSSS: Mean square skill score
• withcj
jj MSE
MSEMSS 1
2
1
)(1
n
ijijcj xx
nMSE
2
1
)(1
n
iijijj xf
nMSE
![Page 8: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/8.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1961-2000 NDJF:
RMSE for Malaria Incidence over Botswana (NDJF 1961-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and
the analysis is applied to the ensemble means.
![Page 9: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/9.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1961-2000 NDJF:
ACC for Malaria Incidence over Bostwana (NDJF 1961-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. The dotted area display
significant correlations at the 95% confidence interval
![Page 10: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/10.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1961-2000 NDJF:
MSSS for Malaria Incidence over Botswana (NDJF 1961-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and
the analysis is applied to the ensemble means.
![Page 11: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/11.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
RMSE for Malaria Incidence over Botswana (NDJF 1981-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and
the analysis is applied to the ensemble means.
![Page 12: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/12.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
ACC for Malaria Incidence over Botswana (NDJF 1981-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. The dotted area display
significant correlations at the 95% confidence interval
![Page 13: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/13.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
MSSS for Malaria Incidence over Bostwana (NDJF 1981-2000). The reference dataset is a companion run forced by ERA40 reanalysis, and
the analysis is applied to the ensemble means.
![Page 14: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/14.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Validation climate drivers ENSEMBLES stream2 1981-2000
seasonal forecasts
![Page 15: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/15.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
RMSE for tasmax (NDJF 1981-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 16: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/16.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
ACC for tasmax (NDJF 1981-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 17: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/17.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
MSSS for tasmax (NDJF 1981-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 18: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/18.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
RMSE for prlr (NDJF 1981-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 19: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/19.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 NDJF:
ACC for prlr (NDJF 1981-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 20: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/20.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 2
Results 1981-2000 DJF:
MSSS for prlr (NDJF 1981-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 21: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/21.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Summary & Way Forward?
• Experience in BSS and ROCSS (probabilistic)
• Question non probabilistic nature/use of RMSE,
ACC and MSSS for EPS verification
• All data should come with sets of verification plots
e.g. ECMWF DEMETERhttp://www.ecmwf.int/research/demeter/d/charts/verification
• Library of scripts? Or portal to where they exsist
![Page 22: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/22.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Questions?
![Page 23: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/23.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
![Page 24: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/24.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
Validation climate drivers ENSEMBLES stream1 1991-2001
seasonal forecasts
also have for DEMETER equivalent years
![Page 25: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/25.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
RMSE for tas (DJF 1991-2000). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means.
![Page 26: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/26.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
RMSE for tas (DJF 1991-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 27: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/27.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
ACC for tas (DJF 1991-2000). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means.
![Page 28: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/28.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
ACC for tas (DJF 1991-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 29: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/29.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
MSSS for tas (DJF 1991-2000). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means.
![Page 30: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/30.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
MSSS for tas (DJF 1991-2000). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means.
![Page 31: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/31.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
RMSE for prlr (DJF 1991-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 32: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/32.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
ACC for prlr (DJF 1991-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 33: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/33.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 DJF:
MSSS for prlr (DJF 1981-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 34: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/34.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 JAS:
RMSE for prlr (JAS 1991-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 35: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/35.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 JAS:
ACC for prlr (JAS 1991-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 36: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/36.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse
ENSEMBLES –stream 1
Results 1991-2000 JAS:
MSSS for prlr (JAS 1991-2000). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means.
![Page 37: Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES](https://reader035.vdocuments.us/reader035/viewer/2022070418/568159a4550346895dc6ff2b/html5/thumbnails/37.jpg)
WGSIP12 – Miami - January 2009 – Impacts verification - Andy Morse