regional production quarterly report on the daily analyses ... · the chimere production, but as it...

Regional ProductionQuarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances

December 2015 – January 2016 – February 2016

Copernicus Atmosphere Monitoring Service

Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances |

This document has been produced in the context of the Copernicus Atmosphere Monitoring

Service (CAMS). The activities leading to these results have been contracted by the

European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the

European Union (Delegation Agreement signed on 11/11/2014). All information in this

document is provided "as is" and no guarantee or warranty is given that the information is

fit for any particular purpose. The user thereof uses the information at its sole risk and

liability. For the avoidance of all doubts, the European Commission and the European Centre

for Medium-Range Weather Forecasts has no liability in respect of this document, which is

merely representing the authors view.



Quarterly report on the dailyanalyses and forecasts activities, and verification ofthe CHIMERE performances

December 2015 – January 2016 – February 2016

INERIS (F. Meleux, A. Ung, L. Rouïl)

METEO-FRANCE (M. Pithon, M. Plu, J. Parmentier,

J. Arteta, S. Guidotti, N. Assar)

Date: 04/05/2016

REF.:

CAMS50_2015SC1_D50.3.2.CHIMERE-2016Q1_201605




Contents:


1. Executive Summary.........................................................................................42. The CHIMERE model (INERIS)...........................................................................5Product portfolio..................................................................................................5Availability statistics.............................................................................................5

Use for observations for data assimilation...............................................................73. Verification report..........................................................................................10Verification of NRT forecasts................................................................................10Verification of NRT analyses................................................................................15

Analysis of CHIMERE performances for the quarter.................................................19


1. Executive Summary

The Copernicus Atmosphere Monitoring Service (CAMS, www.copernicus-atmosphere.eu) is establishing the core global and regional atmospheric environmental

service delivered as a component of Europe's Copernicus programme. The regionalforecasting service provides daily 4-days forecasts of the main air quality species andanalyses of the day before, from 7 state-of-the-art atmospheric chemistry models andfrom the median ensemble calculated from the 7 model forecasts. The regional service

also provides posteriori reanalyses using the latest validated observation datasetavailable for assimilation.

This report covers the deliverables related to Near Real Time Production (NRT) for

CHIMERE: D50.3.2-2016Q1, D50.3.4-2016Q1, D50.5.1-2016Q1, for the quarterDecember 2015 – January 2016 – February 2016. Verification is done against in-situsurface observations; they are described in the report D50.1.2-2016Q1, that will bedelivered shortly. The verification of analyses is done against non assimilated

observations.

A significant improvement was done during this quarter related to the robustness ofthe CHIMERE production, but as it has been set-up in January/February the impact on

the reliability of the CHIMERE production is not complete yet. Nevertheless, theavailability of CHIMERE files (forecasts and analyses) during this quarter is higher thanin the past.

Regarding the quality of the performances, the main point is that the sharp decrease ofthe CHIMERE ozone forecast quality of the last quarter is now over. Scores for allpollutants are now common and for most of them close to the ENSEMBLE values.

The CHIMERE analysis processing provides an overall improvement of the CHIMEREforecasts.

Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances | 4


2. The CHIMERE model (INERIS)

Product portfolio

Name Forecast Analysis

Description Forecast at surface, 50m,

250m, 500m, 1000m, 2000m,

3000m, 5000m above ground

Analysis at the surface

Available for users at 6:00 UTC 09:45 UTC for the day before

Species O3, NO2, CO, SO2, PM2.5, PM10,

NO, NH3, NMVOC, PANs,

Birch pollen at surface during

season

O3, NO2, CO, SO2, PM2.5, PM10,

NO

Time span 0-96h, hourly 0-24h for the day before,

hourly

Availability statistics

The statistics below describe the ratio of days for which the CHIMERE model outputswere available on time to be included in the ENSEMBLE fields (analyses and forecasts)

that are computed at METEO-FRANCE. They are based on the following schedule forthe provision at METEO-FRANCE of:

- forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from49h to 96h)

- analyses data: before 11:00 UTC

These schedules have been set to meet the IT requirements for ENSEMBLE products(no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses).

Indicators:

Availability_model_Forecast

Quarterly basis

D0: 99%

D1: 99%

D2: 88%

D3: 73%

Availability _model_Analysis

Quarterly basis

D: 97%



During this quarter, the following issues have been encountered by the CHIMERE

production system:

Date Problem description

(origin, effects)

Impact on production

From 01/12/2015 to

17/12/2015

and 21/12/2015

From 23/12/2015 to

25/12/2015 and from

27/12/2015 to 29/12/2015

Production delay due to a high

number of forecast on our

computing system. Since then,

this number has been

significantly reduced to meet

the deadline required for the

CHIMERE forecast delivery

Last days (D2-D3) forecast

results arrive too late for

participating to ENSEMBLE

calculation. ENSEMBLE with 6

members for D3 forecasts and

sometimes D2 forecasts

15/01/2016 Crash of the computing

system

No analysis and forecast result

available for ENSEMBLE

calculation

From 15/02/2016 to

16/02/2016

Production delay due to

instability of the computing

system

D3 forecast results arrive too

late to contribute to

ENSEMBLE

20/02/2016 Production delay due to

instability of the computing

system

D3 forecast results arrive too

late to contribute to

ENSEMBLE

From 23/02/16 to 24/02/2016 Analysis missing due to

unavailability of observations

No analysis result available for

ENSEMBLE calculation



Use for observations for data assimilation

Day

Use of observation for CHIMERE

December

O3 NO2 NO SO2 CO PM10 PM2.5

1 7,268 5,022

2 7,304 5,334

3 7,618 5,390

4 7,813 5,555

5 7,518 5,260

6 7,538 5,700

7 7,447 5,218

8 7,451 5,321

9 7,912 5,612

10 7,857 5,322

11 7,831 5,482

12 7,851 5,648

13 7,284 4,972

14 3,184 5,512

15 7,878 5,339

16 7,367 4,654

17 3,332 5,499

18 7,581 5,619

19 125 81

20 7,220 5,541

21 3,459 5,481

22 7,743 5,335

23 7,869 5,350

24 7,617 5,140

25 7,409 5,237

26 750 392

27 0 0

28 653 416

29 858 577

30 952 588

31 955 569

average 5,601 4,231



Day


January


1 1,400 986

2 1,686 1,179

3 262 170

4 3,453 2,903

5 5,213 2,662

6 7,502 5,151

7 1,361 776

8 6,739 4,337

9 1,359 775

10 4 0

11 2,360 1,435

12 5,493 2,464

13 8,040 5,128

14 7,964 5,217

15 3,342 1,843

16 2,320 1,490

17 2,239 1,408

18 8,105 5,368

19 8,120 5,403

20 8,268 5,419

21 8,190 5,536

22 8,173 5,501

23 8,056 5,267

24 7,831 5,219

25 8,212 5,425

26 4,018 2,778

27 7,774 5,216

28 8,055 5,358

29 8,181 5,389

30 7,770 5,192

31 7,930 5,114

average 5,465 3,552



Day


February


1 7,890 5,146

2 2,433 1,421

3 7,983 5,286

4 7,972 5,369

5 8,004 5,470

6 8,036 5,333

7 7,923 5,357

8 7,915 5,318

9 7,773 4,962

10 7,536 5,133

11 7,774 5,232

12 7,162 5,012

13 6,950 4,959

14 7,191 5,070

15 7,581 5,164

16 7,657 5,257

17 7,207 4,941

18 4,215 2,759

19 11 0

20 7,194 4,757

21 7,435 4,617

22 7,476 4,722

23 6,406 4,132

24 7,389 4,980

25 7,437 4,961

26 5,528 3,715

27 174 0

28 173 0

29 8 0

average 6,153 4,106



3. Verification report

This verification report covers the period December 2015 – January 2016 – February2016. The CHIMERE skill scores are successively presented for four pollutants: ozone,NO2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h(hourly values), allowing to evaluate the entire diurnal cycle and the evolution of

performance from day 0 to day 3. The forecasts and the analyses cover a largeEuropean domain (25°W-45°E, 30°N-70°N). The statistical scores that are reported arethe root-mean-square error, the modified mean bias and the correlation. The surfaceobservations that are acquired by METEO-FRANCE and used for verification are

described in D50.1.2-2016Q1, that will be delivered shortly.

Verification of NRT forecasts

The following figures present, for each pollutant (ozone, NO2, PM10, PM2.5):

- in the upper-left panel, the root-mean square error of daily maximum (for ozone and

NO2) or of daily mean (PM10) for the first-day forecasts with regards to surfaceobservations, for every quarter since DJF2014/2015, a target reference value isindicated as an orange line,

- in the upper-right panel, the root-mean square error of pollutant concentration

forecasts with regards to surface observations as a function of forecast term,

- in the lower-left panel, the modified mean bias of pollutant concentration forecastswith regards to surface observations as a function of forecast term,

- in the lower-right panel, the correlation of pollutant concentration forecasts with

regards to surface observations as a function of forecast term.

The graphics show the performance of CHIMERE (black curves) and of the ENSEMBLE(blue curves).



CHIMERE forecasts: ozone skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

Compared to previous quarters, CHIMERE recovers a RMSE close to the ENSEMBLE oneand below the KPI. If we take a closer look at the RMSE, it is noticeable that CHIMERE

has a higher RMSE than the ENSEMBLE, with a maximum difference of 7 µg/m3occurring in the morning and a minimum difference of 3 µg/m3 in the early afternoon,usually when O3 daily peak occurs. The daily time profile of the RMSE is similar forENSEMBLE and CHIMERE and stable along the forecast days. The profile for the bias

look very similar for ENSEMBLE and CHIMERE and consistent with the RMSE. CHIMEREoverestimates the observations more than the ENSEMBLE. CHIMERE correlation is veryclose to the ENSEMBLE correlation, and both depict a continuous decrease along theforecast time.



CHIMERE forecasts: NO2 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

As for the two previous quarters, CHIMERE has a RMSE close to the ENSEMBLE butwith a growing gap. The NO2 RMSE is still below the KPI value. Looking at the MMB, the

score spotlights that the underestimation of CHIMERE is higher than the ENSEMBLE.This result does not impact the RMSE and correlation of CHIMERE, which are similar tothe ENSEMBLE scores even if some differences may occur less than 2 µg/m3 for theRMSE and with a more pronounced gap just before noon, for the correlation. The time

varying evolution of the scores shows a decrease of the performance for RMSE andcorrelation and stability for MMB.



CHIMERE forecasts: PM10 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

CHIMERE depicts a similar RMSE than the ENSEMBLE for the PM10 daily meanconcentrations over the 4 last quarters. The MMB shows a higher underestimation in

CHIMERE than in the ENSEMBLE. The CHIMERE RMSE is also slightly above theENSEMBLE RMSE, and the temporal evolution shows a slight increase of the score forboth. The CHIMERE correlation is 0.1 below the ENSEMBLE one, and both have adecrease with the same magnitude along the forecast time.



CHIMERE forecasts: PM2.5 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

CHIMERE has a similar RMSE than the ENSEMBLE for the PM2.5 daily meanconcentrations over the 4 last quarters. The MMB shows a higher underestimation in

CHIMERE than in the ENSEMBLE. The CHIMERE RMSE is also slightly above theENSEMBLE RMSE. The CHIMERE correlation is slightly below the ENSEMBLE one and ithas a faster decrease of the scores along the forecast time. The daily time profiles arevery similar for both forecasts. The results of the PM2.5 are similar to the PM10 results

with better values for all scores.



Verification of NRT analyses

The following figures present, for each pollutant (ozone, NO2, PM10):

- in the upper-left panel, the root-mean square error of daily maximum (for ozone andNO2) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts(dashed line) with regards to surface observations, for every quarter since

DJF2014/2015, a target reference value is indicated as an orange line,

- in the upper-right panel, the root-mean square error of pollutant concentration of theanalyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,

- in the lower-left panel, the modified mean bias of pollutant concentration forecasts ofthe analyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,

- in the lower-right panel, the correlation of pollutant concentration of the analyses

(solid line) and of the first-day forecasts (dashed line), with regards to surfaceobservations as a function of forecast term.

The graphics show the performances of CHIMERE (black curves) and of the ENSEMBLE(blue curves). The superposition of the analysis scores (solid lines) and of the forecast

scores (dashed lines) computed over the same observation dataset is helpful to assessthe added value of data assimilation.



CHIMERE analyses: ozone skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

CHIMERE analysis has RMSE for this quarter, with similar values than for theENSEMBLE with a slight improvement compared to the previous quarter. Regarding the

other scores, it is worth noting that the assimilation reduces significantly the gapbetween CHIMERE and the ENSEMBLE compared to the scores established from theraw outputs. The assimilation process leads to a significant improvement of the RMSEwhich is better than the ENSEMBLE RMSE, the MMBs are very close whereas the

correlation of the CHIMERE is better than the ENSEMBLE correlation.



CHIMERE analyses: NO2 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

No NO2 assimilation in CHIMERE yet. It is foreseen very soon.



CHIMERE analyses: PM10 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The impact of the assimilation stage provides positive effects on the quality of therepresentation of the PM10 surface concentrations. It is worth noting that the

performances of the analyses are less good at the end of the day, maybe due to adecrease of observations availability.

The MMB shows that the assimilation process provides overestimation of the PM10

surface concentrations whereas the CHIMERE forecasts tend to underestimate.Compared to the previous quarter the RMSE calculated from the daily meanconcentrations shows a stability of the performances.



Analysis of CHIMERE performances for the quarter

The main point of the CHIMERE performance for this quarter is the ozone productionwhich is back to a level close to the ENSEMBLE. Nevertheless, the CHIMERE scores forozone show up differences which indicates that CHIMERE is still perfectible for this

period of the year when ozone concentrations are usually at their lowest level.

For NO2 and PMs, CHIMERE behaviour over this quarter is very close to the ENSEMBLEeven if most of the time the CHIMERE performance are lower that the ENSEMBLE

performances. Still for NO2 and PM10, the results are in good agreement with the lastyear results. The PM2.5 scores show similar temporal variabilities that the PM10, but withbetter scores.

The assimilation process improved significantly the O3 forecasts, and reduced the gapbetween CHIMERE and the ENSEMBLE. For ozone, it is worth noting that CHIMEREanalyses depict scores which are better that the ENSEMBLE.

The PM10 analysis also leads to a significant improvement of the PM10 surfaceconcentration, compared to the CHIMERE raw outputs.


regional production quarterly report on the daily analyses ... · the chimere production, but as it...

Documents