CMAS Conference, October 6 – 8, 2008

The work presented in this paper was performed by the New York State Department of Environmental Conservation with partial support from the U.S. EPA under cooperative agreement CR83228001 and the New York State Energy Research and Development Authority (NYSERDA) under agreement #10599. The views expressed in this paper do not necessarily reflect the views or policies of the New York State Department of Environmental Conservation or those of the sponsoring agencies.

Multi-Model Air Quality Forecasting Over New York State For Summer 2008

Christian Hogrefe1,2,*, Prakash Doraiswamy2, Winston Hao1, Brian Colle3, Mark Beauharnois2, Ken Demerjian2, Jia-Yeong

Ku1, and Gopal Sistla1

1New York State Department of Environmental Conservation, Albany, NY2Atmospheric Sciences Research Center, State University of New York at Albany, Albany, NY3School of Marine and Atmospheric Sciences, Stony Brook University, Stony Brook, NY

Ensemble Forecasting• Consists of using multiple models, multiple configurations of

the same model, multiple initial/boundary conditions, multiple model inputs, or a combination thereof

• Aims at minimizing the effects of uncertainties in model inputs and model parameterizations on model predictions

– In weather forecasting, ensemble-mean forecasts are often found to provide higher accuracy than forecasts from individual models

– Initial applications for air quality forecast studies (McKeen et al., JGR, 2005; Pagowski et al., GRL, 2005) also show promise but were limited in temporal scope

• In addition, ensemble systems can be used to provide probabilistic forecasts of threshold exceedances

CMAQ-Based Air Quality Forecasting at NYSDEC

• Since 2004: CMAQ “next day” air quality forecast driven by 12z NCEP WRF-NMM (ETA-NAM prior to June 2006) (12 km)

• Since April 2008: additional CMAQ “same-day” air quality forecast driven by 00z NCEP WRF-NMM (12 km)

• Since June 2008: two additional CMAQ “same day” air quality forecast driven by two selected members of the SUNY-Stony Brook (SUNY-SB) 00z Short-Range Ensemble Forecast System (SREF) (nested 36km/12 km)

– SUNY SB SREF: consists of a total of 14 MM5 or WRF weather forecasts different by their initial conditions and physics options

– Two members were selected to drive daily CMAQ simulations based on temperature and wind verification results for summer 2007 and operational considerations

– Two of the 14 SREF members use the Ferrier microphysics scheme that is currently not compatible with CMAQ

• June 1 – July 22, 2008: retrospective case study to perform twelve CMAQ “same day” air quality forecasts driven by all compatible members of the SUNY-SB 00z SREF (nested 36km/12 km)

• While there are differences in emission inventories, projection parameters, and horizontal and vertical grid setup, the horizontal grid spacing over New York State is 12 km in all simulations

Air Quality Forecast Regions in NYS• Model-based forecast guidance is issued

and evaluated following the same region-based approach used for the official human-based air quality forecasts issued by NYSDEC

• Each forecast region contains one or more ozone monitor and one or more continuous PM2.5 monitor

• For a given region and day, the forecasted/observed air quality value for ozone (PM2.5) is defined as the maximum ozone (PM2.5) value at any ozone (PM2.5) monitor in that region

• Model values are extracted for the locations of all monitors and the model air quality value for ozone (PM2.5) is defined in the same way as for the observations

The Air Quality Index (AQI) Used by NYSDEC

• Non-dimensional index to communicate air quality forecasts to the public

• Concentrations of ozone and PM2.5 are converted to AQI through piecewise linear functions

• Some PM2.5 thresholds lower than those used in AIRNOW

AQI Level of Health Concern

Daily Maximum 8-hr O3 Concentration (ppb)

24-hr Average PM2.5 Concentration (μg/m3)

0-50 Good 0-59 0-15.4

51-100 Moderate 60-75 15.5-35.4

101-150 Unhealthy for sensitive groups

76-95 35.5-65.4

151-200 Unhealthy 96-115 65.5-150.4

201-300 Very Unhealthy 116-374 150.5-250.4

Discrete and Categorical Evaluation of Ozone and PM2.5 Predictions From the Four Member CMAQ Forecast Guidance System

Summer 2008 (June 4 – August 31)

Daily Maximum 8-hr Ozone June 4 –August 31 for NYS Forecast Regions 1-8

Red: observations

Blue: 4-Model Average

Grey: 4-Model Range (minimum to maximum)

• Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median

• The four member mean or median forecasts often but not always have lower RMSE than individual forecasts

RMSE of predicted 8-hr daily maximum ozone concentrations June 4 – August 31: Individual Models, Ensemble Average, and Ensemble Median

• Left to Right: Observations, 4-Model Average, NCEP/CMAQ 12z, NCEP/CMAQ 00z, SUNY-SB F2 / CMAQ 00z, and SUNY-SB F9 / CMAQ 00z

Relative frequency of AQI categories for 8-hr daily maximum ozone June 4 – August 31: observations, Ensemble Average, and Individual Models

• CSI = correct exceedance forecasts / (correct exceedance forecasts + false alarms + missed exceedance forecasts); range 0 (no skill) to 1

• Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median

Critical Success Index (CSI) for categorical forecasts of an ozone threshold of 75 ppb for predicted 8-hr daily maximum ozone concentrations June 4 – August 31

Daily Average PM2.5 June 4 –August 31 for NYS Forecast Regions 1-8

Red: observations

Blue: 4-Model Average

Grey: 4-Model Range (minimum to maximum)

Bias of predicted 24-hr average PM2.5 concentrations June 4 – August 31: Individual Models, Ensemble Average, and Ensemble Median

• Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median

• For all regions except NYC (region 2), PM2.5 predictions tend to be biased low

Discrete, Categorical, and Probabilistic Evaluation of Ozone and PM2.5 Predictions From the Retrospective Case Study With a Twelve Member CMAQ Forecast Guidance System

Case Study June 4 – July 21 (period of highest O3 during summer 2008 in New York State)

How much variability in meteorological and air quality variables is introduced by using the 12 meteorological SREF members to drive CMAQ?

Calculate coefficient of variation (standard deviation across models / mean across models); averaged June 4 – July 21

Daily Average Wind Daily Maximum PBLDaily Maximum Temperature

Daily Maximum 8-hr O3 Daily Average PM2.5 Variations in meteorology introduced by the twelve SREF members cause a typical ozone variability of 5 – 10% (with higher values in urban areas near land/sea interfaces) and a typical PM2.5 variability of 20 – 25%

Daily Maximum 8-hr Ozone June 4 – August 31 for NYS Forecast Regions 1 (Long Island) and 2 (NYC Metro)

Red: observations

Blue: 12-Model Average

Grey: 12-Model Range (minimum to maximum) (Increased compared to 4-member results)

• The twelve member mean or median forecasts often but not always have better performance statistics than individual forecasts

RMSE, correlation coefficient, and CSI of predicted 8-hr daily maximum ozone June 4 – July 21: Individual Models, Ensemble Average, and Ensemble Median

Example of Using the Ensemble to Predict Exceedance Probabilities: Predicted Probability of Daily Maximum 8-hr O3 > 75 ppb for June 10, 2008

0 out of 12 ensemble members predicting O3 > 75 ppb probability = 0%

1 out of 12 ensemble members predicting O3 > 75 ppb probability = 8.33%

12 out of 12 ensemble members predicting O3 > 75 ppb probability = 100%

Observed AQI

Given that the predicted exceedance probability was xx%, what was the observed exceedance probability? (i.e. if there were 10 days where 3-4 models predicted an exceedance, on how many of these days was there an observed exceedance?)

For a good probabilistic forecast system, the points fall close to the 1:1 line

Example Probabilistic Evaluation of Ensemble Predictions of 8-hr Daily Maximum O3 Exceedances For Regions 1 and 2

Caveat: sample size in this example is very small (48 days, ~10 observed exceedances) for this type of analysis

• Each day, rank-order the 12 forecasts

• Depending on whether the observations are lower than the lowest forecast, fall between the lowest and second-lowest forecast, … are higher than the highest forecast, assign that day to one of 13 bins

• Repeat the analysis for all days and create a histogram based on the number of days assigned to each of the 13 bins

•Ideal shape – flat

•U-shape (inverted U-shape): ensemble is underdispersed (overdispersed)

•L-shape – ensemble is biased

Talagrand Diagram (Rank Histogram) for 8-hr Daily Maximum O3

The ensemble predictions are underdispersed and often biased

Talagrand Diagram (Rank Histogram) for 24-hr Average PM2.5

Summary and Next Steps• Over NYS, the summer of 2008 was characterized by

several ozone episodes from early June through late July and relatively low ozone thereafter

• As measured by discrete and categorical metrics, the four model system utilized since June 1, 2008 appears to have provided good ozone forecast guidance for the summer 2008, especially for Long Island and the NYC metro area

• As reported in earlier studies, forecasts for summertime PM2.5 are characterized by a negative bias for all areas except the NYC metro area the upcoming CMAQ release with updated SOA module may help (or not …)

• The four member mean or median forecasts often but not always have better statistics than individual forecasts weighting or bias correction approaches prior to averaging may provide improved forecasts

Summary and Next Steps• For a 48 day retrospective case study, CMAQ simulations were

performed using twelve SUNY-SB SREF weather forecasts• As for the four member system, the twelve member mean or median

forecasts also often but not always have better statistics than individual forecasts

• Variations in meteorology introduced by the twelve SREF members cause a typical ozone variability of 5 – 10% (with higher values in urban areas near land/sea interfaces) and a typical PM2.5 variability of 20 – 25%

• The 12-member system provides a realistic spread of ozone concentrations for regions 1 and 2 but tends to be underdispersed for other regions and for PM2.5, indicating that uncertainties from other factors such as emissions or chemistry need to be included for a better representation of uncertainty

• Next steps:– Increase the number of daily forecast members (add CAMx)– Perform retrospective 12-member case studies for additional seasons– Investigate and implement weighting and bias correction approaches