protocol for determining the best performing air quality model 8 th modeling conference september...

PROTOCOL FOR PROTOCOL FOR DETERMINING THE DETERMINING THE BEST PERFORMING BEST PERFORMING AIR QUALITY MODELAIR QUALITY MODEL

88THTH MODELING MODELING CONFERENCECONFERENCE

SEPTEMBER 22, 23 2005SEPTEMBER 22, 23 2005

MODEL PROTOCOLMODEL PROTOCOL

• THESE SLIDES ARE BASED ON AN THESE SLIDES ARE BASED ON AN EPA DOCUMENT – EPA-454/R-92-025 – EPA DOCUMENT – EPA-454/R-92-025 – ENTITLED:ENTITLED:

“ “PROTOCOL FOR DETERMINING THE PROTOCOL FOR DETERMINING THE BEST PERFORMING MODEL”BEST PERFORMING MODEL”

DECEMBER 1992DECEMBER 1992

• REFERENCED IN APPENDIX WREFERENCED IN APPENDIX W

OB = OBSERVED DATA: PR = PREDICTED DATA


• SCREENING TEST IS AVAILABLESCREENING TEST IS AVAILABLE– BASED ON FRACTIONAL BIAS (FB)BASED ON FRACTIONAL BIAS (FB)• FB = 2 [PR - OB] / [OB + PR]FB = 2 [PR - OB] / [OB + PR]

– BASED ON THE HIGHEST 25 VALUESBASED ON THE HIGHEST 25 VALUES– IF ABSOLUTE FB EXCEEDS 0.67 (FACTOR IF ABSOLUTE FB EXCEEDS 0.67 (FACTOR

OF 2), THE MODEL MAY BE EXCLUDEDOF 2), THE MODEL MAY BE EXCLUDED


• THE MORE ADVANCED STATISTICAL THE MORE ADVANCED STATISTICAL TEST HAS 2 COMPONENTSTEST HAS 2 COMPONENTS

– A REGULATORY COMPONENTA REGULATORY COMPONENT

– A SCIENTIFIC COMPONENTA SCIENTIFIC COMPONENT


• THE REGULATORY COMPONENTTHE REGULATORY COMPONENT

– THE PRECISE TIME, LOCATION AND THE PRECISE TIME, LOCATION AND METEOROLOGICAL CONDITIONS ARE METEOROLOGICAL CONDITIONS ARE NOT OF CONCERNNOT OF CONCERN

– THE MAGNITUDE OF THE HIGHEST THE MAGNITUDE OF THE HIGHEST CONCENTRATIONS IS OF PRIMARY CONCENTRATIONS IS OF PRIMARY CONCERNCONCERN


• THE REGULATORY COMPONENTTHE REGULATORY COMPONENT

– USES A TEST STATISTIC – THE ROBUST USES A TEST STATISTIC – THE ROBUST HIGHEST CONCENTRATION (RHC) - A HIGHEST CONCENTRATION (RHC) - A SMOOTHED ESTIMATE OF THE HIGHEST SMOOTHED ESTIMATE OF THE HIGHEST CONCENTRATIONSCONCENTRATIONS

– BASED ON A TAIL EXPONENTIAL FIT TO THE BASED ON A TAIL EXPONENTIAL FIT TO THE UPPER END OF THE DISTRIBUTIONUPPER END OF THE DISTRIBUTION

– GENERALLY BASED ON THE HIGHEST 25 GENERALLY BASED ON THE HIGHEST 25 CONCENTRATIONSCONCENTRATIONS

– CONDUCTED FOR REGULATORY AVERAGING CONDUCTED FOR REGULATORY AVERAGING TIME (E.G. 3- AND 24-HOUR) TIME (E.G. 3- AND 24-HOUR)

– USES A PERFORMANCE MEASURE - THE USES A PERFORMANCE MEASURE - THE FRACTIONAL BIAS OF THE RHCsFRACTIONAL BIAS OF THE RHCs


• THE SCIENTIFIC COMPONENTTHE SCIENTIFIC COMPONENT

– THE LOCATION AND METEOROLOGICAL THE LOCATION AND METEOROLOGICAL CONDITIONS ARE OF CONCERNCONDITIONS ARE OF CONCERN

– THE MAGNITUDE OF THE HIGHEST THE MAGNITUDE OF THE HIGHEST CONCENTRATIONS UNDER EACH CONCENTRATIONS UNDER EACH CONDITION IS OF CONCERNCONDITION IS OF CONCERN


• THE SCIENTIFIC COMPONENTTHE SCIENTIFIC COMPONENT

– USES A TEST STATISTIC – THE ROBUST HIGHEST USES A TEST STATISTIC – THE ROBUST HIGHEST CONCENTRATION (RHC) - A SMOOTHED ESTIMATE OF CONCENTRATION (RHC) - A SMOOTHED ESTIMATE OF THE HIGHEST CONCENTRATIONSTHE HIGHEST CONCENTRATIONS

– BASED ON A TAIL EXPONENTIAL FIT TO THE UPPER END BASED ON A TAIL EXPONENTIAL FIT TO THE UPPER END OF THE DISTRIBUTIONOF THE DISTRIBUTION

– GENERALLY BASED ON THE HIGHEST 25 GENERALLY BASED ON THE HIGHEST 25 CONCENTRATIONSCONCENTRATIONS

– USES A PERFORMANCE - THE FRACTIONAL BIAS OF THE USES A PERFORMANCE - THE FRACTIONAL BIAS OF THE RHCsRHCs

– CONDUCTED FOR CONDUCTED FOR 1 HOUR CONCENTRATIONS1 HOUR CONCENTRATIONS– CONDUCTED FOR 2 WIND SPEED GROUPS AND 3 CONDUCTED FOR 2 WIND SPEED GROUPS AND 3

STABILTIY CATEGORIESSTABILTIY CATEGORIES


• COMBINE THE SCIENTIFIC AND COMBINE THE SCIENTIFIC AND REGULATORY COMPONENTS INTO REGULATORY COMPONENTS INTO ONE METRICONE METRIC


• USES A COMPOSITE PERFORMANCE METRIC USES A COMPOSITE PERFORMANCE METRIC (CPM) (CPM)

CPM = 1/3 [AVERAGE AFBCPM = 1/3 [AVERAGE AFBI,J I,J ]] + +

2/3 [AFB2/3 [AFB33 + AFB + AFB2424 ] / 2 ] / 2

– AFBAFBI,J I,J = absolute fractional bias for met category i = absolute fractional bias for met category i at station jat station j

– AFBAFB3 3 = absolute fractional bias for 3 hour averages= absolute fractional bias for 3 hour averages– AFBAFB24 24 = absolute fractional bias for 24 hour = absolute fractional bias for 24 hour

averagesaverages


• MODEL TO MODEL COMPARISONMODEL TO MODEL COMPARISON

USES THE MODEL COMPARISON MEASURE USES THE MODEL COMPARISON MEASURE (MCM)(MCM)

MCMMCMA,BA,B = CPM = CPMAA - CPM - CPMBB

CPMCPMA A IS A COMPOSITE PERFORMANCE MEASURE IS A COMPOSITE PERFORMANCE MEASURE FOR MODEL AFOR MODEL A

CPMCPMB B IS A COMPOSITE PERFORMANCE MEASURE IS A COMPOSITE PERFORMANCE MEASURE FOR MODEL AFOR MODEL A


• THE YARDSTICKTHE YARDSTICK– THE RATIO OF THE 1) THE RATIO OF THE 1) COMPOSITE COMPOSITE

DIFFERENCEDIFFERENCE TO THE 2) TO THE 2) STANDARD STANDARD ERRORERROR

– BLOCKED BOOTSTRAP METHOD IS USED BLOCKED BOOTSTRAP METHOD IS USED TO GENERATE THE ESTIMATE OF THE TO GENERATE THE ESTIMATE OF THE STANDARD ERRORSTANDARD ERROR

– RATIOS > 1.7 INDICATE A SIGNIFICANT RATIOS > 1.7 INDICATE A SIGNIFICANT DIFFERENCE IN THE 2 MODELS AT DIFFERENCE IN THE 2 MODELS AT ABOUT 90% CONFIDENCE LEVELABOUT 90% CONFIDENCE LEVEL

protocol for determining the best performing air quality model 8 th modeling conference september...

Documents