Calculating Statistics: Concentration Related
Performance Goals
James W. BoylanGeorgia Department of Natural Resources
PM Model Performance WorkshopChapel Hill, NC
February 11, 2004
Outline
• Performance Statistic– Standard Bias and Error Calculations
• Model Performance Goals for PM– Speciated Bias and Error Goals– Relative Proportions Goals
Performance Metrics Equation
Mean Bias (g/m3)
Mean Error (g/m3)
Mean Normalized Bias (%) (-100% to +) Mean Normalized Error (%) (0% to +)
Normalized Mean Bias (%) (-100% to +) Normalized Mean Error (%) (0% to +)
Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)
N
iom CC
NMB
1
1
N
i mo
om
CC
CC
NMFE
1
2
1
N
i mo
om
CCCC
NMFB
1
2
1
N
io
N
iom
C
CCNME
1
1
N
io
N
iom
C
CCNMB
1
1
N
i o
om
C
CC
NMNE
1
1
N
i o
om
C
CC
NMNB
1
1
N
iom CC
NME
1
1
Example
GT showed a positive bias of 11 points
NB = 14.3%FB = 13.3%
North Carolina 77Georgia Tech 88
Performance Metrics
• Mean Normalized Bias and Error – Usually associated with observation-based minimum
threshold• Some components of PM can be very small making it
difficult to set a reasonable minimum threshold value without excluding a majority of the data points
– Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small
• A few data points can dominate the metric
– Overestimations are weighted more than equivalent underestimations
Performance Metrics
• Normalized Mean Bias and Error – Biased towards overestimations
• Mean Fractional Bias and Error– Bounds maximum bias and error – Gives additional weight to underestimations and
less weight to overestimations
Example Calculations
• Mean Normalized Bias and Error– Most biased and least useful of the three metrics
• Normalized Mean Bias and Error • Mean Fractional Bias and Error
– Least biased and most useful of the three metrics
Model g/m3
)
Obs. g/m3
)
MBg/m3)
NMB (%)
MNB (%)
MFB (%)
ME g/m3
)
NME (%)
MNE (%)
MFE (%)
0.05 1.0 -0.95 -95 -180.95 +0.95 +95 +180.95
1.0 0.05 +0.95 +1900 +180.95 +0.95 +1900 +180.95
1.0 0.01 +0.99 +9900 +196.04 +0.99 +9900 +196.04
0.683 0.353 +0.33 +93.4 +3901.7 +65.3 0.96 272.9 3965.0 186.0
SAMI Model Performance Summary
Species
# Obs
Mean
g/m3
MB g/m
3
NMB (%)
MNB (%)
MFB (%)
ME g/m
3
NME (%)
MNE (%)
MFE (%)
SO4 134 6.71 -1.16 -17.3 1.1 -22.7 2.48 37.0 55.1 50.2
NO3 134 0.63 -0.30 -47.6 6.8 -73.6 0.52 81.8 112.8 107.2
NH4 134 2.70 -1.25 -46.4 -27.4 -57.4 1.43 53.1 61.6 70.0
NH4 Bi
134 1.44 0.01 0.4 34.2 -2.6 0.62 42.9 70.4 44.4
ORG 132 3.41 -0.27 -7.8 15.8 -6.0 1.37 40.4 53.8 43.9
EC 132 0.56 -0.05 -8.6 15.1 -12.7 0.27 48.3 61.9 50.4
Soils 135 0.55 0.25 46.2 171.6 21.9 0.57 102.9 207.4 72.5
PM2.5 130 17.05 -4.79 -28.1 -9.1 -28.8 6.8 39.8 48.9 47.6
PM10 130 23.44 -5.21 -22.2 -6.2 -21.0 9.18 39.1 44.2 43.5
PMC 126 6.98 -0.48 -6.9 43.9 7.8 3.86 55.2 78.7 54.1
bext 132 133.1 -27.91 -21.0 -10.2 -23.7 43.70 32.8 40.0 40.4
Proposed Performance Goals• Based on Mean Fractional Error (MFE) and Mean
Fractional Bias (MFB) calculations• Performance goals should vary as a function of
species concentrations– More abundant species should have a MFE +50%
and MFB ±30%– Less abundant species should have less stringent
performance goals
• Goals should be continuous functions with the features of:– Asymptotically approaching +50% MFE and ±30%
MFB when the concentrations (mean of the observed and modeled concentrations) are greater than 2.5 g/m3
– Approaching +200% MFE and ±200% MFB when the concentrations (mean of the observed and modeled concentrations) are extremely small
Proposed Mean Fractional Error and Bias Goals
301703/5.0
)(5.0
mg
CC mo
eMFB
501503/75.0
)(5.0
mg
CC mo
eMFE
Example Calculations
Species X Model g/m3)
Obs. g/m3)
FB (%) FE (%)
Day 1 – Site A 2.0 1.0 +66.7 +66.7
Day 1 – Site B 1.0 2.0 -66.7 +66.7
Day 2 – Site A 1.0 0.4 +85.7 +85.7
Day 2 – Site B 0.5 1.5 -100.0% +100.0%
Average 1.125 1.225 -3.6% 79.8%
Average CO + CM = 0.5*(1.125 + 1.225) = 1.175
MFE performance goal for “Species X” = 81.3%MFB performance goal for “Species X” = ±46.2%
Mean Fractional Error Goal
Speciated Fine PM Performance
0
50
100
150
200
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Mean Fractional Bias Goal
Speciated Fine PM Performance
-200
-100
0
100
200
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
SAMI – 6 Episodes
Speciated Fine PM Performance
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Ammonium (Sulfate)
Nitrate
Soils
Elem. Carbon
Organics PMC PM2.5 PM10
Ammonium (Bisulfate) Sulfate
SAMI – 6 Episodes
Speciated Fine PM Performance
-200
-100
0
100
200
0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Elem. Carbon
PMC
Organics
Nitrate
Soils
Ammonium (Sulfate)
PM2.5 PM10Ammonium (Bisulfate)
Sulfate
VISTAS – July 1999 Episode
Speciated Fine PM Performance
0
50
100
150
200
0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Ammonium (Sulfate)
Nitrate
Soils
Elem. Carbon
Organics
PMCPM2.5 PM10
Sulfate
VISTAS – July 1999 Episode
Speciated Fine PM Performance
-200
-100
0
100
200
0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Elem. Carbon
PMC
OrganicsNitrate
Soils
Ammonium (Sulfate)
PM2.5 PM10
Sulfate
VISTAS – January 2002 Episode
Speciated Fine PM Performance
0
50
100
150
200
0.0 2.0 4.0 6.0 8.0 10.0 12.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Err
or
Ammonium (Sulfate)
Nitrate
Soils
Elem. Carbon
Organics
PMC PM2.5 PM10
Sulfate
VISTAS – January 2002 Episode
Speciated Fine PM Performance
-200
-100
0
100
200
0.0 2.0 4.0 6.0 8.0 10.0 12.0Average Concentration (g/m3)
Me
an
Fra
cti
on
al
Bia
s
Elem. Carbon
PMC
Organics
Nitrate
Soils
Ammonium (Sulfate)
PM2.5 PM10
Sulfate
Relative Proportions (RP) PERF Goals
• EPA draft guidance (2001)– “For major components (i.e., those observed to
comprise at least 30% of measured PM2.5), we propose that the relative proportion predicted for each component averaged over modeled days with monitored data agrees within about 20% of the averaged observed proportion. For minor observed components of PM, we suggest a goal that the observed and modeled absolute proportion of each minor component agree within 5%.”
N
i
oN
i m
m
Total
component
Total
component
Co
C
NC
C
NBias
11
11 0.2 RP%(5%)
Example Calculation
• Calculating component proportions based on concentrations averaged over multiple days can hide poor model performance
Observed RP (%) Modeled RF (%)
Day 1 50% 95%
Day 2 50% 95%
Day 3 50% 5%
Day 4 50% 5%
Average 50% 50%
Observed Simulated
26.7%
47.1%12.3%
2.8%
4.3%6.8%
25.6%
50.5%
10.8%
4.7%
4.2% 4.1% SO4
ORG
NH4
NO3
EC
Soils
Relative Proportions for SAMI
Relative Proportions for SAMI
Relative Proportions Evaluation
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60
Relative Proportion of Fine PM (%)
Bia
s (
%)
Sulfate
Ammonium (Bisulfate)
Organics
Nitrate
Elem. Carbon
Soils
Proposed Relative Proportions Performance Goals
• Propose to use an equation that accounts for the day-to-day variability of species relative proportions:
RP 30%, Error 10%RP 15%, Error 5%RP 15% - 30%, Error [RP]/3
N
i o
o
m
m
Total
component
Total
component
C
C
C
C
NError
1
1
Proposed Relative Proportions Performance Goals
Relative Proportions Evaluation
0
5
10
15
0 10 20 30 40 50 60
Relative Proportion of Fine PM (%)
Err
or
(%)
Sulfate
Ammonium (Bisulfate)
OrganicsNitrate
Elem. Carbon
Soils
Concluding Remarks
• Recommended performance values are model goals, not model criteria– Failure to meet proposed performance goals
should not prohibit the modeling from being used for regulatory purposes• Help identify areas that can be improved upon
in future modeling
• If performing episodic modeling, performance evaluation should be done on an episode-by-episode basis
• If performing annual modeling, performance evaluation should be done on a month-by-month basis
Concluding Remarks (cont.)
• As models mature, performance goals can be made more restrictive by simply: – Adjusting the coefficients in the MFE and MFB
goal equations – Lowering the relative proportion error goals
• Q: Is there a need for performance goals for gaseous precursors or wet deposition species?– “One-atmosphere” modeling system– If not, still should be evaluated to help identify
potential problems with PM model performance