1 quality control review of e3 calculator inputs comparison to deer database brian horii energy and...

1

Quality Control Review of E3 Calculator Inputs

Comparison to DEER Database

Brian HoriiEnergy and Environmental Economics, Inc.

November 16, 2006

2

Overview

• Purpose was to review how well data entered into the E3 Calculator matches the DEER database.

• A review of all IOU submissions reveals that very few measures actually state the DEER Run ID or Measure ID

• The review also shows many measures that were entered, but had no planned installations.– 9,236 total measures for all IOUs

• 6,136 with some installations – 2,424 RunID matches

• 1,304 with installations – 155 Measure ID matches

• 151 with installations

• The lifecycle and peak impacts of these measures are shown in the next slides

3

Lifecycle Gross kWh

Small share of measures have DEER identifiers in E3 Calculator.

Values not adjusted for NTG ratio.Values use entered EUL

-

5,000,000,000

10,000,000,000

15,000,000,000

20,000,000,000

25,000,000,000

30,000,000,000

35,000,000,000

40,000,000,000

45,000,000,000

50,000,000,000

PG&E SCE SDG&E SoCalGas

NoneMeasureIDRunID

Sum of Lifecycle kWh

Utility

Match

Sum of Lifecycle kWh MatchUtility RunID MeasureID None Grand TotalPG&E 6,951,239,701 647,313,460 25,474,360,783 33,072,913,944 SCE 6,780,090,380 82,666,312 39,296,589,058 46,159,345,750 SDG&E 5,470,817,260 7,122,941,647 12,593,758,907 SoCalGas 1,995,236,833 2,040,297,660 4,035,534,494 Grand Total 21,197,384,175 729,979,772 73,934,189,149 95,861,553,096

4

Gross Peak Reduction

-

200,000

400,000

600,000

800,000

1,000,000

1,200,000


None

MeasureID

RunID

Sum of kW

Utility

Match

Sum of kW MatchUtility RunID MeasureID None Grand TotalPG&E 88,101 29,727 339,271 457,099 SCE 339,493 847 712,279 1,052,619 SDG&E 78,395 188,179 266,575 SoCalGas 29,823 63,789 93,612 Grand Total 535,812 30,574 1,303,519 1,869,905

5

Lifecycle Gross Therms

-

200,000,000

400,000,000

600,000,000

800,000,000

1,000,000,000

1,200,000,000

1,400,000,000


NoneMeasureIDRunID

Sum of Lifecycle Th

Utility

Match

Sum of Lifecycle Th MatchUtility RunID MeasureID None Grand TotalPG&E 6,380,284 9,381,605 743,225,988 758,987,877 SCE - - - - SDG&E 38,519,192 128,480,814 167,000,005 SoCalGas 136,486,689 1,066,350,907 1,202,837,596 Grand Total 181,386,165 9,381,605 1,938,057,709 2,128,825,478

6

Run ID Matching• For the subset that had Run ID inputs, we compared

how the entries match the DEER database.• Several criteria for “matches” were used. All matches

were to only 2 or 3 significant digits to allow for rounding

1. Matches for either DEER common or code impacts AND either Incremental or Full costs.

2. Matches that ignore a commodity (G or E) if no savings claimed

3. Matches on ratios of Impacts to Costs. Either Common or Code can match. Choice of Incremental or Full costs for denominator based on DEER Application and CostBasis. – Full costs used with Common impacts 86% of the time. – Incr costs used with Code impacts 69% of the time.

4. Matches if entered ratios are lower than DEER ratios.– Note that these methods will NOT match cases where

direct install costs are excluded from IMC and put into Admin.

7

RunID Details

• 994 measures match DEER• 1,070 match if entry ignored when no savings claimed• 1,077 measures match for Impact/Cost ratios • 1,893 match if all impact/cost ratios LOWER than DEER

are deemed OK• 244 measures do not match under any test.

– 41 measures have negative peak impacts that were not entered (3.4 MW)

– 244 measures have cost and impact units that do not match. Of those, 149 passed one of the matching tests, we did not perform any unit reconciliation tests.

• Impact of changing inputs for the 244 non-matches is shown on the next table. (i.e.: how different are those inputs?)

8

Run ID Subset• Impacts for the measures with a DEER Run ID that

“match” the DEER database, versus the non-matches are shown in the top half of the table.

• The bottom half of the table shows the change in impacts if we modified the inputs for the non-match cases to match the DEER database.

• There is little effect on the kWh forecast, but significant impact on kW, and less so on Therms.

Lifecycle kWh Peak kW Lifecycle ThermsMatches 14,653,969,784 210,426 84,949,550 Non-Matches 6,543,414,391 325,385 96,436,615 Percentage unmatched 31% 61% 53%Corrections to match DEER 204,740,481 (223,816) (37,000,477) Amount of difference 1% -42% -20%

9

kWh Detail for Measures w/ RunIDs

• Match includes measures where entered ratios are less than DEER ratios.• Bottom table shows the change in kWh needed to match the DEER ratios.• Negative value indicates that entered values have larger impacts than

DEER

Sum of Lifecycle kWh MatchUtility Match Non-Match Grand Total % matchPG&E 3,101,697,390 3,849,542,312 6,951,239,701 45%SCE 5,025,100,298 1,754,990,082 6,780,090,380 74%SDG&E 4,792,326,968 678,490,292 5,470,817,260 88%SoCalGas 1,734,845,128 260,391,706 1,995,236,833 87%Grand Total 14,653,969,784 6,543,414,391 21,197,384,175 69%

Sum of Correction - kWh MatchUtility Match Non-Match Grand Total % of TotalPG&E 986,237 656,823,274 657,809,512 9%SCE - (251,340,665) (251,340,665) -4%SDG&E 14,139,105 (234,625,268) (220,486,163) -4%SoCalGas 115,826,249 (97,068,452) 18,757,797 1%Grand Total 130,951,591 73,788,890 204,740,481 1%

10

kW detail

• SCE overestimates are largely (if not entirely) due to “pasting error” mentioned at the prior QA/QC workshop.

Sum of Total kW MatchUtility Match Non-Match Grand Total % matchPG&E 34,800 53,301 88,101 39%SCE 83,231 256,262 339,493 25%SDG&E 66,087 12,308 78,395 84%SoCalGas 26,309 3,514 29,823 88%Grand Total 210,426 325,385 535,812 39%

Sum of Correction kW MatchUtility Match Non-Match Grand Total % of TotalPG&E 22 (9,703) (9,681) -11%SCE - (215,944) (215,944) -64%SDG&E 348 (896) (548) -1%SoCalGas 2,828 (470) 2,358 8%Grand Total 3,198 (227,013) (223,816) -42%

11

Lifecycle Therm Detail

• Note that negative correction values for Sempra “matches” are due to cases where the measure had a negative therm reduction, but no therm impacts were claimed in the input section.

Sum of LifecycleTherms MatchUtility Match Non-Match Grand Total % matchPG&E 1,666,601 4,713,683 6,380,284 26%SCE - - - SDG&E 15,072,247 23,446,945 38,519,192 39%SoCalGas 68,210,702 68,275,987 136,486,689 50%Grand Total 84,949,550 96,436,615 181,386,165 47%

Sum of Correction Th MatchUtility Match Non-Match Grand Total % of TotalPG&E 325,502 (595,398) (269,895) -4%SCE 10,162 100,239 110,401 SDG&E (2,246) (5,175,031) (5,177,277) -13%SoCalGas (831) (31,662,876) (31,663,706) -23%Grand Total 332,587 (37,333,065) (37,000,477) -20%

12

Other Run ID Findings• kW (Impact of matching to DEER shown in parentheses)

– 104 cases where entered kW significantly above DEER. (228 MW) – 2 cases where DEER Watts entered as kW, plus 4 other conversion

errors. (0.4MW)– 6 additional cases where negative DEER impact is entered as positive

(0.7 MW)– 41 cases where negative kW impacts are not entered. Of those, 30

have no installations. (3.2 MW)• Therms

– 5 cases where DEER values are Therms, not kBTU (24.4 MTh)– 4 cases where impact and cost units differ despite DEER indicating

“same” (12.1 MTh).• Other findings

– 33 case where measure is RETROFIT, but base for impacts is CODE, not COMMON. This is conservative, and probably not an error, but highlights that users could use the DEER database to arrive at very different results by mixing the two sets of inputs and costs.

13

Review of Measures with No Run ID

14

Process

• Develop avg & max ratios of impacts per $GIMC– Utility submissions that match DEER RunID, by

measure end use categories (based on end use shapes in the E3 calculator)

– DEER measures by 60 subcategories.

• Map Measures– Match measure end use shapes.– Manually map 1739 cases.

• Compare entered ratios to maximums from step 1

15

Ratio test overview• 6916 measures did not have RunID matches in the input data• 4681 have some installations• 317 have no GIMC• Initial filtering using DEER ratios

– 579 measures with kWh ratios in excess of the DEER sample’s maximum• But if average ratios are used, the utility submissions are very conservative in

aggregate for kWh and kW.– 419 measures with Therm ratios in excess of the DEER sample’s maximum

• The therm “overestimate” would be slightly higher in aggregate if the average ratios were used for all measures.

• Problems with filtering analysis– Impact and cost unit mismatches make comparisons difficult– 53,014 DEER runs have different units (out of about 120K)– Results shown exclude all DEER runs where units are not the same– Mapping of utility measures to categories is imprecise– Aggregation into categories is imprecise

16

Impact Ratios by End Use

• Average and Maximum kWh ratios

-

20

40

60

80

100

120Li

ght

Ag

Ref

rig -

Non

-Res

Ref

rig

Poo

l

Oth

er

Indu

stria

l

Coo

ling

Foo

d S

ervi

ce

Ext

Lig

ht

Res

AC

Mot

or

DH

W

Spa

ceC

oolH

eat

Dry

er

Clo

thes

Coo

king

kWh

per

$ G

MIC

17

• WattsMax &Average

• kBTUMax &Average

-

2

4

6

8

10

12

14

Ligh

t

Ag

Coo

ling

Res

AC

Oth

er

Poo

l

Ref

rig

Indu

stria

l

Mot

or

Spa

ceC

oolH

eat

Ref

rig -

Non

-R

es DH

W

Ext

Lig

ht

Clo

thes

Dry

er

Coo

king

Foo

d S

ervi

ce

Wat

ts p

er $

GIM

C

-200

20406080

100120140160180

Sp

ace

Co

olH

ea

t

Re

s A

C

Re

frig

DH

W

Co

oki

ng

Dry

er

Clo

the

s

Lig

ht

Co

olin

g

Ag

Ext

Lig

ht

Fo

od

Se

rvic

e

Ind

ust

ria

l

Mo

tor

Oth

er

Po

ol

Re

frig

- N

on

-R

es

kB

TU

pe

r $

GIM

C

18

Results based on DEER Extract• Based on reductions relative to CODE.

• Similar results if larger of CODE or COMMON is used

Lifecycle kWh kW Lifecycle ThermsTotal for Subset 84,770,701,645 1,517,217 2,365,793,158 No GIMC 1,939,316,722 89,317 208,425,769 Amount Exceeding Max Ratios 2,451,725,955 155,778 1,272,910,735

Amount Exceeding Average Ratios (69,457,837,560) (761,347) 1,840,056,654

Lifecycle kWh kW Lifecycle ThermsNo GIMC 2% 6% 9%Amount Exceeding Max Ratios 3% 10% 54%

Amount Exceeding Average Ratios -82% -50% 78%



19

kWh ratios for DEER sub categories

050

100150200250300

Equ

ip (

57)

CF

L LA

MP

s (7

5)C

ontr

ols

(58)

Con

trol

s (8

3)H

eatR

ej (

64)

Equ

ip (

84)

Irrig

atio

n (6

7)C

ontr

ols

(81)

Fou

r ft

. F

luor

esce

nt (

37)

MO

TO

R (

42)

De-

lam

p (3

3)M

aint

enan

ce (

71)

CF

L LA

MP

s (6

3)P

ool P

ump

(26)

Ven

ding

Mac

hine

(48

)V

FD

(69

)E

xit

Sig

n (3

5)E

xter

ior

Ligh

ting

(36)

Mai

nten

ance

(77

)M

etal

Hal

ide

(41)

Bal

last

(28

)H

oldi

ng C

abin

et (

40)

Ene

rgy

Sta

r R

efrig

erat

ors

(6)

Low

flo

w s

how

erhe

ad (

24)

Equ

ip (

52)

Occ

upan

cy S

enso

r (4

3)H

igh

effic

ienc

y w

ater

hea

ter

Cop

y M

achi

ne (

32)

Tim

eclo

ck (

47)

Ene

rgy

Sta

r C

loth

es W

ashe

rH

eat

pum

p w

ater

hea

ter

(21)

Pho

toce

ll (4

4)E

nerg

y S

tar

Dis

h W

ashe

r (1

9)F

ryer

(38

)S

hell

(74)

kW

h p

er

$ G

IMC

20

Watt ratios by DEER subcategories

-

1020

3040

5060

7080

90E

quip

(57

)H

eatR

ej (

64)

CF

L LA

MP

s (7

5)C

ontr

ols

(58)

Con

trol

s (8

3)C

ontr

ols

(81)

Equ

ip (

84)

Irrig

atio

n (6

7)

Mai

nten

ance

(71

)M

aint

enan

ce (

77)

MO

TO

R (

42)

Poo

l Pum

p (2

6)F

our

ft.

Flu

ores

cent

(37

)O

ccup

ancy

Sen

sor

(43)

De-

lam

p (3

3)E

quip

(52

)C

FL

LAM

Ps

(63)

VF

D (

69)

Exi

t S

ign

(35)

Ene

rgy

Sta

r C

loth

es W

ashe

r (5

)Lo

w f

low

sho

wer

head

(24

)E

nerg

y S

tar

Ref

riger

ator

s (6

)H

igh

effic

ienc

y w

ater

hea

ter

(23)

Met

al H

alid

e (4

1)B

alla

st (

28)

Hea

t pu

mp

wat

er h

eate

r (2

1)C

opy

Mac

hine

(32

)S

hell

(74)

Ene

rgy

Sta

r D

ish

Was

her

(19)

Hol

ding

Cab

inet

(40

)F

ryer

(38

)

Wa

tts

pe

r $

GIM

C

21

Using DEER groups yields comparable results

• Based on ratios using 60 DEER subcategories, results are similar to end-use matching

• Matches for gas measures remains poorLifecycle kWh kW Lifecycle Therms

Total for Subset 74,928,316,618 1,345,914 2,365,642,955 No GIMC 1,945,144,960 89,162 208,425,769 Amount Exceeding Max Ratios 2,025,831,264 153,511 1,278,622,386

Amount Exceeding Average Ratios (65,083,266,762) (871,224) 1,823,569,946



22

Unmatched Measures

• We used the maximum impact per GIMC as a very generous criteria

• Yet, even with that criteria, 562 cases where kWh ratio is exceeded, 406 cases for kW ratio, and 490 cases for Therm ratio. (1,045 unique cases).– Cases by utility

• PG&E: 393• SCE: 121• SDG&E: 167• SoCal Gas: 364

• Note: may indicate an input problem, or a problem with the assumed mapping or a problem introduced by the large number of DEER runs excluded due to unit mismatches.

23

With better data, the ratio test could be a useful screen

• Distribution of kWh ratios from the utility measures

• Largest ratios are for measures such as– Pre-rinse spray valve – electric water heating– Strip curtains– Lighting

24

Identification of measures with largest ratio mismatches

• Based on sorting lifecycle amounts

• The measures will differ from column to column

# of measures

kWh kW Therms

100 88% 95% 93%

50 79% 88% 81%

25

Some DEER Measure Gaps

• Non-Res freezers and refrigeration

• Non-Res ovens

• Non-Res pool heaters

• Computers

• Non-Res gas measures

26

Summary of Findings• 76% of cases with some installations have no easy links to RunIDs.

– Note that for PG&E, 27% of their 1668 cases w/o DEER RunIDs are for calculated measures

• The Max Ratio test is a blunt instrument, but more precision would be very time consuming given the data in DEER and in the E3 calculators.

• With that caveat…– KWh estimates appear reasonable. (3% above max)– kW appears high, compared to DEER data (11% above max)– Therms are hard to judge via DEER (54% above max)– The max ratios “pass” 80% of the measures (4,274 out of 5,319)

• Recommendations– Future tool should require users to explicitly indicate if savings are relative to common or

code, and if costs are installation or incremental.– Need a way to link to DEER sub categories, at a minimum, to allow for automated checks– Need a process for creating new approved measures, and for updating DEER.– EULs should be reconsidered for retrofits (remaining life gets common benefit, and EUL-

remaining life gets code benefit)– Secondary impact on other fuels should be considered.– Direct install costs should be input on a measure basis (not moved into lump sum admin

costs) to allow for QA/QC review.

1 quality control review of e3 calculator inputs comparison to deer database brian horii energy and...

Documents