microsoft word - 37-rto workpaper_041816.docx · web view2020/10/26 · this whitepaper proposes...

Microsoft Word - 37-RTO workpaper_041816.docx

XX-XX: Behavioral Modification: Home Energy Reports

Residential (Single and Multi-Family)

Electric and Natural Gas

Issued: 10/26/2020

Prepared by: Guidehouse Inc,

Version: 1.1

Michigan Residential Measures1Version 1.1

Document Revision History

Version #

Revision Date

Description of Revisions and Affected Sections

Edited By

1.1

10/26/2020

· Added requirement for number of reports to section 4.

· Updated required components in Section 4.1 based on conversations with Oracle on what key components have appeared in all past reports.

· Confirmed that multi-family residences are eligible for the program and were included in calibration. Edited language on title page and in Section 4.2.

· Clarified relationship between coincidence factor and demand savings factor in section 9.

Guidehouse

EWR Collaborative Review and Decision

Date

Description of Collaborative Decision

Lead

Table of Contents

1.New Measure and Update to Existing Measure11.1 Updates to Existing Measures11.2 New Measures31.3 Missing Measures32.Purpose and Measure Background33.Measure Description44.Measure Specification & Savings Summary44.1 Measure Specification44.2 Requirements for Application44.3 Savings Summary55.Baseline and Proposed Improvement Description75.1 Baseline Measure, Practice, or Technology75.2 Improved Measure, Practice, or Technology86.Methodology and Assumptions86.1 Data96.2 Energy Savings106.2.1 Regression Analysis106.2.2 Annualizing Savings106.2.3 Potential Savings Overlap (or Uplift Adjustment)116.3 Coincident Peak Demand Savings126.3.1 Regression Analysis126.3.2 Potential Savings Overlap (or Uplift Adjustment)126.3.3 Demand Savings Factor137.Estimated Savings over Baseline138.Potential Savings Overlap149.Coincidence Factor1410.Measure Life1411.Measure Cost1412.Any Recurring Cost1413.Relevant Codes and Standards1414.Ongoing EM&V, Research, and Calibration Planning1415.Sources of Information1516.Attachments15

Michigan Behavior Resource Manual Residential Measures11Version 1.1

1. New Measure and Update to Existing Measure

Based on the results of a calibration study, this whitepaper proposes:

(1) Updates to existing electric and gas measures based on the results of the calibration study. In some cases, there was insufficient data to calibrate the energy savings estimate. In these cases, this whitepaper proposes updates to the existing measures just to account for the calibrated uplift adjustment factors and the demand savings factor.

(2) New measure submissions resulting from the introduction of new electric usage bands (splitting the >11k kWh usage band into an 11k-13k kWh usage band and a >13k kWh usage band) and a new gas usage band (601-900 therms), or where a BRM value did not exist but for which there was sufficient data to calibrate.

(3) In some cases, a usage band/year combination does not exist in the BRM and there was insufficient data to calibrate. In these cases, this whitepaper proposes that a utility would claim savings using the last year available in the BRM for that usage band.

The following table identifies the type of proposed update for each measure:

Fuel

Usage Band

Year 1

Year 2

Year 3

Year 4

Year 5

Year 6

Year 7

Year 8

Electric

5k-7k kWh

CV

CV

CV

CV

CV

CV

Electric

7k-9k kWh

CV

CV

CV

CV

CV

PCV

CV

Electric

9k-11k kWh

CV

CV

CV

CV

CV

CV

PCV

CV

Electric

11k-13k kWh

NM

NM

NM

NM

NM

Electric

>13k kWh

NM

NM

NM

NM

NM

NM

Gas

601-900 Therms

NM

NM

NM

NM

Gas

901-1200 Therms

CV

CV

CV

CV

CV

CV

CV

Gas

>1200 Therms

CV

CV

CV

CV

PCV

PCV

PCV

CV

CV = Calibrated Value: Update to existing measure based on results of the calibration study

PCV = Partially Calibrated Value: Update to existing measure based on results of the calibrated uplift and demand savings factor only

NM = New Measure: New measure submission

Blank: Missing value

1.1 Updates to Existing Measures

This whitepaper proposes updates to the following existing electric and gas measures based on the results of calibration.

Measure Code[footnoteRef:2] [2: The measure code uses the following structure: MeasureType-Sector-MeasureCategory-MeasureNumber-FuelType-Climate-BuildingType-System-Vintage-MeasureVersion. All measures included in this Behavior Resource Manual will be Measure Type “B”. ]

Electric/ Gas

Behavior Measure Description

B-RE-MS-000003-E-XX-XX-XX-XX-01

Electric

Behavior Modification: Home Energy Reports, 9k-11k kWh Annual Usage (1st year)


Electric



Electric



Electric

Behavior Modification: Home Energy Reports, 9k-11k kWh Annual Usage (2nd year)


Electric


Measure Code

Electric/ Gas



Electric



Electric

Behavior Modification: Home Energy Reports, 9k-11k kWh Annual Usage (3rd year)


Electric



Electric

Behavior Modification: Home Energy Reports, 9k-11k kWh Annual Usage (4th year)


Electric



Electric



Electric



Electric



Electric



Electric



Electric


B-RE-MS-000020-G-XX-XX-XX-XX-01

Gas

Behavior Modification: Home Energy Reports, 900-1200 Therms Annual Usage (1st year)


Gas

Behavior Modification: Home Energy Reports, >1200 Therms Annual Usage (1st year)


Gas

Behavior Modification: Home Energy Reports, 900-1200 Therms Annual Usage (2nd year)


Gas

Behavior Modification: Home Energy Reports, >1200 Therms Annual Usage (2nd year)


Gas

Behavior Modification: Home Energy Reports, 900-1200 Therms Annual Usage (3rd year)


Gas

Behavior Modification: Home Energy Reports, >1200 Therms Annual Usage (3rd year)


Gas

Behavior Modification: Home Energy Reports, 900-1200 Therms Annual Usage (4th year)


Gas

Behavior Modification: Home Energy Reports, >1200 Therms Annual Usage (4th year)


Gas



Gas



Gas



Gas



Gas



Gas


1.2 New Measures

This whitepaper proposes the addition of two new usage bands: (1) splitting the current >11k kWh annual usage band into an 11k-13k kWh and >13k kWh annual usage band, and (2) adding a new 600-900 therm annual usage band. In addition, we propose new values where values did not exist within the BRM but for which there was sufficient data to calibrate.

Electric/ Gas


Electric

Behavior Modification: Home Energy Reports, >13k kWh Annual Usage (1st year)

Electric


Electric

Behavior Modification: Home Energy Reports, >13k kWh Annual Usage (2nd year)

Electric


Electric

Behavior Modification: Home Energy Reports, >13k kWh Annual Usage (3rd year)

Electric


Electric


Electric

Behavior Modification: Home Energy Reports, >13k kWh Annual Usage (4th year)

Electric


Electric


Electric


Electric


Electric


Electric


Electric


Electric


Gas

Behavior Modification: Home Energy Reports, 600-900 Therms Annual Usage (1st year)

Gas

Behavior Modification: Home Energy Reports, 600-900 Therms Annual Usage (2nd year)

Gas

Behavior Modification: Home Energy Reports, 600-900 Therms Annual Usage (3rd year)

Gas


Gas


1.3 Missing Measures

This whitepaper proposes that when a usage band/year combination does not exist in the BRM, utilities should claim savings using the last year available in the BRM for that usage band. For example, if a wave in Year 4 was in the 600-900 therm usage band, the utility would claim the Year 3 value as the Year 4 value does not exist.

2. Purpose and Measure Background

This whitepaper reports the results of a 2020 calibration of the Behavior Modification: Home Energy Report measure. All usage bands with available data for a given program year were calibrated and this whitepaper proposes updated or new savings values accordingly. The calibration study also calibrated the demand savings factor for this measure.

Note, this whitepaper should not serve as an evaluation guidance document for custom evaluation of behavior modification measures. Additionally, any new measure submissions for the Behavior Modification: Home Energy Report measure should include descriptions of how the new measure values were estimated.

3. Measure Description

The Home Energy Reports (HER) program provides residential households accurate and timely information on their energy consumption through a variety of communication methods to change the consumers’ energy usage behavior.

·

The HER program is organized around two concepts. First, to motivate consumers through normative messaging to change their energy-use behavior. Personalized neighbor comparisons based on the size of the home, location, and heating fuel type—among other criteria—give households a motivational benchmark for their energy usage. Second, to provide them with salient, personalized advice to capitalize on this motivation to use less energy and save money.

Program reports are delivered through direct mail and are often supplemented with digital communications such as email, web, mobile phones, and social networks. This platform approach ensures that all households can access this information in the most effective way.

Applicable Building Type(s)

Applicable Sub-categories

Applicable Fuel Type(s)

Affected End-Use Load(s)

Electricity

Natural Gas

Space Heating

Space Cooling

Water Heating

Lighting

Appliance

Process

Other

Residential

n/a

ü

ü

ü

ü

ü

ü

ü

ü

4. Measure Specification & Savings Summary4.1 Measure Specification

HERs are a tool that can be deployed to residential customers to drive a behavioral change to reduce home energy consumption. To qualify for this measure, a program must send recurring reports by direct mail targeting a minimum of four reports per year.[footnoteRef:3] A program can supplement with digital communication but an all-digital program would not qualify for this measure. Key information required to qualify for the measure includes, at a minimum: [3: A wave might not reach four reports in a program year if it is launched late in the year but should reach four reports in the first 12-months after wave launch.]

1. Comparison of the customer’s home energy use to energy usage of other homes

2. Suggested actions the customer can take to improve energy efficiency

In addition, the program must have a simple opt-out process.

4.2 Requirements for Application

The Behavior Modification: Home Energy Report measure is restricted to residential customers.

4.3 Savings Summary

Electric Measures

Measure Code[footnoteRef:4] [4: The measure code uses the following structure: MeasureType-Sector-MeasureCategory-MeasureNumber-FuelType-Climate-BuildingType-System-Vintage-MeasureVersion. All measures included in this Behavior Resource Manual will be Measure Type “B”. For existing measures, the measure version has been updated from “01” to “02”. For new and missing measures, the measure code must be assigned.]

Year

Usage Band

Electric Savings per Customer

Coincident Peak Demand Savings

1st Year

>13k kWh Annual Usage

1.13%

0.88%

1st Year

11k-13k kWh Annual Usage

0.90%

0.70%


1st Year


0.78%

0.61%


1st Year


0.72%

0.56%


1st Year


0.12%

0.09%

2nd Year


1.88%

1.47%

2nd Year


1.42%

1.11%


2nd Year


1.36%

1.06%


2nd Year


1.27%

0.99%


2nd Year


0.59%

0.46%

3rd Year


2.19%

1.71%

3rd Year


1.66%

1.29%


3rd Year


1.50%

1.17%


3rd Year


1.29%

1.01%

3rd Year


0.87%

0.68%

4th Year


2.11%

1.65%

4th Year


1.84%

1.44%


4th Year


1.42%

1.11%


4th Year


1.81%

1.41%

4th Year


0.77%

0.60%

5th Year


2.19%

1.71%

5th Year


1.54%

1.20%


5th Year


1.24%

0.97%


5th Year


1.64%

1.28%

5th Year


0.89%

0.69%

6th Year

> 13 kWh Annual Usage

1.98%

1.54%

6th Year


*

*


6th Year


1.59%

1.24%


6th Year


1.61%

1.25%

6th Year


0.93%

0.73%

7th Year


*

*

7th Year


*

*


7th Year


2.58%

2.01%


7th Year


1.98%

1.54%

7th Year


*

*

8th Year


*

*

8th Year


*

*

8th Year


2.02%

1.58%

8th Year


*

*

8th Year


*

*

Note: Energy savings quoted as percentage of baseline energy consumption.

* These are the missing values described in Section 1.3. For these values, this whitepaper recommends the utility claim the

last year available in the BRM for that usage band. For example, if a utility would like to claim savings for a wave in Year 7 in

the 5k – 7k kWh band, we recommend using the Year 6 value. These values could be explicitly included in the BRM or a

statement of how to handle them could be included.

Gas Measures

Measure Code

Year

Usage Band

Gas Savings per Customer

1st Year

600-900 Therms Annual Usage

0.27%


1st Year


0.47%


1st Year

>1200 Therms Annual Usage

0.41%

2nd Year


0.45%


2nd Year


0.68%


2nd Year


0.74%

3rd Year


0.51%


3rd Year


0.71%


3rd Year


0.73%

4th Year


*


4th Year


0.74%


4th Year


0.82%

5th Year


0.27%


5th Year


0.75%


5th Year


1.04%

6th Year


*


6th Year


0.73%


6th Year


0.69%

7th Year


*


7th Year


0.89%


7th Year


0.78%

8th Year


*

8th Year


*

8th Year


0.82%

Note: Energy savings quoted as percentage of baseline energy consumption.

* These are the missing values described in Section 1.3. For these values, this whitepaper recommends the utility claim the last year available in the BRM for that usage band. For example, if a utility would like to claim savings for a wave in Year 4 in the 600 – 900 therms band, we recommend using the Year 3 value. These values could be explicitly included in the BRM or a statement of how to handle them could be included.

5. Baseline and Proposed Improvement Description5.1 Baseline Measure, Practice, or Technology

The baseline measure is a control group customer that does not receive energy usage feedback and benchmarking from a direct mail or email HER.

5.2 Improved Measure, Practice, or Technology

The behavior modification report is an improved measure in that it provides customer-specific energy usage feedback, benchmarking and tips to identify potential savings opportunities, thus altering participant behavior and energy consumption. See Section 4 for specific measure requirements.

6. Methodology and Assumptions

The HER program, implemented by the investor-owned utilities in Michigan, employs a randomized controlled trial (RCT) design that allows for the accurate and unbiased measurement of program impacts. In an RCT design, eligible households are randomly assigned to either a treatment group that receives HERs or a control group that does not receive them. A sound randomization leads to treatment and control groups that are statistically equivalent along observed and unobserved characteristics with the sole difference being that the treatment group receives HERs, and the control group does not. Therefore, any difference in energy usage observed between the treatment and control group can be attributed to the HERs. The RCT program design is summarized in Figure 1.

Figure 1: Experimental Setup for a Randomized Controlled Trial

Experimental design is considered the “gold standard” of program evaluation and an RCT is recognized as an appropriate method for evaluation in the United States Department of Energy’s Uniform Methods Project and in an evaluation protocol report authored by the State and Local Energy Efficiency (SEE) Action Network in 2012. Experimental designs eliminate self-selection bias, wherein customers who choose to join a program are different from those who choose not to join, making the savings estimates from experimental designs known to be unbiased.

The remainder of this section discusses the data used for calibration and then the methodology for the energy and coincident peak demand savings results.

6.1 Data

For this calibration study, Guidehouse included data for over 1.2 million HER participants and just under 400,000 controls across 22 program waves for DTE Energy and Consumers Energy. For DTE Energy, data spanned from 2010 to 2019. For Consumers Energy, data was only included from 2011 to early 2015.[footnoteRef:5] The number of participants and controls in the evaluation of savings from behavior modification varies with the program year and fuel type. Figure 2 shows how many HER participants and controls Guidehouse included in the calibration study for each usage band/program year combination. [5: Newer data was not incorporated for Consumers Energy waves due to a gap in the program in 2015 and then changing implementers between 2016 and present. ]

Figure 2: Summary of Uplift Adjustment Factor

Source: Guidehouse

For demand savings, Guidehouse was only able to include DTE Energy waves that launched after 2016 due to AMI data limitations. The following table shows the waves, years, and customer counts included in the demand calibration.

Wave

Utility

Fuel

Start Date

PY1

PY2

PY3

PY4

Participants*

Controls*

dte_201602_d

Dual

DTE

2/1/2016

Y

Y

Y

Y

18,915

9,255

dte_201602_e

Elec

DTE

2/1/2016

Y

Y

16,830

9,665

dte_201606_d

Dual

DTE

6/1/2016

Y

Y

Y

Y

9,188

6,453

dte_201606_e

Elec

DTE

6/1/2016

Y

Y

19,555

12,739

dte_201710_d

Dual

DTE

10/1/2017

Y

Y

27,348

9,928

dte_201803_d

Dual

DTE

3/1/2018

Y

Y

25,676

11,789

dte_201901_d

Dual

DTE

1/1/2019

Y

23,296

9,750

* Counts represent unique number of customers included in regression analysis for demand savings across all years.

Source: Guidehouse

6.2 Energy Savings

For calibration, the energy savings resulting from the HER program were calculated using a regression-based ex post evaluation grouping individual program waves together into a single regression for each usage band and program year.[footnoteRef:6] These regressions compare participant energy use to a counterfactual, baseline usage scenario in the absence of the program. In the case of an RCT, baseline usage is determined by the control group. The estimates are then annualized and adjusted to account for savings overlap. [6: As noted in Section 2, this whitepaper is not an evaluation guidance document for custom evaluation of behavior modification measures. Evaluation of individual program waves (as opposed to usage band groups) may utilize different regression specifications.]

6.2.1 Regression Analysis

Guidehouse used a lagged dependent variable (LDV) model to estimate daily per participant savings from HERs, by usage band and program year, using monthly billing data. Formally, the model is:

Where,

is the average daily usage for household i during month t

is the average daily usage for household i during the same month as time t lagged to the pre-program year[footnoteRef:7] [7: For DTE’s leading wave (which launched in July 2011) data was missing for more than 90% of the wave’s customers in the first three months of the pre-period (July-September 2010). To address the missing data, Guidehouse imputed pre-period data for this wave (but no other waves). Imputing the pre-period data involved assigning each customer missing a given pre-period usage observation the average value for all customers who have data in the same month of the pre-period. For example, all customers missing usage data in July 2010 were assigned the average for all customers with data for that month.]

comprise a set of month-of-year indicators, which equal 1 if t falls in month-of-year j, and 0 otherwise

is a binary variable that equals 1 if household i is in the treatment group and 0 otherwise

is the number of cooling degree days for household i during month t

is the number of heating degree days for household i during month t

comprise a set of wave indicators, which equal 1 if household i is in wave w and 0 otherwise

is a binary variable that equals 1 if household i is a DTE customer and 0 if a Consumers Energy customer

is the cluster-robust error term for household i in time t

New measure submissions to the BRM for the Behavioral Modifications: Home Energy Reports measure should use an LDV model specification.

6.2.2 Annualizing Savings

To get one annualized savings value, average daily savings is multiplied by 365 days.

6.2.3 Potential Savings Overlap (or Uplift Adjustment)

HERs may increase participation in other utility energy efficiency programs; this additional participation is known as efficiency program uplift and could result in double counted savings if not addressed. To avoid double counting savings, the savings from program uplift are subtracted from behavior modification report savings and attributed to the lifted program.

Chapter 17 of the Uniform Methods Project protocols identifies the following three steps for measuring savings overlap from programs with individual customer tracking data[footnoteRef:8] using a difference-in-difference (DID) statistic: [8: Guidehouse also considered uplift into upstream lighting programs, but evidence from 2018 and 2019 DTE Energy HER program surveys suggests that treatment and control customers purchase LED bulbs at almost identical rates. This evidence indicates that no upstream lighting uplift adjustment is necessary.]

1. Match the [behavior-based] program treatment and control group subjects to the utility energy efficiency program tracking data.

2. Calculate the uplift savings per treatment group subject as the difference between treatment and control groups in average efficiency program savings per subject, where the savings are obtained from the utility tracking database of installed measures. (The averages should be calculated over all treatment group subjects and all control group subjects, not just those who participated in efficiency programs.)

3. Multiply the uplift savings per treatment group customer by the number of subjects who were in the treatment group to obtain the total uplift savings. (pg. 37)

The general process for removing double counted savings from programs with individual customer tracking data from the savings estimate out of the regression analysis is described in Figure 3. The green shapes outline the steps for the regression analysis, the yellow shapes outline the steps for the double counting analysis. The equation at the bottom of the figure shows that double counted savings are removed from savings coming out of the regression analysis to get the savings value without double counted savings which is recommended for the BRM.

Figure 3: Double Counting Adjustment Process for Programs with Individual Customer Tracking Data

Source: Guidehouse

The calibration study calculated uplift for all DTE Energy waves using a difference-in-difference (DID) statistic based on pre-period and post-period average savings from other energy efficiency programs with individual customer tracking data.[footnoteRef:9] These estimates were combined with uplift estimates from the prior Consumers Energy evaluations[footnoteRef:10] and weighted by the proportion of participants from each utility. [9: Estimates were based on evaluations of program tracking data for all programs and program years where data was available. The other program savings values were adjusted for an installation rate adjustment factor (IRAF) and a net-to-gross ratio (NTGR) to reflect the fact that HER savings produced by the regression inherently account for these items.] [10: The program uplift estimates for Consumers Energy were based on a simple difference between treatment and control participation rates (not savings) during the program year.]

Guidehouse only accounted for cases of positive uplift (i.e., where HER increased participation in other programs) for a given wave and program year. Cases where uplift is negative (i.e., where HER decreased participation in other programs) did not lead to an adjustment to the savings. There is no issue of double counting savings at the portfolio level for negative uplift which is why it was not adjusted for.[footnoteRef:11] [11: The issue of negative uplift is a baseline issue. If negative uplift occurs and you are trying to estimate savings for the HER program in the absence of all other programs, negative uplift should be added back to the HER savings to account for the fact that the control customers’ usage (which represents the baseline) is too low. However, when estimating HER while considering all of the other programs, this adjustment is not appropriate.]

Figure 4 shows the uplift adjustment found in this calibration study and the equation for adjusting the regression-savings.

Figure 4: Uplift Adjustment Factor

Source: Guidehouse

6.3 Coincident Peak Demand Savings

6.3.1 Regression Analysis

Guidehouse estimated demand savings using the same regression model presented in Section 6.2.1 above but including only coincident peak data.[footnoteRef:12] Guidehouse then estimated an analogous energy savings model using the same HER waves and program years. [12: The coincident peak is defined as 3-6pm on the three hottest, consecutive, non-holiday weekdays in July.]

6.3.2 Potential Savings Overlap (or Uplift Adjustment)

For coincident peak demand savings, Guidehouse excluded customers in DTE Energy’s demand response program to avoid double counting with demand response event savings.

6.3.3 Demand Savings Factor

A demand savings factor was calculated based on the following equation.[footnoteRef:13] [13: The energy and demand savings in this equation are after adjusting for uplift as described in Section 6.2.3 and 6.3.2.]

Guidehouse found a demand savings factor of 0.78.

The reported coincident peak demand savings are calculated by multiplying the energy savings rate by the demand savings factor as shown in the following equation.

7. Estimated Savings over Baseline

This section provides information and formulas to calculate energy and coincident peak demand savings for HERs, along with an illustrative example.

Estimating energy savings requires the calculation of three input values:

· Average Annual Energy Usage – calculated using billing data from the 12-month program year prior to the program year for which savings are being estimated for customers randomly assigned to the HER control group.[footnoteRef:14] [14: For example, if savings are being claimed for PY2019, average annual energy usage would be calculated using control customers from January to December 2018.]

· Savings Rate – Average Annual Energy Usage along with year[footnoteRef:15] determine the appropriate BRM percent savings rate to be applied for both energy savings and coincident peak demand savings. [15: Since savings are annualized, the year a wave launches (regardless of the month) is year 1, the next year would be year 2, and so on. For example, if a wave launched in June 2017, PY2017 would be year 1, PY2018 would be year 2, PY2019 would be year 3, etc. ]

· Number of Active Participants – the number of HER participants that had an active utility account at the premise where they receive HERs as of the end of the program year.[footnoteRef:16] [16: Note that participant households who opt out of the HER program should be included in this count as they are included in determining savings to preserve the validity of the RCT.]

Estimating coincident peak demand savings requires one additional input:

· Average Coincident Peak Demand – calculated using hourly demand data during the coincident peak from July of the program year prior to the program year for which savings are being estimated for customers randomly assigned to the HER control group.[footnoteRef:17] [17: For example, if savings are being claimed for PY2019, average coincident peak demand would be calculated using control customers during 2018’s coincident peak.]

These values are used to estimate savings using the following equations:

Example Savings Calculation

Consider a dual fuel HER wave with 10,000 active participants in the first program year, average annual electric usage of 8,000 kWh, 5 coincident peak kW and 1,300 Therms for the control group in the prior program year.

· Electric Energy Savings are:

576,000 kWh = 8,000 kWh * 0.72% * 10,000

· Coincident Peak Demand Savings are:

280 kW = 5 kW * 0.56% * 10,000

· Gas Energy Savings are:

53,300 Therms = 1,300 Therms * 0.41% * 10,000

8. Potential Savings Overlap

Refer to Sections 6.2.3 and 6.3.2.

9. Coincidence Factor

The rate of coincident peak savings is 1 for all measures in this whitepaper.[footnoteRef:18] [18: The demand savings factor of 0.78 was determined by calculating the ratio of coincident peak demand savings relative to energy savings during the coincident peak period. As a result, the demand savings rate (calculated as the energy savings rate x 0.78) represents coincident peak demand savings rate. As such, the coincidence factor is 1.0.]

10. Measure Life

The measure life is 1 year for all measures in this whitepaper.

11. Measure Cost

$6.91 per measure (i.e., cost to deliver approximately four print HERs to one participant for one year)

12. Any Recurring Cost

There are no recurring costs associated with this measure.

13. Relevant Codes and Standards

There are no relevant codes or standards associated with this measure.

14. Ongoing EM&V, Research, and Calibration Planning

· Year 9 savings to be re-submitted by Oracle and DTE in accordance with new modelling guidelines from the Michigan Public Service Commission.

· Year 10 savings to be submitted by Oracle and DTE for 2022 BRM.

· In the absence of significant changes in program design or delivery, savings values for Year 6 and up should be calibrated in 3-4 years. Significant changes in program design or delivery may warrant calibration sooner or covering more years.

15. Sources of Information

· U.S. Department of Energy, “Chapter 17: Residential Behavior Protocol,” in The Uniform Methods Project: Methods for Determining Energy Efficiency Savings for Specific Measures, 2015. Accessed 22 March 2017, https://energy.gov/sites/prod/files/2015/02/f19/UMPChapter17-residential-behavior.pdf.

· SEE Action Network, “Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations,” 2012. Accessed 22 March 2017, https://www4.eere.energy.gov/seeaction/system/files/documents/emv_behaviorbased_eeprograms.pdf.

16. Attachments

· Attachment A: “Home Energy Report Measure Calibration”, presentation given to Michigan Public Service Commission on October 5, 2020

· File Name: Attachment_A_MI BRM Calibration Results 2020-10-05 - Presentation to MPSC.pdf

Attachment_A_MI

BRM Calibration Results 2020-10-05 - Presentation to MPSC.pdf

Home Energy Report Measure CalibrationPresentation to the

Michigan Public Service Commission

October 5, 2020

1©2020 Guidehouse Inc. All Rights Reserved

1. Measure Description Slide 2

2. Motivation Slide 3

3. Methodology Slide 4

4. Data Slide 11

5. Model Comparison Results Slide 15

6. BRM Savings Value Recommendations Slide 20

7. Next Steps Slide 25

8. Appendices Slide 27

Agenda


HERs change energy use behavior1 through two primary

mechanisms:

1. Motivates residential customers through normative messaging to

change their behavior. Personalized neighbor comparisons based

on home size, location and energy type—among other criteria—

give households a motivational benchmark for their energy usage.

2. Provides residential customers with salient, personalized advice

to capitalize on this motivation to use less energy and save

money.

HERs are delivered through direct mail and are often supplemented

with digital communications such as email, the web, telephones,

mobile phones, and social networks. This platform approach ensures

all households have access to the information.

1. Measure DescriptionHome Energy Reports (HERs) seek to achieve energy savings by providing households accurate

monthly electric and/or gas usage information, motivating a change in energy use behavior.

Figure 1. Sample Home Energy Report

Source: DTE Energy’s HER Program Implemented by Oracle 1 Allcott, H. Social norms and energy conservation. Journal of Public Economics (2011), Volume 95, Issues

9-10: 1082-1095.


The primary drivers of conducting calibration

research on the HER measure at this time are:

– These measures have not been calibrated

since 2017 at which time values for Year 1 through

Year 5 were calibrated using data from all DTE

Energy (DTE) and Consumers Energy (CE)1

commercialized waves, with a few exceptions.2

– Since that time, savings values for Years 6-9 have

been added or proposed for inclusion in the BRM

based on results from DTE’s 2011 pilot wave.

– The peak demand savings factor in the BRM has

never been calibrated. It is currently 1.5 times

energy savings.

2. MotivationThe HER measure in the Behavior Resource Manual (BRM) was last calibrated using data through

2016.

1 Oracle’s file names abbreviate Consumers Energy as CMS so this appears

in their wave naming conventions throughout this deck.2 Year 5 9-11k kWh and Year 5 >1200 therms were not calibrated in the prior

study due to insufficient data. The DTE_201504_D wave was not included in

the gas calibration due to usage being below the 900 CCF cutoff. The

CMS_201204_E_BC, CMS_201203_G, and CMS_201403_D waves were

not included in the study due to the lack of zip code data for these waves.

Fuel

Type

Usage

Band

Year

1

Year

2

Year

3

Year

4

Year

5

Year

6

Year

7

Year

8

Year

9

Elec5k-7k

kWh0.70% 0.77%

Elec7k-9k

kWh0.84% 1.50% 1.82% 1.25% 2.01% 1.70% 2.20% * *

Elec9k-11k

kWh1.08% 1.52% 1.77% 2.27% 2.18% 2.12% 2.73% * *

Elec>11k

kWh1.20% 1.78%

Gas900-1200

Therms0.34% 0.53% 0.91% 0.86% 0.66% 0.73% 0.82% * *

Gas>1200

Therms0.43% 0.60% 0.57% 0.66% 1.09% 0.73% 0.82% * *

Table 1. Current BRM Usage Bands and Savings Rates

*Values have been proposed by DTE/Oracle for these usage bands.

Source: MI BRM


3. MethodologyThis calibration study estimates energy savings by usage band and program delivery year (i.e., Year

1, Year 2) using monthly billing data for the DTE (between 2010 and 2019) and CE (between 2011

and early 2015)1 programs offering HERs.2

1 New data was not incorporated for CE waves due to a gap in the program in 2015 and then changing implementers between 2016 and present. 2 Each HER wave/program year included in the calibration analysis had 12 months of post-program data available for the applicable program year.3 State and Local Energy Efficiency (SEE) Action Network. 2012. Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues and

Recommendations. Prepared by A. Todd, E. Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National Laboratory. http://behavioranalytics.lbl.gov.

Stewart, J. and A. Todd. Chapter 17: Residential Behavior Evaluation Protocol. In NREL’s Uniform Methods Project Protocols.

Note: Complete model specifications are shown in Appendix A.

DTE and CE implement their HER programs as randomized controlled trials (RCTs), wherein customers are randomly assigned to treatment and control groups (Figure 2). This program design is known to produce unbiased estimate of program impacts.2

Because customers are randomly assigned into a treatment group or a control group, they are expected to be equivalent in every way expect program treatment - in this case, receipt of the report. As such, any differences in usage between the treatment group and the control group observed in the program period are necessarily the result of the program.

Source: SEE Action Report3

Figure 2. Illustration of an RCT


3. MethodologyGuidehouse estimated savings using both a lagged dependent variable (LDV) model and a linear

fixed effect regression (LFER) to inform which model class should be used going forward.

LDV and LFER are two model classes used for evaluating savings for HER programs.

• Both are endorsed by the Uniform Methods Project (UMP)1 and, because of the RCT, are expected to produce unbiased

savings estimates.

• The LDV model is more commonly used by evaluators for claiming savings, although most evaluators estimate multiple

models for robustness and investigate if the models are not producing consistent results.

Each of these model classes account for naturally occurring randomization imbalances that lead to differences in usage

between the treatment and control groups in the pre-program period (i.e., before reports are sent), but they do so in different

ways. In a balanced RCT, these two model classes are expected to result in unbiased and similar (i.e., not statistically

different) results.

Guidehouse estimated models both for (1) each individual wave/program year to test differences between the two models,

and (2) for each usage band/program year (combining individual waves in the same usage band into one model) for

calibrating the BRM.

1 Stewart, J. and A. Todd. Chapter 17: Residential Behavior Evaluation Protocol. In NREL’s Uniform Methods Project Protocols.


𝐴𝐷𝑈𝑖𝑡= 𝜶𝒊 + 𝛽1𝑃𝑜𝑠𝑡𝑡 ∙ 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 +

𝐽

𝛽2𝑗𝑌𝑟𝑀𝑜𝑗𝑡 + 𝛽3𝐶𝐷𝐷_𝑎𝑛𝑑_𝑜𝑟_𝐻𝐷𝐷𝑖𝑡

+𝛽4𝑃𝑜𝑠𝑡𝑡 + 𝜀𝑖𝑡

• LFER controls for differences by including a customer-specific fixed

effect (a single value for each customer) which controls for all time

invariant observable and unobservable characteristics.

• The fixed effect does not control for time varying characteristics (for

example, if usage is highly seasonal).

3. MethodologyEquation 1 and 2 show the model specifications Guidehouse estimated for this calibration study.

𝐴𝐷𝑈𝑖𝑡=

𝐽

𝛽1𝑗𝑌𝑟𝑀𝑜𝑗𝑡 +

𝐽

𝛽2𝑗𝑌𝑟𝑀𝑜𝑗𝑡 ∙ 𝑨𝑫𝑼𝒍𝒂𝒈𝒊𝒕 + 𝛽3𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖

+ 𝛽4𝐶𝐷𝐷_𝑎𝑛𝑑_𝑜𝑟_𝐻𝐷𝐷𝑖𝑡 +

𝑊

𝛽5𝑊𝑎𝑣𝑒𝑖𝑤 + 𝛽6𝑈𝑡𝑖𝑙𝑖𝑡𝑦𝑖 + 𝜀𝑖𝑡

• LDV controls for differences between the treatment and control

groups by including lagged usage (from the pre-period) as an

explanatory variable.

• The lagged usage does a good job of controlling for differences in

usage over time.

• Time invariant customer characteristics must be explicitly added to

the model to be accounted for. With a RCT these characteristics

are expected to be well-balanced between the treatment and

control groups.

Equation 1. LDV Model1 Equation 2. LFER Model2

1 This is the LDV specification used for this calibration study; it is one example within the broader class of LDV models. Note 𝛽5 and 𝛽6 are included in pooled specifications (where different waves are modelled together) but are not included in single wave regressions as all customers would have the same value for wave and utility.2 This is the LFER specification used for this calibration study; it is one example within the broader class of LFER models. Note 𝛽4 is included in pooled specifications (where different waves are modelled together) but is not included in single wave regressions as Post is perfectly collinear with the year-month dummies in 𝛽2 when waves are modelled individually.


3. MethodologyThe figures below illustrate how the lagged usage and the fixed effect work for one individual

customer. Note, that both models produce an unbiased estimate of savings.

Figure 3. LDV Model Illustration Figure 4. LFER Model Illustration

Source: Guidehouse

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

201

9-0

1

201

9-0

2

201

9-0

3

201

9-0

4

201

9-0

5

201

9-0

6

201

9-0

7

201

9-0

8

201

9-0

9

201

9-1

0

201

9-1

1

201

9-1

2

202

0-0

1

202

0-0

2

202

0-0

3

202

0-0

4

202

0-0

5

202

0-0

6

202

0-0

7

202

0-0

8

202

0-0

9

202

0-1

0

202

0-1

1

202

0-1

2

Pre Post

Ave

rage

Da

ily U

sage

(T

he

rms)

Avg Daily Use (Therms) Fixed Effect

Note, the LDV model only includes unique observations of the dependent variable (average

daily usage) for the post period and the pre-period enters as an independent variable (lagged

usage). Thus, in estimating one year of savings, the model includes 12 observations.

Note, the LFER model includes unique observations of the dependent variable (average

daily usage) for both the pre and post periods. Thus, in estimating one year of savings,

the model includes 24 observations.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

202

0-0

1

202

0-0

2

202

0-0

3

202

0-0

4

202

0-0

5

202

0-0

6

202

0-0

7

202

0-0

8

202

0-0

9

202

0-1

0

202

0-1

1

202

0-1

2

Post

Ave

rage

Da

ily U

sage

(T

he

rms)

Avg Daily Use (Therms) Lagged Usage


3. MethodologyHERs may increase participation in other energy efficiency programs (also referred to as program

uplift). To avoid double-counting, the savings associated with program uplift are subtracted from the

HER program and attributed to the lifted program measures.

Figure 5. Report Savings Calculation

1Stewart, J. and A. Todd. Chapter 17: Residential Behavior Evaluation Protocol. In NREL’s

Uniform Methods Project Protocols.2 For example, HER treatment customers may participate in a weatherization program at the

same rate as HER controls, but when they participate, they may install more insulation leading

to higher savings.

Report Savings w/

Double Counted

Savings

Downstream

Double Counting

Adjustment

Factor

Report Savings w/out

Double Counted

Savings

Reg.

Savings

Estimate5.7%

BRM

Reported

Savings

Elec

Gas

Reg.

Savings

Estimate9.4%

BRM

Reported

Savings1

1

• The calibration study calculated uplift for all DTE waves using a

difference-in-difference (DID) statistic based on pre-period and

post-period average savings from other energy efficiency

programs. These estimates were combined with uplift estimates

from the prior CE evaluations and weighted by the proportion of

participants from each utility.

– This approach differs from the method used in the prior

calibration study in that:

1. It measures uplift associated with increased net savings

within a program rather than just increased participation in

accordance with UMP chapter 17.1,2

2. It removes estimates of negative uplift for a given wave /

program year, as negative uplift is a baseline issue rather

than a double counting issue.

• Compared to the prior calibration study uplift increased from

0.17% for electric and 5.04% for gas.

Source: Guidehouse


3. MethodologyMichigan-specific data, as well as secondary data from other jurisdictions, does not provide

evidence of lift in upstream lighting.

1 Ohio is conducting a study looking into upstream lighting uplift, but results are not expected

until late 2020.2 See Appendix B for references to all the studies shown in notes 3-8.

Jurisdiction1,2 Approach

National Grid

New York

Conducted a survey in 2014 in which they found near-zero

joint savings with upstream lighting that were not statistically

significant.3

PSE Applies a value based on a 2015 survey of treatment/control.

The evaluation estimates uplift for LEDs of 2.5-15 kWh per

household (results are not statistically significant).4

PG&E Calculates an adjustment value for LEDs by wave, based on

surveys of treatment/control. The 2015 evaluation estimated

uplift of 0-1 kWh per household for LEDs.5

SDG&E and

SCE

Use adapted versions of PG&E’s results.6, 7

Pennsylvania Adopted an adjustment factor into the TRM based on the

PSE (2015) and PG&E (2013) studies, and professional

judgement by the statewide evaluator. Year 1 adjustment

factor is 0.75% increased by 0.75 percentage points up to

Year 4.8

Maryland Adjust based on the Pennsylvania TRM.

• 2018 and 2019 surveys for DTE’s HER program show that

treatment and control customers purchase LED bulbs at almost

identical rates suggesting no change in participation in upstream

lighting because of the reports (Appendix B).

• Relatively few jurisdictions/utilities account for upstream lighting.

Of those that do (Table 2), adjustments are based on differences in

bulb purchases between treatment/control groups. HERs are not

found to consistently result in more bulb purchases in treatment

customers relative to the control group across these studies.

• In addition, upstream lighting is expected to make up less of the

utility’s portfolios in the coming years as the baseline shifts

reducing the potential for lift in upstream lighting.

Table 2. Upstream Lighting Approaches

Source: Guidehouse literature review


• Due to AMI data limitations, calibration of demand savings only included DTE waves launched in 2016 or later.

– Uplift for DTE’s demand response program was accounted for in these estimates by excluding customers in the demand response

programs.

• Guidehouse estimated a demand savings regression model for all waves with available AMI data and an analogous energy

savings regression model for the same waves. These regression specifications are included in Appendix A.

– The demand savings model only includes data for 3-6pm on the three hottest, consecutive, non-holiday weekdays in July.

• We calculated a demand savings factor by comparing the demand savings to the energy savings using the following

equation:

Demand Savings Factor =% 𝐷𝑒𝑚𝑎𝑛𝑑 𝑆𝑎𝑣𝑖𝑛𝑔𝑠

% 𝐸𝑛𝑒𝑟𝑔𝑦 𝑆𝑎𝑣𝑖𝑛𝑔𝑠

• The resulting demand savings factor can replace the current 1.5x demand savings factor used for all waves. The waves

used for demand calibration cover all but the highest usage band (>13 MWh).

3. MethodologyDemand savings have not been previously calibrated in the BRM and there is currently an assumed

1.5x demand savings factor. This study used available data to calibrate that demand savings factor.

Note: Due to data limitations the calibration study is unable to determine whether the demand savings factor varies by usage band. As a result, we assume a single demand savings factor for all

waves.


• In scoping this study, a question arose of whether older

program waves savings values were pertinent to the

current/future program given changes in design and time

period.

• Guidehouse found that, within a given usage band, savings for

older waves were neither systematically higher or lower than

newer waves.

• Guidehouse also reviewed the report layout and content and

found that although it has changed some over time the

information is largely similar.

• Report frequency has changed with print reports becoming

less frequent and email reports more frequent, but without

systematic changes in savings we didn’t conclude that this

warrants wave exclusion.

– DTE waves average between 3 and 6 print reports per year.

• Based on this review, Guidehouse included all waves with

data in a given year in our calibration.

4. DataGuidehouse included data from 1,256,027 treatment customer and 378,997 control customers across

22 program waves for DTE (spanning 2010-2019) and CE (spanning 2011-early 2015).

Waves Included in 2017

Calibration StudyNew Waves

CMS_201105_D

CMS_201203_D

CMS_201204_E_MUSK

CMS_201303_E

DTE_201107_D

DTE_201309_D

DTE_201309_E

DTE_201401_D

DTE_201401_E

DTE_201504_D

DTE_201504_E

DTE_201602_D*

DTE_201602_E*

DTE_201602_G

DTE_201606_D*

DTE_201606_E*

DTE_201610_G

DTE_201710_D*

DTE_201710_G

DTE_201711_G

DTE_201803_D*

DTE_201803_G

DTE_201901_D*

Table 3. Waves Included in Calibration1

* Wave included in demand savings factor calibration. Due to AMI data availability, only

DTE electric customers in waves starting after 2016 were included.1 Appendix D has more information about the included waves including customer counts

and program years included. This appendix also includes information about which

waves were excluded and why.Note: More information can be found in Appendix C.

Source: Guidehouse analysis


4. DataIn addition to calibrating savings based on existing BRM usage bands, this study also introduces

usage bands of greater than 13 MWh (breaks the current high end of >11 MWh into 11-13 and >13

MWh instead), and 600-900 Therms. Not all deemed savings values will be calibrated.

Fuel Usage Band

Electric 5 to 7 MWh

Electric 7 to 9 MWh

Electric 9 to 11 MWh

Electric > 11 MWh

Existing BRM

Usage BandsProposed BRM

Usage Bands

Table 4. Existing and Proposed Usage Bands

Fuel Usage Band

Gas 900-1200 Therms

Gas >1200 Therms

Fuel Usage Band

Electric 5 to 7 MWh

Electric 7 to 9 MWh



Electric > 13 MWh

Fuel Usage Band

Gas 600-900 Therms

Gas 900-1200 Therms

Gas >1200 ThermsSource: Guidehouse analysis

• Guidehouse proposes splitting the highest electric usage band

currently in the BRM (>11 MWh) in two bands (11-13 MWh and

>13 MWh).

– Both bands are large enough to always produce statistically

significant savings estimates (i.e., statistically different from

zero with 90% confidence).

– The >13 MWh band has higher savings than the 11-13 MWh

band in each program year where both bands have savings

estimates. The savings are or are close to statistically

different between the two bands in most years.

• Guidehouse proposes adding a lower usage band for gas from

600-900 therms.

– Four waves fall into this band and we see that they produce

savings that are statistically different from zero.


4. DataBetween DTE and CE, Guidehouse included 22 waves across 8 program years in the calibration

study.

Table 5. Number of Waves per Usage Band by Year1

Fuel Usage Band

Year

1

Year

2

Year

3

Year

4

Year

5

Year

6

Year

7

Year

8

Year

9

Electric 5 to 7 MWh 3 2 1 1 1 1

Electric 7 to 9 MWh 4 6 3 2 2 * 1 * *

Electric 9 to 11 MWh 5 5 5 4 1 3 * 1 *

Electric 11 to 13 MWh 4 2 2 1 1

Electric > 13 MWh 3 3 1 1 1 1

Gas 600 to 900 Therms 3 4 1 1

Gas 900 to 1200 Therms 8 7 6 5 3 3 1 * *

Gas > 1200 Therms 5 4 3 3 * * * 1 *

* These are deemed in the BRM (or have been proposed by DTE/Oracle) but there is not enough data to

calibrate them in this study.

1Refer to Appendix D for additional information on specific DTE and CE cohorts included in the calibration study.



4. DataThe earlier years have more customers in the calibration than the later years. The savings estimates

for program year/band combinations with fewer customers are less precise.

Figure 6. Customer Counts by Usage Band and Program Year



• Both the individual wave/program year regressions

and the pooled usage band/program year regressions

do not exhibit statistically significant differences

between the two model specifications.

• The point estimates of savings can be practically

different even if they are not statistically different.

There is no way to know which estimate is “correct”

as they are both unbiased estimates of savings.

• Importantly, we see that neither model is

systematically higher or lower than the other (51%

higher for LDV), such that choosing one model will

result in higher savings for some wave/program years

and lower savings for others.

• We see that the LDV model has slightly more precise

estimates than the LFER model. The difference is

about 5%.

5. Model Comparison ResultsGuidehouse found the point estimates of electric savings were never statistically different between

the LDV and LFER models.

Figure 7. LDV vs. LFER Electric Savings by Wave and Program Year


Note, each data point in the figure represents the savings estimate (LDV, LFER) for one

wave/program year. As such, the diagonal line represents cases where the LDV and

LFER savings are identical.

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

-0.2 0 0.2 0.4 0.6 0.8 1 1.2

LF

ER

Sa

vin

gs (

kW

h/d

ay)

LDV Savings (kWh/day)


• Similar to electric, both the individual wave/program year

regressions and the pooled usage band/program year

regressions do not exhibit statistically significant

differences between the two models.

• The LDV model is higher than the LFER model more

frequently for gas. However, most of the unbalanced

estimates belong to DTE’s leading wave (201107).

– This aligns with results from Oracle showing large

differences between the LDV and LFER models for

this wave.

• We see that the LDV model has slightly more precise

estimates than the LFER model. The difference is about

5% to 10%.

5. Model Comparison ResultsGuidehouse found the point estimates of gas savings were never statistically different between the

LDV and LFER models.

Figure 8. LDV vs. LFER Gas Savings by Wave and Program Year


Note, each data point in the figure represents the savings estimate (LDV, LFER) for one

wave/program year. As such, the diagonal line represents cases where the LDV and

LFER savings are identical.

0

0.01

0.02

0.03

0.04

0.05

0 0.01 0.02 0.03 0.04 0.05

LF

ER

Sa

vin

gs (

the

rms/d

ay)

LDV Savings (therms/day)

dte_201107_d


• Guidehouse’s LDV and LFER handle the missing pre-period data in different ways such that the two models end up using different data sets

to estimate savings.

– The LDV model drops data in the post-period if the pre-period data is missing. This means LDV is using only 9 months of post data to estimate savings.

– The LFER model only drops the pre-period data. This means LFER is correctly using the full 12 months of post data to estimate the savings.

5. Model Comparison Results – DTE 2011 WaveGuidehouse determined a primary cause of the differences for the DTE 201107 was missing pre-

period data (more than 90% of customers were missing Jul-Sep 2010 data).

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

201

2-0

7

201

2-0

8

201

2-0

9

201

2-1

0

201

2-1

1

201

2-1

2

201

3-0

1

201

3-0

2

201

3-0

3

201

3-0

4

201

3-0

5

201

3-0

6Post

Ave

rage

Da

ily U

sa

ge

(T

he

rms)

Avg Daily Use (Therms) Lagged Usage

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

201

0-0

7

201

0-0

8

201

0-0

9

201

0-1

0

201

0-1

1

201

0-1

2

201

1-0

1

201

1-0

2

201

1-0

3

201

1-0

4

201

1-0

5

201

1-0

6

201

2-0

7

201

2-0

8

201

2-0

9

201

2-1

0

201

2-1

1

201

2-1

2

201

3-0

1

201

3-0

2

201

3-0

3

201

3-0

4

201

3-0

5

201

3-0

6

Pre Post

Ave

rage

Da

ily U

sage

(T

he

rms)

Avg Daily Use (Therms) Fixed Effect

Figure 9. LDV Model Illustration w/ Missing Pre-Period Data Figure 10. LFER Model Illustration w/ Missing Pre-Period Data

Source: Guidehouse


5. Model Comparison Results – DTE 2011 WaveTo address the missing data, Guidehouse imputed pre-period data for the LDV model for the DTE

201107 wave so each model could estimate savings using 12 months of post-period data. Figure 12

shows LDV (with imputed data for DTE 201107) and LFER (with original data) are more similar.


Figure 11. Original Data Figure 12. Missing Pre-Data Imputed for

LDV Model for DTE 201107*

-0.2

0

0.2

0.4

0.6

0.8

1

-0.2 0 0.2 0.4 0.6 0.8 1LF

ER

Savin

gs (

kW

h/d

ay)


0

0.01

0.02

0.03

0.04

0.05

0 0.01 0.02 0.03 0.04 0.05LF

ER

Savin

gs (

therm

s/d

ay)


* Note in these graphics only the LDV model for the DTE

201107 wave (in orange) is re-estimated. The LFER model for

this wave does not change. All other waves (green dots) also

remain identical to the original.

Ele

ctr

icG

as

• For the LDV model, imputing the pre-

period data involved assigning each

customer missing a given pre-period

usage observation the average value for

all customers who have data in the same

month of the pre-period.

– For example, all customers missing

usage data in July 2010 were assigned

the average for all customers with data

for that month.

• Imputing data for LDV brings the models

closer together although there are still

some differences.

– These are not statistically significant

differences and in practical terms we

are discussing hundredths of therms

per day.

-0.2

0

0.2

0.4

0.6

0.8

1

-0.2 0 0.2 0.4 0.6 0.8 1LF

ER

Savin

gs (

kW

h/d

ay)


0

0.01

0.02

0.03

0.04

0.05

0 0.01 0.02 0.03 0.04 0.05LF

ER

Savin

gs (

therm

s/d

ay)



5. Model Comparison ResultsTable 6 shows options to move forward with the modelling for the DTE 201107 wave. Both options

result in an unbiased estimate of annual savings.

Options Implication for BRM Pros Cons

1 LFER, original

data for all

waves

LFER would be selected as the model

approach for calibration and all new measure

submissions

- Does not require any imputation of

missing data

- Achieves consistency in data cleaning

steps across all waves (i.e., no data

cleaning exceptions/additions are required

for DTE 201107 with this approach)

- Results in unbiased standard errors for

all waves

- Would achieve a result in which the model class

used (LFER) is inconsistent with most other

jurisdictions

- Across the program, standard errors are higher

for the LFER model than the LDV (not

considering imputed data)

2 LDV, imputed

data for DTE

201107

LDV would be selected as the model approach

for calibration and all new measure

submissions

Pre-period data would be imputed for the

missing portion of the DTE 201107 wave, but

would not be for other waves. BRM would

specify how to do this

- Moves BRM to using LDV model class

rather than LFER model class aligning with

most other jurisdictions

- Across the program, standard errors are

lower for the LDV model than for LFER

(not considering imputed data)

- Imputing data biases the standard error

downward (i.e., implies the estimate is more

precise than it is) for the DTE 201107 wave by

giving the model more information than we

actually have

- Requires additional data processing steps of

imputing data for DTE 201107 wave relative to all

other waves

Table 6. Model Options for DTE 201107 Wave



In general, savings increase and then level off for each usage band over time and are generally higher for higher usage bands than lower usage

bands. These results are consistent with evaluated results across many jurisdictions reflecting program ramp-up and the larger savings

opportunities among higher users.

6. Calibrated BRM Savings Values - LDVCalibrated BRM values, accounting for uplift, are shown below from the LDV model (with imputed

data for the DTE 201107 wave).

Table 7. Calibrated BRM Usage Bands - LDV

Fuel Usage Band Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9

Elec 5-7 MWh 0.12% 0.59% 0.87% 0.77% 0.89% 0.93%

Elec 7-9 MWh 0.72% 1.27% 1.29% 1.81% 1.64% * 1.98% ** **

Elec 9-11 MWh 0.78% 1.36% 1.50% 1.42% 1.24% 1.59% * 2.02% **

Elec 11-13 MWh 0.90% 1.42% 1.66% 1.84% 1.54%

Elec >13 MWh 1.13% 1.88% 2.19% 2.11% 2.19% 1.98%

Gas 601 - 900 Therm 0.27% 0.45% 0.51% 0.27%

Gas 901 - 1200 Therm 0.47% 0.68% 0.71% 0.74% 0.75% 0.73% 0.89% ** **

Gas >1200 Therm 0.41% 0.74% 0.73% 0.82% * * * 0.82% **

* These savings values have not changed from the deemed values in the current BRM as there was not enough data to calibrate them.

**Values have been proposed by DTE/Oracle for these usage bands.

Data for the DTE 201107 wave is imputed for the missing portion of the pre-period.

Note, when a usage band/year combination does not exist in the BRM for a wave for which savings are being claimed, we recommend the utility claim the last year available

in the BRM for that usage band. For example, if a utility would like to claim savings for a wave in Year 7 in the 5 – 7 MWh band, we recommend using the Year 6 value,

0.93%.



Similar to the LDV results, savings increase and then level off for each usage band over time and are generally higher for higher usage bands

than lower usage bands. These results are consistent with evaluated results across many jurisdictions reflecting program ramp-up and the

larger savings opportunities among higher users.

6. Calibrated BRM Savings Values - LFERCalibrated BRM values, accounting for uplift, are shown below from the LFER model.

Table 8. Calibrated BRM Usage Bands - LFER

Fuel Usage Band Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9

Elec 5-7 MWh 0.13% 0.57% 0.89% 0.81% 0.83% 0.88%

Elec 7-9 MWh 0.70% 1.11% 1.29% 1.78% 1.62% * 2.33% ** **

Elec 9-11 MWh 0.79% 1.45% 1.59% 1.64% 1.40% 1.73% * 2.29% **

Elec 11-13 MWh 0.96% 1.42% 1.67% 1.75% 1.41%

Elec >13 MWh 1.27% 1.98% 2.10% 1.99% 2.02% 1.95%

Gas 601 - 900 Therm 0.31% 0.36% 0.60% 0.35%

Gas 901 - 1200 Therm 0.44% 0.69% 0.65% 0.70% 0.72% 0.65% 0.59% ** **

Gas >1200 Therm 0.45% 0.75% 0.68% 0.81% * * * 0.58% **

* These savings values have not changed from the deemed values in the current BRM as there was not enough data to calibrate them.

**Values have been proposed by DTE/Oracle for these usage bands.

Note, when a usage band/year combination does not exist in the BRM for a wave for which savings are being claimed, we recommend the utility claim the last year available in

the BRM for that usage band. For example, if a utility would like to claim savings for a wave in Year 7 in the 5 – 7 MWh band, we recommend using the Year 6 value, 0.88%.



6. Comparison of Calibrated Savings Values to the BRMComparisons of calibrated values from the LDV model (with imputed data for the DTE 201107 wave)

and LFER model to the current BRM values for electric waves are shown below. Generally the newly

calibrated values are lower than the current BRM.Figure 13. Difference between BRM and Newly Calibrated Savings Rates - Electric

Note, while some calibrated results are not statistically different from zero, we recommend including the point estimate in the BRM as it is the best estimate available for the usage band/year at this time.



6. Comparison of Calibrated Savings Values to the BRMComparisons of calibrated values from the LDV model (with imputed data for the DTE 201107 wave)

and LFER model to the current BRM values for gas waves are shown below. The newly calibrated

values are mixed compared to the current BRM.

Figure 14. Difference between BRM and Newly Calibrated Savings Rates - Gas

Note, while some calibrated results are not statistically different from zero, we recommend including the point estimate in the BRM as it is the best estimate available for the usage band/year at this time.



6. BRM Savings Value RecommendationUsing either the LDV or LFER model, Guidehouse found a demand savings factor near or below 1.

Table 9. Calibrated BRM Demand Savings Factor

Model Energy

Savings

Demand

Savings

Demand

Savings Factor

LDV 0.86% 0.67% 0.78

LFER* 0.69% 0.70% 1.02

* The LFER energy savings are based on a weighted average of individual

wave / program year results.



• Reach consensus regarding model specification options outlined on slide 19.

• Develop and submit a modified measure workpaper for review by the EWR Collaborative’s Technical

Subcommittee, including:

– Updated energy and demand savings values based on calibration

– Specify the demand savings factor

– Provide additional specifications on the measure description (in particular, report frequency)

– Updates to the description for how to determine a wave’s usage band for calculating savings

– Model specification requirements for future new measure submissions, including how to handle missing

data

• Recommend calibration of later years’ savings values in 3-4 years when an additional 5-6 electric waves

and 2-3 gas waves could be included.

7. Next StepsGuidehouse will develop an updated measure workpaper for the BRM.


Contact

©2020 Guidehouse Inc. All rights reserved. This content is for

general information purposes only, and should not be used as

a substitute for consultation with professional advisors.

Carly Olig

Associate Director

[email protected]

Debbie Brannan

Director

[email protected]

Kathleen Ward

Managing Consultant

[email protected]

Cherish Smith

Associate Director

[email protected]

Rachel Marty

Managing Consultant

[email protected]

Jackson Lines

Senior Consultant

[email protected]


Appendices


Where:

• i indexes the customer

• t indexes time

• w indexes program wave

• 𝐴𝐷𝑈𝑖𝑡 is the customer’s average daily energy consumption during time t

• 𝐴𝐷𝑈𝑙𝑎𝑔𝑖𝑡 is the customer’s average daily energy consumption during the same month as time t lagged to the pre-program year

• 𝑌𝑟𝑀𝑜𝑗𝑡 comprise a set of month-of-year indicators, which equal 1 if

t falls in month-of-year j, and 0 otherwise

• 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 is a binary indicator that equals 1 if customer i is atreatment customer, and 0 otherwise

• 𝐶𝐷𝐷𝑖𝑡 are the cooling degree-days during time t for customer i

• 𝐻𝐷𝐷𝑖𝑡 are the heating degree-days during time t for customer i

• 𝑊𝑎𝑣𝑒𝑖𝑤 is a binary indicator that equals 1 if customer i falls inwave w, and 0 otherwise (only included in pooled models)

• 𝑈𝑡𝑖𝑙𝑖𝑡𝑦𝑖 is a binary indicator that equals 1 if customer i is a DTE customer, and 0 otherwise (only included in pooled models)

• The 𝛽1 − 𝛽6 are unknown parameters to be estimated

• 𝜀𝑖𝑡 is a mean-zero disturbance term

Appendix A. Energy Regression Model - LDVAn LDV model was used to calculate energy savings.

𝐴𝐷𝑈𝑖𝑡=

𝐽

𝛽1𝑗𝑌𝑟𝑀𝑜𝑗𝑡 +

𝐽

𝛽2𝑗𝑌𝑟𝑀𝑜𝑗𝑡 ∙ 𝐴𝐷𝑈𝑙𝑎𝑔𝑖𝑡 + 𝛽3𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 + 𝛽4𝐶𝐷𝐷_𝑎𝑛𝑑_𝑜𝑟_𝐻𝐷𝐷𝑖𝑡 +

𝑊

𝛽5𝑊𝑎𝑣𝑒𝑖𝑤

+ 𝛽6𝑈𝑡𝑖𝑙𝑖𝑡𝑦𝑖 + 𝜀𝑖𝑡


Where:

• i indexes the customer

• t indexes time

• 𝐴𝐷𝑈𝑖𝑡 is the customer’s average daily energy consumption during time t

• 𝛼𝑖 is the customer specific fixed effect capturing all time-invariant observable and unobservable customer characteristics

• 𝑃𝑜𝑠𝑡𝑡 is a binary indicator that equals 1 if observation t for customer i is after customer i’s HER wave started (only included

independently of treatment in pooled models)

• 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 is a binary indicator that equals 1 if customer i is atreatment customer, and 0 otherwise

• 𝑌𝑟𝑀𝑜𝑗𝑡 comprise a set of month-of-year indicators, which equal 1 if

t falls in month-of-year j, and 0 otherwise

• 𝐶𝐷𝐷𝑖𝑡 are the cooling degree-days during time t for customer i

• 𝐻𝐷𝐷𝑖𝑡 are the heating degree-days during time t for customer i

• The 𝛽1 − 𝛽4 are unknown parameters to be estimated

• 𝜀𝑖𝑡 is a mean-zero disturbance term

Appendix A. Energy Regression Model - LFERAn LFER model was used to calculate energy savings.

𝐴𝐷𝑈𝑖𝑡= 𝛼𝑖 + 𝛽1𝑃𝑜𝑠𝑡𝑡 ∙ 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 +

𝐽

𝛽2𝑗𝑌𝑟𝑀𝑜𝑗𝑡 + 𝛽3𝐶𝐷𝐷_𝑎𝑛𝑑_𝑜𝑟_𝐻𝐷𝐷𝑖𝑡 + 𝛽4𝑃𝑜𝑠𝑡𝑡 + 𝜀𝑖𝑡


• The model for demand savings used only the data from the coincident peak demand period (i.e., the three

hottest, consecutive weekdays in July from 3 to 6 p.m.).

• For the LDV model, the usage lag (ADUlagit) is defined as the average usage during the same hour on the

three peak days from the pre-program year.

Appendix A. Demand Regression ModelAnalogous models were used to calculate demand savings.


Appendix B. Upstream Lighting - DTE 2019 Survey1

Over three-quarters of respondents said they purchased small energy efficiency devices in PY2019. LED light

bulbs were the most common measure purchased by over 90 percent of small energy efficiency device

purchasers. There was little difference between recipient and control group customers in rates of purchasing

the different small energy efficient devices.Small EE Devices

» Over three-quarters of HER recipient (76 percent) and control (78 percent) group customers reported purchasing small energy efficient devices in 2019.

» LED light bulbs were the most common measures installed, with over 90 percent of small energy efficiency device purchasers reporting buying this measure. The difference in purchase rates between recipient and control group customers was not statistically significant (Figure 1-12). The majority of treatment and control customers purchase bulbs from participating Upstream Lighting retailers. These findings do not support the hypothesis that HER savings are primarily attributable to lighting.

» Rates of purchasing all other small energy efficient devices were similar for treatment and control group customer.

FIGURE B-1. HOME ENERGY REPORT

Small EE Purchases (Recipient vs. Control)2, n=1165

3%

4%

11%

12%

12%

15%

95%

3%

4%

12%

10%

14%

15%

94%

Other non-connected devices

Other smart home devices

Smart outlets

Smart or WiFi-enabled thermostat

Advanced power strip

Programmable thermostat

LED lightbulbs

Recipient (n=588) Control (n=577)

2 Survey results were weighted by HER wave to ensure that survey sample was

representative of the program population. See slide 22 in PY2019 report for more detail on how wave-based weights were constructed.

Source: 2019 year-end HER recipient and control survey data.

1 Due to changes to the program survey, results should not be compared across 2018

and 2019. However, the HER recipient and control groups can be accurately compared

within each year.


Appendix B. Upstream Lighting - DTE 2018 Survey1LED bulbs remain the most popular energy efficient purchase among respondents who reported making an

energy efficient purchase in the last 12 months.

• Generally, HER recipients report engaging in energy efficient

purchases more often than their control counterparts (Figure 4-16).

• The following energy efficient appliance are those which HER

recipients reported purchases at a higher rate than their control

counterparts:

– Freezers, 80 percent confidence

– Furnace Fans, 80 percent confidence

– Insulation, 80 percent confidence

– Other non-LED lighting, 90 percent confidence

• LED bulbs are the most common energy efficient item purchased

by both HER recipients and their control counterparts at almost

identical rates which may suggest that the HER tips designed to

motivate purchases of LED bulbs are not effective. It is also

possible that general consumer preferences of DTE customers has

shifted towards LED bulbs and thus there is no purchasing habit

modification potential.

Energy efficient appliance purchasesFigure B-2. HOME ENERGY REPORT

Reported Energy Efficiency Purchases by

Appliance (Recipient vs. Control), n = 210

41%

17%

15%

13%

7%

9%

10%

9%

5%

6%

3%

41%

18%

11%

8%

10%

7%

6%

6%

3%

3%

1%

0% 10% 20% 30% 40% 50%

LED Light Bulbs

Electronics

Programmable Thermostat

Other Lights**

Clothes Dryer

Dishwasher

Insulation*

Water Heater

Dehumidifier

Furnace Fan*

Freezer*

Control Treatment

Source: 2018 monthly HER recipient survey data

1 Due to changes to the program survey, results should not be compared across 2018

and 2019. However, the HER recipient and control groups can be accurately compared

within each year.


3 DNV-GL. 2014. National Grid Residential Building Practices and Demonstration Program Evaluation, Final Results.

http://viget.Opower.com/company/library/verification-reports.

4 DNV-GL. 2018. PSE Home Energy Reports Program. 2017 Impact Evaluation – Final Report. https://conduitnw.org/Pages/File.aspx?rid=4415

5 DNV-GL. 2017. Review and Validation of 2015 Pacific Gas and Electri

microsoft word - 37-rto workpaper_041816.docx · web view2020/10/26 · this whitepaper proposes...

Documents