EvergreenEcon.com

ESA 2011 Impact EvaluationResearch Plan Public Workshop #1

February 20, 2013

Presented By: Steve Grover, President

2

Agenda for Today

Introductions / Opening Remarks (15 min)

Presentation of Draft Research Plan (30 min)

Q&A / Discussion on Research Plan (1 hour)

Break for Lunch

Discussion on Research Plan (Continue)

Wrap Up / Action Items (15 min)

Evaluation Team

The ESA Impact Evaluation team consists of the following:

Evergreen Economics (Prime Contractor)

Michaels Energy (Engineering Support)

CIC Research (Phone Surveys)

Betsy Wilkins (Technical Editor)

John Stevenson (Survey Design)

3

ESA Impact Evaluation Tasks

Task 1: Project Management

Task 2: Kickoff Meeting

Task 3: Research Plan

Task 4: Public Workshop #1

Task 5: Data Collection and Analysis

Task 6: Draft Report

Task 7: Public Workshop #2

Task 8: Final Report

Task 9: Data Documentation

4

Task 3: Research Plan

The ESA Draft Research Plan available for public comment on the CPUC website www.energydataweb.com.

The Research Plan includes the following:

• Complete discussion of analysis methods and project tasks

• Discussion of project issues and pros/cons of approach

• Sampling plan for phone surveys

• General report outline

• Timeline for key deliverables

• Description of project management procedures and communication protocols

Comment period ends February 22, 2013.

5

Task 5: Data Collection and Analysis

The primary analysis method will be a billing regression model:

•Utilizes a fixed effects model specification (same as prior evaluations)

•Will use model from the last impact evaluation as a starting point (will allow for comparisons with last 2 evaluations)

•More resources devoted to exploring other billing regression options, data screening methods, etc.

•Model will produce impacts by utility, house type, and measure type/group (same as last evaluation)

•Will explore using hourly (rather than daily) temperature data to calculate CDD and HDD variables.

6

Fixed Effects Model (General Form)

7

Fixed Effects Model Advantages

Primary model advantages :

•Fixed effects specification contains the desirable statistical properties for producing unbiased estimates

•Allows for controlling some of the non-program influences that affect savings

•Provides opportunity to compare current impact estimates with prior ESA evaluations

•Changes in savings estimates from the last evaluation due to weather and/or household energy use can be tracked.

8

Two-Stage Regression Model

A second model will be estimated that utilizes a two-stage format:

•Stage 1: Estimate whole-house savings using a fixed effects regression model.

•Stage 2: Estimate share of household savings from Stage 1 attributable to individual measures using an ordinary least squares (OLS) regression model.

The two-stage model may provide a better opportunity to develop measure-level impacts.

9

Two-Stage Regression Model (2nd Stage)

10

Data Screening for Regression Models

Some data screening will be needed for any regression model.

Screens that we will explore include:

•Monthly usage less than or equal to zero

•Very high monthly usage (threshold levels TBD)

•Customers with less than 12 months of pre-installation billing data

•Customers with less than 12 months of post-installation billing data (may relax to 10 months)

•Customers with very large changes in energy use between pre and post periods (thresholds TBD)

All screenings and their effect on savings will be clearly documented.

11

Task 5: Phone Surveys

Phone Surveys

• Participant phone surveys will be conducted (150 per utility, 600 total)

• Due to limited sample, surveys will focus on sample of customers with increased energy use after ESA participation.

• Questions will be designed to collect additional information that might help explain increased consumption:

• Changes in how equipment is used after ESA participation

• How equipment is used in extreme weather

• Condition of existing equipment

• Demographic information / changes to household

12

Task 5: Engineering Review

Engineering review

• Review savings estimates if needed (both ex ante and ex post)

• Help attribute savings to individual measures if only measure group savings can be estimated from the billing regression

• Provide insights and analysis from on-sites from previous ESA evaluation (if needed)

13

Task 6: Draft Report

Draft Report will document all analysis and results

• Similar to last evaluation report; summary information contained up front with detailed information included as appendices

• Report outline included in the Research Plan

• Our technical editor Betsy Wilkins will edit the draft report prior to delivery

14

Task 7: Public Workshop #2

Public Workshop covering the Draft Report will include:

• Overview of analysis methods and results

• Draft report will be posted on CPUC website prior to Public Workshop

• Meeting materials (e.g., agenda, presentation) will also be made publicly available prior to the Workshop.

• Majority of meeting will be spent fielding questions and comments on the Draft Report

• We will take notes on the Workshop and summarize key comments and action items.

15

Task 8: Final Report

Final Report

•Will address questions and comments received during the Public Workshop #2

•Also will address comments and questions from the IOUs and Energy Division

•Will be delivered no later than August 31, 2013

16

Project Management / Data Documentation

Project Management (Task 1)• Weekly conference calls with SDG&E Project Manager

• Monthly written progress reports

• Weekly data collection dispositions while surveys are in the field

• Other ad hoc calls and meetings as needed throughout the project.

Data Documentation (Task 9)

• All data files and programs will be provided to the Energy Division at the end of the project

• Datasets will include a data dictionary that defines each variable in the dataset

• A separate document will list all the programs used to compile the raw data, run the billing regression models, and produce the final impact estimates.

• Documentation will be comprehensive so that an independent third party can replicate the results.

17

Project Timeline (Highlights)

Key Dates: Draft Research Plan Posted: February 14

Public Workshop #1: February 20

Research Plan Comment Period Ends: February 22

Final Research Plan: March 1 (estimated)

Participation and billing data compiled from IOUs: March 1

Data Collection and Analysis Complete: May 31, 2013

Draft Report: July 7, 2013

Public Workshop #2 on Draft Report: TBD

Final Report: August 31, 2013

18