review of reported efficiency savings for program …€¦ · web view2019/05/24 · most notably,...

Report Group D – D0.01

Review of Reported Efficiency Savings for Program Year 2017: Custom Projects

Submitted to California Public Utility Commission505 Van Ness AvenueSan Francisco, CA 94102

Submitted by SBW Consulting, Inc.2820 Northup Way, Suite 230Bellevue, WA 98004

In association with BuildingMetrics Inc.Energy350Jai J Mitchell

AnalyticsKolderup Consulting

Ridge & AssociatesSteve KromerSugarPineTierra Resource Consultants

May 24, 2019

Review of Reported Efficiency Savings For Program Year 2017: Custom Projects

1

SBW Consulting, Inc. ii


AcknowledgementsWe want to acknowledge the many contributions that made it possible for our team to complete this review quickly. Peter Lai, from California Public Utilities Commission (CPUC) Energy Division staff, guided the development of the work plan, provided critical review, and clarified CPUC policies and relevant practices. Members of program administrator staff commented on our work plan, provided the necessary files to document sampled projects, and responded quickly to our numerous requests for supplemental project information.

SBW Consulting, Inc. iii


Table of ContentsList of Tables...............................................................................................

List of Acronyms and Abbreviations.............................................................

Executive Summary.....................................................................................Methodology...............................................................................................................Findings.....................................................................................................................Recommendations......................................................................................................Structure of this Report............................................................................................

1 Background........................................................................................1.1 Prior Evaluations.............................................................................................1.2 Issues Identified in ESPI Ex Ante Review Performance Memos..........................1.3 CPUC Policies and Guidance.............................................................................

2 Review of Claims Database.................................................................2.1 Data Request..................................................................................................2.2 Review Reported Claims..................................................................................

2.2.1 Claim Year............................................................................................................2.2.2 Classification Based on Measure Description...........................................................2.2.3 Early-Retirement-Measure Parameters...................................................................2.2.4 Net-to-Gross Ratios...............................................................................................2.2.5 Effective Useful Lifetimes.......................................................................................2.2.6 Gross Realization Rates.........................................................................................2.2.7 Installation Rates..................................................................................................

2.3 Calculate Lifetime Net Savings.........................................................................

3 Sample Selection................................................................................

4 Review of Sampled Project Files..........................................................4.1 Data Requests.................................................................................................

4.1.1 Review Project Files...............................................................................................4.1.2 Request Missing Data, Models or Documents..........................................................

4.2 Review Project Files.........................................................................................4.2.1 Eligibility-Related Savings Adjustments..................................................................4.2.2 Measure Application Type......................................................................................4.2.3 Deemed Measures in Custom Projects....................................................................4.2.4 Gross Efficiency Savings........................................................................................4.2.5 Net Efficiency Savings............................................................................................

4.3 Calculate Lifetime Net Savings.........................................................................

5 Findings.............................................................................................5.1 Custom Projects Portfolio.................................................................................

SBW Consulting, Inc. 4


5.2 PG&E Custom Projects.....................................................................................5.2.1 Cumulative Changes from Database and Sample Review.........................................5.2.2 Eligibility-Related Adjustments for Sampled Projects..............................................

5.3 SCE Custom Projects........................................................................................5.3.1 Cumulative Changes from Database and Sample Review.........................................5.3.2 Eligibility-Related Adjustments for Sampled Projects..............................................

5.4 SDG&E Custom Projects...................................................................................5.4.1 Cumulative Changes from Database and Sample Review.........................................5.4.2 Eligibility-Related Adjustments for Sampled Projects..............................................

5.5 SCG Custom Projects.......................................................................................5.5.1 Cumulative Changes from Database and Sample Review.........................................5.5.2 Eligibility-Related Adjustments for Sampled Projects..............................................

6 Conclusions and Recommendations.....................................................6.1 Applicable to All Reported Claims.....................................................................6.2 Applicable to Specific PAs................................................................................

6.2.1 PG&E....................................................................................................................6.2.2 SCE.......................................................................................................................6.2.3 SDG&E..................................................................................................................6.2.4 SCG......................................................................................................................

7 Data Products.....................................................................................

Appendix A: Stakeholder Comments on Work Plan......................................



List of TablesTable 1: Correction to Savings from Reported Claims...................................................................Table 2: Correction to Savings Due to Database Review and Sampled Project File Reviews.............Table 3: Example of Keyword List...............................................................................................Table 4: Definition of Measure Application Types........................................................................Table 5: Number of Reported and Sampled Projects and Claims..................................................Table 6: Reported Lifetime Net Savings and Percent Sampled for Project File Review...................Table 7: Change in Savings Due to Database Review and Sampled Project File Reviews...............Table 8: Cumulative Changes Due to Database and Sample Review for PG&E...............................Table 9: Number of Sampled Claims with Eligibility-Related Savings Adjustments for PG&E.........Table 10: Cumulative Changes Due to Database and Sample Review for SCE...............................Table 11: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SCE..........Table 12: Cumulative Changes Due to Database and Sample Review for SDG&E...........................Table 13: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SDG&E......................................................................................................................................Table 14: Cumulative Changes Due to Database and Sample Review for SCG...............................Table 15: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SCG.........



List of Acronyms and AbbreviationsAcronymAbbreviation Description

CEDARS California Energy Data and Reporting System

CPUC California Public Utilities Commission

DEER Database for Energy Efficient Resources

EAR Ex ante Review

EM&V Evaluation Measurement and Verification

ER Early Retirement

ESPI Efficiency Savings and Performance Incentive

EUL Effective Useful Life

GRR Gross Realization Rate

GW Gigawatts

GWh Gigawatt-Hours

IOUs Investor Owned Utilities

ISP Industry Standard-Practice

kW Kilowatts

kWh Kilowatt-Hours

M&V Measurement and Verification

MMBtu Millions of British Thermal Units

MW Megawatts

MWh Megawatt-Hours

NC New Construction

NR Natural Replacement

NRNC Nonresidential New Construction

NTG Net-to-Gross

NTGR Net-to-Gross Ratio

PA Program Administrators

PG&E Pacific Gas and Electric

QA/QC Quality Assurance/Quality Control

READI Remote Ex ante Database Interface

RC Reach Code

ROB Replace on Burnout

RUL Remaining Useful Life

SCE Southern California Electric

SCG Southern California Gas

SDG&E San Diego Gas and Electric

UTCUse category, use sub-category, Tech group and tech type, all used in classifying a claim for the purpose of associating NTG or EUL DEER values.



Executive SummaryIn this report, we—the SBW team—describe how we completed the Review of Reported Efficiency Savings for Program Year 2017: Custom Projects (Industrial, Agricultural and Large Commercial). Our objective was to perform a due-diligence review of the net lifecycle savings—kilowatt-hour (kWh), kilowatt (kW), and therm—associated with custom projects reported by program administrators (PA) during 2017. We reviewed all the reported claims and documentation for a random sample of projects to examine and determine if they comply with California Public Utilities Commission (CPUC) policies and rules. In the absence of the 2017 ex post evaluation activity, the results from this review1 serve as a proxy for the 2017 ex post results in the calculation of the 2019 Efficiency Savings and Performance Incentive (ESPI) awards.

MethodologyWe used two approaches to review reported savings. The first involved a database review of all claims records associated with 2017 custom projects. The second involved review of relevant files associated with a sample of projects.

Our database review of all claims records was limited to the data submitted by the PAs to the California Energy Data and Reporting System (CEDARS). To the extent allowed by this data, we examined whether the claims conformed to the CPUC policies in effect for 2017, and, to the extent that the claims differed from those policies, we corrected portions, such as 1st or 2nd baseline2 gross savings, net-to-gross ratio (NTGR) and effective useful life (EUL), as necessary to bring them into alignment with applicable policies.

For our file review we requested and reviewed relevant documentation associated with a random sample of projects. The sample accounted for 49% of the lifecycle net energy savings of the reported claims when electric and gas savings were combined into millions of British thermal units (MMBtu). To support our review, we requested supplemental information from the PAs for specific projects. The requested files enabled us to more fully ensure compliance with CPUC policies. Our findings from the project file review superseded any findings that resulted from our initial database review.

Since this review is not equivalent to an ex post evaluation of the 2017 claims, the CPUC determined that the results from the file review of sampled projects would not be extrapolated. Therefore, claims for projects not included in the sample were only changed based on our database review. Our work plan for this review was distributed for stakeholder comment. Our

1 CPUC Decision 13-09-023, Decision Adopting Efficiency Savings and Performance Incentive Mechanism, San Francisco, CA, California Public Utilities Commission, 2013. Attachment 6 directs the annual process for review and resolution of the ESPI awards payments. Specifically, on page 1, “All custom projects and a specific subset of deemed measures with parameters identified as highly uncertain will not be locked down during the portfolio cycle and will be subject to ex post review in order to determine the applicable savings award.” Attachment 6, page 2 further describes staff’s “due diligence” review and correction of errors in the utilities’ reported claims estimates.

2 2nd baseline savings are required for early retirement measures.

SBW Consulting, Inc.


responses to stakeholder comments are provided in Appendix A: Stakeholder Comments on Work Plan.

FindingsBased on our review, we corrected the lifetime net savings reported by the PAs for 2017 custom projects. Table 1 summarizes our findings by PA and for the statewide PA portfolio. Except for SDG&E’s MW and therms claims, the corrected values were lower than reported (MWh, MW and therms) by between 1% and 64%. For the statewide PA portfolio, the corrections reduced the reported claimed savings by 15% for MWh, 28% for MW, and 13% for therms.

Even though SCE does not supply gas and SCG does not supply electricity, they each claimed savings for those other fuels. This is may be due to interactive effects, such as changes in lighting energy use causing changes in heating energy use. In other cases, it is due to the PA claiming savings that accrued to a municipal utility serving the other fuel.

Table 1: Correction to Savings from Reported Claims

Lifetime Net Savings

PG&E SCE SDG&E SCG Total

MWh

Reported Claims 1,364,619 894,335 164,824 31,689 2,455,467

Database and Sample Review 1,284,843 633,559 143,116 15,140 2,076,658

% change from Claim -6% -29% -13% -52% -15%

MW

Reported Claims 214 117 24 10 366

Database and Sample Review 150 81 27 4 262

% change from Claim -30% -31% 10% -64% -28%

Therms

Reported Claims 63,614,914 1,305,975 7,671,599 18,648,295 91,240,782

Database and Sample Review 62,787,667 627,179 7,932,992 8,026,293 79,374,131


Table 2 shows the corrections we made based on the database review and then how those corrections were further adjusted when we combined the file review of sampled projects.

Table 2: Correction to Savings Due to Database Review and Sampled Project File Reviews



MWh

Reported Claims 1,364,619 894,335 164,824 31,689 2,455,467

Database Review 1,296,740 601,916 153,695 16,272 2,068,622








MW


Database Review 147 77 23 5 252




Therms

Reported Claims 63,614,914 1,305,975 7,671,599 18,648,295 91,240,782

Database Review 56,386,449 638,853 7,625,305 5,116,958 69,767,565




RecommendationsIn summary, our recommendations are as follows:

1. Prohibit projects installed in other years. CEDARS should only allow submission of claims installed in prior years if the PA certifies that the delay was due to M&V. No claims should be allowed for future years.

2. Improve measure descriptions. Measure descriptions should be concise, cite the system, equipment or practice that was changed; describe the change in a way that clearly indicates why it is more efficient; and, indicate whether the measure was early retirement. The CPUC should develop and widely disseminate a guide to the creation of best practice measure descriptions.

3. Connect ex ante review (EAR) dispositions to CEDARS claims. EAR dispositions should be associated with unique project identifiers, and PAs should be required to maintain the same project identifier throughout a custom project’s lifecycle from initial application to submission to CEDARS.

4. Require reproducible savings claims. The CPUC should require that all custom projects have a set of files that enables an appropriately trained reviewer to reproduce the claimed savings.

5. Improve tables for the database for energy efficient resources (DEER), net to gross (NTG), and EUL. The CPUC should revise these tables so that each row is uniquely



defined by a combination of classification parameters, such as delivery type, sector, use category, use sub-category, tech group and technology type. The possible values for the classification parameters must be non-overlapping and comprehensive.

6. Confirm claims are nonoverlapping and accrue to an IOU. For each project, documentation should be provided to confirm that the claimed savings do not overlap with another PA’s claim, and that all savings are accrue to one of the four California investor-owned utilities (IOU) energy suppliers.

Structure of this ReportThe balance of our report consists of the following sections:

Background. This section summarizes findings from prior evaluations, as well as issues raised in recent ESPI performance memos and relevant CPUC policies and guidance.

Review of Claims Database. This section describes the methods we used to examine and determine whether all reported claims conformed to CPUC policies and guidance. It also explains the corrections we made for those claims that did not conform. This review was based only on the data provided in the CEDARS claim.

Sample Selection. This section describes how we selected a random sample of projects for more detailed review of the supporting files.

Review of Sampled Project Files. This section describes the methods we used in our review of sampled project files, as well as the changes we made based on findings from our review.

Findings. This section summarizes the corrections we made to the entire portfolio of reported custom project claims for 2017. It also provides details regarding the frequency and impact of the corrections made for each PA.

Conclusions and Recommendations. This section provides our recommendations to improve claim reporting, review of reported claims, ex ante review, and ex post evaluation of custom projects.

Data Products. This section describes the data products that we have prepared to assist the PAs in their review of this report.



2 BackgroundIn this section, we summarize our understanding of the findings from prior evaluations of custom projects, as well as issues raised in recent ESPI Ex Ante Review Performance memos, and relevant CPUC policies and guidance.

2.1 Prior Evaluations Custom programs were evaluated in 2013, 2014, and 2015.3 The results of these evaluations were remarkably similar. Both gross energy-realization rates and NTGRs hovered around 0.5 to 0.6. The reasons for the relatively low realization rates and NTGRs were also consistent across the studies. The primary reasons include:

Errors in the calculation methods of the PAs

Inappropriate baseline specifications

Savings claims for ineligible projects

Use of default operating conditions instead of forecast or observed operating conditions.

Recommendations across the studies paralleled these reasons and included:

The IOUs should improve calculation methods and protocols to increase the accuracy of savings estimates.

The IOUs should improve the documentation and reporting of the EUL of projects and the remaining useful life (RUL) of early retirement (ER) projects.

The IOUs should improve the quality control of project operating conditions, ex ante baseline determinations, savings calculations, and eligibility rules.

Adjust project savings based on post-installation inspections and measurement and verification (M&V).

PAs should consider adopting program-implementation procedures and features designed to increase program influence.

2.2 Issues Identified in ESPI Ex Ante Review Performance Memos

Issues identified during ex-ante review were documented in the ESPI EAR performance memos. Recurring themes across PAs and program years included:

3 The 2013 study separated custom retrofit and nonresidential new construction (NRNC) projects as separate reports, while the 2014 and 2015 reports combined custom retrofit and NRNC projects.



Calculation tools. Issues with the EnergyPro software used to calculate savings for the Savings by Design program were identified on projects submitted by PG&E, SCE and SDG&E in late 2017.

Quality assurance/quality control (QA/QC). Persistent issues with internal due diligence review were identified across all PAs.

Measure life. Persistent problems with the correct assignment of measure effective useful life and remaining useful life for accelerated replacement projects were identified. EULs for retrofit add-on measures that exceed the RUL of the host equipment were also identified.

Program influence documentation. Inadequate documentation of program influence was a persistent problem across all PAs.

M&V. EAR dispositions required additional M&V on multiple projects and measure types for which measured performance data was needed to estimate savings. Measures requiring additional M&V included: guest room energy management systems, non-DEER lighting projects, and first-time projects with little performance history.

Problematic measures. Issues were identified with savings calculations for several measure types, requiring improvements to calculations, input parameters, or documentation. These measures included: chiller optimization, air compressor plant projects, oil pumping, monitoring-based commissioning, variable air volume conversions, and waste water treatment plants.

2.3 CPUC Policies and Guidance An important goal of this study is to determine whether the utility-reported energy savings for 2017 custom projects conform to applicable CPUC policies and guidance. Due to the accelerated timeframe for this review, we did this only for an important subset of these policies and guidance from the following documents:

CPUC Energy Efficiency Policy and Procedures Manual (v. 5 or the most recent version)

Utility Statewide Custom Policy and Procedures manual

Statewide Savings by Design Policy and Procedures manual

PA-specific policy and procedures manuals

Industry standard-practice (ISP) studies completed and adopted before or during 2017

Title 20 and 24 requirements in place when projects were permitted

CPUC policy papers and state-government memos addressing topics such as the EUL/RUL -preponderance-of-evidence requirements and non-IOU fuel sources

CPUC resolution E-4818 affecting assignment of project baselines

Review dispositions of custom projects issued by CPUC staff



NTGR and EUL/RUL support tables downloaded from READI v.2.5.1 available at http://www.deeresources.com/index.php/deer-versions/readi


http://www.deeresources.com/index.php/deer-versions/readi


3 Review of Claims DatabaseWe reviewed all savings claims associated with custom measures reported by the PAs for 2017 to examine and determine whether these claims conformed to CPUC policies and guidance. Claims were corrected only when clear evidence required a correction. The following subsections describe the criteria used to determine if savings claims complied with CPUC polices and guidance, as well as the procedures for updating any claims that do not comply.

3.1 Data RequestData from CEDARS does not provide documentation to justify installation dates prior to the claim year, in this case, 2017. On March 28, 2019, the CPUC sent emails directing PAs to provide a list of reported claims for 2017 for which extended post installation M&V was conducted. In response to this request, PAs submitted documentation to substantiate any claims that were installed prior to 2017 for which they sought exemptions.

3.2 Review Reported Claims

3.2.1 Claim YearThe CPUC directed the PAs to include only savings for measures that were installed in the same program year for which they are reporting. The CPUC provided further guidance to the PAs that savings towards ESPI payments will be counted4 only for measures that were installed in the program year for which ESPI payments are being requested. The guidance requires the PAs to identify the installation year for all projects that are included in their claim for that program year. Additionally, the CPUC directed PAs to identify claims with justifiable exceptions to the rule regarding installation-dates. For example, there were claims with installation dates in years more recent than 2017. These claims were not eligible for an exception. Exceptions were granted to all claims on the lists that the PAs provided in response to the data request described in section 3.1.

3.2.2 Classification Based on Measure DescriptionWe examined information in the measure description field of the claims database and corrected any errors in the values of the following fields: use category, use sub-category, tech group, and tech type.5 These corrections led to changes to NTGR, EUL, and RUL values to comply with CPUUC policies and rules.

4 In the data set created for ESPI, we had to set these claims savings to zero. Removing them would have caused ESPI processing to errors. Having these savings claims set to zero effectively removes them from the utility’s claim.

5 The CPUC’s Remote Ex Ante Database Interface (READI) utilizes use category and sub use -category to define how or where a technology is used. All technologies that are referenced in measure definitions are further organized into tech groups and tech-types (i.e. a subcategory of tech group).



The measure descriptions are used to establish each claim’s NTG, EUL, RUL, and measure application type. For each claim, we compared the measure description to the DEER ex ante, NTG, EUL and RUL values6 that were adopted for use in 2017. This enabled us to map the measure descriptions to specific field values for use category, use sub-category, tech group, and tech type (herein referred to collectively as “UTCs” for brevity).

To ensure claims were correctly categorized according to their UTCs, we developed a keyword list of names and terms that are common to each measure description use category. This provided the highest level of classification for claims and was used to match claims to adopted values in the ex ante database. We developed the keyword list from the measure description of each claim and from the use categories that were contained in the ex ante database7 measure list that was adopted for use in 2017. Table 3 gives examples of keyword matches.

Table 3: Example of Keyword List

Ex Ante Database Use Category Claims Database Measure Description KeywordsHVAC (HVAC) Chiller, HP, heat pump, package, AC

Appliance or plug load (AppPlug) Television

Building envelope (BldgEnv) Envelope, SHGC, insulation, weatherization, shell, windows, fenestration

To ensure the correct label was assigned we searched the measure descriptions in the claims database for common keywords and mapped them to the appropriate use category. If a claim was found to have a correct use category, then that use category was assigned and used to match the claim to any other remaining UTC values. If a claim had an incorrect use category, then an appropriate use category was assigned based on the keyword mapping and then used to match the claim to the remaining UTC values. If a use category could not be matched based on information contained in the measure description, then the values for all four UTCs in the claims database were accepted as originally submitted in the claim.

Once a use category was matched for a claim, we scanned the measure description field again to identify any subsequent keywords associated with the use sub-category, tech group, and tech type, and then corrected those values in a similar manner until we successful matched the claim to a unique measure in the ex ante database.

Upon completion all claims were associated with four UTCs, either updated to a corrected value based on our keyword mapping, or unmodified if we were unable to match a keyword in the claim’s measure description to any of the four UTCs.

6 NTGR and full EUL and RUL support tables downloaded from READI v.2.5.1 available at: http://www.deeresources.com/index.php/deer-versions/readi

7 READI - Remote Ex Ante Database Interface version 2.5.1. Available at ftp://deeresources.com/DEER/READI_v2.5.1.zip




3.2.3 Early-Retirement-Measure ParametersMeasure parameters must be consistent between measure application type and reported energy savings. Table 4Error: Reference source not found8 below shows measure application types and associated savings baselines.

Table 4: Definition of Measure Application Types

Measure Application Type Measure Life Basis

First Period Energy Savings Baseline (Rul)

Second Period Energy Savings Baseline (EUL-RUL)

New Construction (NC) EUL Code or Industry Standard Baseline

N/A

Replace on Burnout (ROB)

EUL Code or Industry Standard Baseline

N/A

Natural Replacement (NR)

EUL Code or Industry Standard Baseline

N/A

Early Retirement / Retrofit (ER/RET)

RUL/EUL-RUL

Customer Existing Baseline

Code or Industry Standard Baseline

Add-On Retrofit (REA) RUL or EUL Customer Existing Baseline

N/A

Measures whose measure application type is early retirement must have energy savings reported for the first and second baselines. Similarly, measures with energy savings reported for both the first and second baseline must have the measure application type set to early retirement. For all measures that had different energy savings values in the first and second baseline periods, we resolved inconsistencies between the measure application type and energy savings fields by ensuring that the measure application type was correctly set to early retirement.

We investigated the number of claims in the claims database that had measure application type marked as early retirement since it indicates the primary conditions under which the measure was installed and establishes the correct baseline and potential use of RUL. We found that the measure descriptions did not provide sufficient detail to determine whether the measure application type was correctly entered. Therefore, for purposes of the database review, we assumed the PA correctly entered the measure application type. For any claims marked as early retirement that lacked an RUL value we assigned a value of zero for the RUL period.

3.2.4 Net-to-Gross RatiosThe DEER NTG table provides approved NTGR values for kWh and therm savings that apply to combinations of the following fields: building type, building vintage, measure delivery, sector, use category, use sub-category, tech group, and tech type. We updated NTGR values for kWh

8 Baselines are sourced from CPUC document: Early Retirement Using Preponderance of Evidence, Version 1.0, July 16, 2014.



and therms based on corrected values for these fields. We used the corrected values to find the appropriate NTGR value that applies to a unique measure in the ex ante database9. We only applied NGTR values that the CPUC approved for use between January 1, 2017 and December 31, 2017.

We used the methodology described in section 3.2.2 to determine the correct UTCs for each claim. Once we determined the correct UTCs, we associated the updated NTGR values for each unique measure using a lookup and match to a unique measure in the ex ante database that was approved for use in 2017.

In order to match unique measures in the DEER NTG values, we relied on the corrected UTCs and listed delivery types, or on the NTG_ID provided as part of the claim when it was found to be correct. If a claim was matched to a single measure, the NTGR kWh and therms were assigned to that claim. Any claim not matched to a single row in the table was left unchanged with respect to the UTCs, NTG, and NTG_ID values. We did not modify the delivery type as part of this review.

3.2.5 Effective Useful LifetimesThe DEER EUL table provides approved EUL values defined by various combinations of the building type, building location, tech group, tech type, use category, use sub-category, and sector. We updated EUL and RUL values as appropriate based on any corrected values for UTC fields that define the EUL value in the ex ante measure database. We only applied EUL and RUL values that were approved for use between January 1, 2017 and December 31, 2017.

Like the methodology for measure description in section 3.2.2 above, we determined the correct UTCs for each claim and associated the correct EUL/RUL values by matching unique measures in the ex ante measure database.

In order to match a unique measure in the ex ante measure database for corrected EUL and RUL values, we relied on the corrected UTCs and the listed delivery type, or the EUL_ID provided as part of the claim, when it was found to be correct. If a claim was matched to a unique measure, the EUL, RUL, EUL_ID, and the associated UTCs were assigned to that claim. Any claim not matched to a single row in the table was left unchanged with respect to the UTCs, EUL, and EUL_ID values. As with NTG values, we did not modify the delivery type as part of this review.

Measures with measure application type set to early retirement are required to have a value assigned for the RUL. Based on matching claims to a unique measure in the ex ante measure database, we updated the RUL values as necessary to be consistent with the corrected measure application type and corrected EUL. Any claim not matched to a single row in the table was left unchanged.

9 READI - Remote Ex Ante Database Interface version 2.5.1. Available at ftp://deeresources.com/DEER/READI_v2.5.1.zip



3.2.6 Gross Realization RatesFor custom projects the CPUC allows an ex ante gross realization rate (GRR)10 of 1 for projects that were subject to ex ante review and a rate of 0.9 for all other projects. We reviewed all gross realization rates and found only these two values. We also examined the CPUC’s ex ante review records and attempted to find a way to associate prior dispositions with claims reported in CEDARS. We could not find any method for associating the dispositions to the claims, so we were not able to confirm that the correct GRR had been assigned to all claims.

3.2.7 Installation Rates We examined the installation rate to ensure that all custom projects and each of their constituent claims had an ex ante reporting installation rate of 1.0.

3.3 Calculate Lifetime Net SavingsFor each claim we calculated net lifecycle savings for kWh, kW and therms based on the adjustments we made during our database review.

For claims associated with early retirement measures, we calculated savings as follows:

Net Lifecycle Savings =

[(1st Baseline Savings * Remaining Useful Life) + [2nd Baseline Savings * (Effective Useful Life – Remaining Useful Life)]] * (Net-to-Gros Ratio + Market Effects Adder) * Gross Realization Rate * Installation Rate

For all other measures, savings were calculated as follows:


(1st Baseline Savings * Effective Useful Life) * (Net-to-Gros Ratio + Market Effects Adder) * Gross Realization Rate * Installation Rate

The market effects adder for both formulas is 0.0511.

10 D.11-07-030 Attachment B11 D.12-11-015



4 Sample SelectionWe examined the energy efficiency savings claims that the program administrators reported and submitted to CEDARS for the 2017 program year.12 Their reported savings data documents the gross and net lifecycle efficiency savings for each measure delivered by all programs. Our sample design is based on this data.

We defined the population of custom projects by selecting those measures where the deemed flag = 0,13 which indicates that those measure savings are based on custom calculations. Claims may consist of a single measure or a bundle of measures, and each claim is associated with a project identifier. Projects may comprise one or more measures that were delivered to a single location, or a project may comprise multiple measures delivered to multiple locations that are operated by the same customer. In the latter case, the measures tend to be the same or similar technology, such as a similar improvement to the same type of lighting system.

The PA-reported 2017 data contains net and gross lifecycle energy efficiency savings values for kWh and therms. These values are derived by multiplying the annual efficiency savings by the effective useful life (EUL) of the measures.14 We summed the net lifecycle efficiency savings (kWh and therms) of all claims associated with each project.

In all there were 8,491 projects associated with claimed savings for the 2017 program year. These projects constituted the sampling frame for this review. From this sample, we selected 114 projects for file review. Our file review methods are described in section 5 below.

Table 5 shows the total number of projects and claims reported by each PA and number of projects and claims that we selected for our sample. In all, we selected 3% of both projects and claims. There were approximately two claims per project for all reported claims and for our sample. We sampled between 12 and 46 projects from each PA. A larger sample was assigned to PAs with more projects and savings associated with those projects.

Because our file review of sampled projects is not equivalent to an ex post evaluation of the 2017 claims, the CPUC staff determined that the results from the file review would not be extrapolated. Therefore, claims for projects that were not included in the sample were only corrected based on our database review (as described in section 3).

Table 6 shows the total number of reported and sampled claims for each PA. While the table breaks out separate counts for MWh, MW, and therms, we only considered energy (MWh and therms) savings when we selected the sample. For those PAs that provide electric service, the savings for our sample ranged between 28% and 35% of their total reported claims. For those PAs that provide gas service, the savings for our sample ranged between 47% and 81% of the

12 The evaluation team for Group A provided a copy of the entire 2017 reported data they extracted from CEDARS, which they used in designing the sample for 2017 deemed measures.

13 The Group A data had updated entries for the deemed flag so that it correctly identified all the measures that were deemed, based on the description of the measure.

14 For early retirement measures, lifecycle savings and different annual savings are applied to the remaining useful life of the existing baseline and to the balance of the effective useful life.



claims. Table 6 also shows combined MWh and therm savings as MMBtu savings. In total, our sample accounts for 68% of the reported lifetime net MMBtu savings.

Table 5: Number of Reported and Sampled Projects and Claims


Projects

Reported Claims 3,432 629 79 246 4,386

Sampled 46 33 12 23 114

% in Sample 1% 5% 15% 9% 3%

Claims

Reported Claims 6,119 1,970 111 291 8,491

Sampled 107 79 22 30 238

% in Sample 2% 4% 20% 10%

3%

Claims per Project

Reported Claims 1.8 3.1 1.4 1.2 1.9

Sampled 2.3 2.4 1.8 1.3 2.1

Table 6: Reported Lifetime Net Savings and Percent Sampled for Project File Review

Claimed Lifetime Net Savings


MWh

Reported Claims 1,364,619 894,335 164,824 31,689 2,455,467

Sampled 382,021 264,603 57,435 1,132 705,191

% in Sample 28% 30% 35% 4% 29%

MW


Sampled 42 35 9 1 88

% in Sample 20% 30% 39% 10% 24%

Therms

Reported Claims 63,614,914 1,305,975 7,671,599 18,648,295 91,240,782

Sampled 47,615,441 (122,120) 6,192,614 8,770,659 62,456,594

% in Sample 75% -9% 81% 47% 68%

MMBtu

Reported Claims 11,017,573 3,182,068 1,329,539 1,972,952 17,502,132

Sampled 6,064,999 890,613 815,231 880,928 8,651,772



Claimed Lifetime Net Savings


% in Sample 55% 28% 61% 45% 49%



5 Review of Sampled Project FilesThis section summarizes the methods we used to review net lifecycle efficiency savings for sampled projects.

5.1 Data Requests

5.1.1 Review Project FilesProgram administrators uploaded all files for the sampled projects to the EM&V platform on www.DEERresources.info. Then we reviewed the project files to determine whether any critical data, models, or documentation were missing.

5.1.2 Request Missing Data, Models or DocumentsWe used the EM&V platform messaging system to notify PAs regarding any missing critical data, models, or documentation for a project. PAs were given three working days to supply the requested missing files via the EM&V platform. The requests included:

Working calculators. These included spreadsheet models or hourly model input files.

Fuel switching justification. If the project resulted in an energy consumption shift from electricity to gas or vice versa, then the three-prong test15 was required to assess eligibility for switching fuel .

On-site generation analysis. Any on-site generation, such as cogeneration or behind the meter solar arrays at the site, required analysis that considered the marginal change in IOU-supplied fuels resulting from the project.

Justification of early retirement and remaining useful life. Early-replacement projects required evidence that the existing equipment was functional and could continue to be used for the claimed remaining useful life.

Evidence of program influence. Eligible projects required evidence that project initiation was influenced by the program.

Permitting. If permits are required for measure installation, the customer/installer must obtain such permits. For 2017 programs this requirement was usually satisfied by such a statement on the signed customer agreement.

In total we made 25 supplemental data requests, and we received responses for 24 of those 25 requests. Our supplemental data requests also included a couple of requests to provide replacement files for originally delivered files that we found to be corrupted or otherwise unopenable. Between April 11 and April 19, 2019 we submitted 21 requests, each as soon as we

15 Energy Efficiency Policy Manual Version 5, page 24, Cost Effectiveness Rule XV.10



identified missing items to allow time to receive information potentially critical to recalculate savings. We received responses to 19 of these requests within the three-day response period. One response was one day late, and we did not receive a response to one request. On May 10, 2019 we also submitted four requests for simple items (three for permit requirements and one to confirm that a project was not connected to the customer cogeneration system). The responses for these four requests were provided within the three-day response period.

5.2 Review Project FilesIn this section we describe the methods we used to review and update lifecycle efficiency savings for each measure in the sampled projects.

5.2.1 Eligibility-Related Savings AdjustmentsOur analysts reviewed the files for each project and determined whether each measure in a project satisfies the CPUC policies and guidance that define whether a measure is eligible16 for a savings claim. We applied the following tests to determine whether savings should be allowed or set to zero due to measure or project ineligibility.

Valid 2017 project completion. If the project was installed prior17 to 2017 and if no required M&V occurred during 2017, then the project was ineligible to claim savings during 2017.

Fuel switching. If the project shifted energy consumption from electricity to gas or vice versa, then the three-prong test was required to assess eligibility for switching fuel. This test did not apply to new construction projects.

On-site generation. Analysis must be performed that shows the project resulted in a change in IOU-supplied energy, considering on-site generation, such as cogeneration or behind the meter solar arrays.

Regressive baseline. If the new equipment was less efficient than the equipment it replaced, then the measure was ineligible.

Required permits. If there was no evidence that required permits were obtained for the installed measure, then the measure was ineligible.

Claimed Baselines. We also considered whether a measure was eligible for the claimed baseline based on its measure application type. We adjusted savings when we found violations in the rules governing baselines.

Exceeds code requirements. We reviewed the baseline efficiency used in the claim calculations to confirm that the efficiency was equal to or exceeded relevant codes. If noncompliant baselines were used, we adjusted the baseline efficiency to be consistent with code requirements and then recalculated savings.

16 Eligibility policies were taken from 2018 Statewide Customized Offering Procedures Manual for Business.17 There were claims with installation dates in 2018, but none of them were part of the file review sample.



Exceeds industry standard practice (ISP). When measure baselines were not defined by code minimum efficiency standards, we searched for other documentation that defined ISP baseline efficiency. If we identified that a baseline was not in compliance with an established ISP baseline, we made appropriate adjustments and then recalculated savings.

Early retirement dual baseline. Early retirement measures require documentation indicating that the existing equipment is operational and is expected to remain operational for its RUL, which is nominally defined as one-third of the equipment’s EUL. The baseline efficiency during the RUL period (first baseline period) is typically defined as the efficiency of the existing equipment. The baseline efficiency during the period after the RUL period and prior to the end of the equipment EUL period (second baseline period) is then defined by the baseline definition rules for new construction (NC) or replace on burnout (ROB) measure application types that are typically defined by code or ISP. If no early retirement justification was provided in the project documentation, then we redefined the baseline type as ROB and calculated savings with only a single EUL baseline period.

For each claim we also considered whether program influence was documented with the following types of information:

Documentation of the decision-making process

Description of when the project was initiated

Description of how the installed measures were identified

Description of viable alternative options to the installed measures

Documentation of the project’s energy and cost savings

Documentation of the non-energy benefits for the project

Evidence indicating when in the project development process the program first influenced the decision to install energy efficient equipment

Cost-effectiveness calculations tied to customer payback criteria

Evidence of any other influences the program may have had on the decision to install energy efficient equipment

If none of the above types of documentation were found, then we adjusted savings by setting the NTGR to zero (prior to the addition of the market effects adder described in section 5.3). In addition, we set gross savings that accrued to utilities other than the four IOUs to zero.

5.2.2 Measure Application TypeWe reviewed the measure application type, measure description, and related information to determine whether the measure application type was correctly assigned. We corrected measure application types as appropriate, and then updated energy savings based on the correct baseline for the measure application type. For example, for a normal replacement measure, single annual



savings values (kW, kWh, and therm) and one baseline condition apply during the entire measure life. For early retirement measures, two baselines must be provided:

One baseline efficiency that is applicable at the time of measure installation, and

Another baseline efficiency at the end of the measure’s remaining useful life.

5.2.3 Deemed Measures in Custom ProjectsDuring the project reviews, we identified at least 30 savings claims that consisted of or included deemed measures. The 2018 Statewide Customized Offering Procedures Manual for Business18 states “Customized incentives are only available when the measure is not offered through a Deemed (including upstream and midstream) or Express rebate program. Measures that have an equivalent Deemed incentive, but where the Customer cannot meet all Deemed measure eligibility requirements, are not eligible for Customized incentives” However, some PA program documentation indicates that deemed measures are acceptable in custom projects. For instance, the PG&E program manual19 states, “A deemed application/project need not be submitted. Deemed measures can be processed through the custom programs as long as the deemed savings values are used and deemed rebate amounts are not exceeded.” Since the deemed measures that we encountered were not included in the Group A evaluation, we included them as valid and verified that savings were consistent with the savings from the deemed measure catalog. We did not assess if the paid incentives were consistent with the deemed measure catalog because review of incentive amounts was not within the scope of this review.

5.2.4 Gross Efficiency SavingsWe verified each claim as correct or we re-estimated the gross savings for applicable baseline periods (first baseline or second baseline starting at the end of the RUL). We used either the program model or our replacement model, and then we applied the best available model inputs from the project files provided by the PAs. For projects that included measures that were offered through a deemed savings program, we verified that the claimed savings aligned with the appropriate deemed savings value from DEER. If the savings values did not align, we substituted the appropriate value. When our resulting savings values were different than the claimed savings, we identified and corrected the reason for the difference. Those reasons include the following:

Baseline specification. This indicates that we identified an error in the baseline specification.

Calculation method. This indicates that the calculation method contained an error or that the calculation may have considered too many or too few savings aspects.

18 Item 6., page 9 from “2018 Statewide Customized Offering Procedures Manual for Business” formerly known as “The 2017 Customized Offering.”

19 From section 5.7 Notes of “PG&E Customized Energy Efficiency Policy & Programs Rulebook,” Version 1.3, 2/2/2017.



Claim discrepancies. This indicates that the final accepted savings shown in the documentation was different than the savings shown in database claim. This may have been a data entry error.

Multiple reasons. This indicates that there were at least two reasons for the difference. One error may have offset another, or they may have had additive effects.

Other. This indicates that there was some other reason for the difference not listed above.

Next, we assessed the EUL and RUL values associated with each measure and modified them as necessary to align with the most applicable DEER values.20 To select the most applicable DEER values, we used the project documentation to assign the following characteristics to each claim:

Measure description that accurately reflects the measure. If the measure description was not representative of the actual installed measure or it was not detailed enough for us to select the applicable EUL value in DEER, then we modified the description.

Building type that best described the building or facility type where the measure was installed

Sector that best described the sector in which the measure was installed

Technology group that best described the installed measure

Technology type that best described the type of technology of the installed measure

Use category that best described the end-use category of measure savings

Use sub-category that best described the end-use subcategory of measure savings.

For savings claims that consisted of multiple measures combined into a single project, we assigned these characteristics based on the measure in the project with the greatest energy savings. For ER measures that resulted in a RUL value21 of less than 1.0, we re-assigned the measure application type from ER to ROB and then recalculated the savings for the single baseline associated with ROB. If we could not definitively associate a measure with a specific row in the DEER table, we did not correct the EUL value of the claim.

5.2.5 Net Efficiency SavingsNext, we assessed the NTGR values associated with each measure and modified them as necessary to be consistent with the most applicable DEER values.22 To select the most applicable DEER values, we used the project documentation to assign the following characteristics to each claim:

20 EUL/RUL Support Tables downloaded from READI v.2.5.1 available at http://www.deeresources.com/index.php/deer-versions/readi.

21 Calculated as one-third the EUL value.22 NTG ratio support tables downloaded from READI v.2.5.1 available at

http://www.deeresources.com/index.php/deer-versions/readi.





Measure description that accurately reflects the measure. If the measure description was not representative of the actual installed measure or not detailed enough for us to select the applicable NTGR values in DEER, we modified the description.

Building type that best described the building or facility type where the measure was installed

Building vintage that best described whether the measure was installed in a new or existing building

Hard to reach. Did the customer qualify as hard to reach?

Delivery type that best described the method by which the measure was delivered to the customer, such as direct install or custom incentive

Sector that best described the sector in which the measure was installed

Technology group that best described the installed measure

Technology type that best described the type of technology of the installed measure

Use category that best described the end-use category of measure savings

Use sub-category that best described the end-use subcategory of measure saving.

For savings claims that consisted of multiple measures combined into a single project, we assigned these characteristics based on the measure in the project with the greatest energy savings. If we could not definitively associate a measure with a specific row in the DEER table, we did not correct the NTGR values of the claim.

5.3 Calculate Lifetime Net SavingsBased on our file review adjustments, described in section 5, we calculated net lifecycle savings for kWh, kW and therms for each claim. For the claims associated with sampled projects, these adjustments superseded adjustments made based on our database review, as described in section 3.

For claims associated with early retirement measures, we calculated savings as follows:


[(1st Baseline Savings * Remaining Useful Life) + [2nd Baseline Savings * (Effective Useful Life – Remaining Useful Life)]] * (Net-to-Gros Ratio + Market Effects Adder) * Gross Realization Rate * Installation Rate

For all other measures, savings were first calculated as follows:


(1st Baseline Savings * Effective Useful Life) * (Net-to-Gros Ratio + Market Effects Adder) * Gross Realization Rate * Installation Rate



The market effects adder is 0.05.23

23 D.12-11-015



6 FindingsIn this section, we provide the findings from our review of reported claims for 2017. First, we present a summary of our findings for the entire custom project portfolio. Following that summary, we provide more detailed findings for each of the PAs.

6.1 Custom Projects PortfolioAs described in section , we conducted both a database review that applied to all reported claims and a more detailed file review for a sample of projects and their associated claims. Our review of files for the sample of projects will be referred to as the sample review. We tracked the changes made for each review. Any changes made to projects and their associated claims during the initial database review were superseded by any changes made during the subsequent sample review.

Table 7 summarizes our findings for each of the PAs and the entire portfolio of 2017 custom projects. The table shows MWh, MW, and therm savings for the reported claims, as well as the adjusted savings after the database review and the cumulative adjusted savings after both the database and sample reviews. In addition, the table shows the percent of changes to the original claim.

For the portfolio, our database review reduced MWh savings by 16%, MW savings by 31%, and therm savings by 24%. When we add the adjustments made during the sample review, the reported savings claims were reduced 15%, 28%, and 13%, respectively for MWh, MW and therms. For the portfolio, the combined reviews resulted in somewhat higher savings than the database review alone. However, for individual PAs this was not always the case.

Across the PAs supplying electricity, the combined review resulted in MWh savings reductions of between 6% to 29%. For the PAs supplying gas, the combined review resulted in changed savings values ranging from an increase of 3% to a reduction of-57%. We explore the reasons for these diverse results in the following sections devoted to individual PAs.

Table 7: Change in Savings Due to Database Review and Sampled Project File Reviews



MWh Reported Claims 1,364,619 894,335 164,824 31,689 2,455,467

Database Review 1,296,740 601,916 153,695 16,272 2,068,622




MW Reported Claims 214 117 24 10 366




PG&E SCE SDG&E SCG Total Database Review 147 77 23 5 252




Therms Reported Claims 63,614,914 1,305,975 7,671,599 18,648,295 91,240,782

Database Review 56,386,449 638,853 7,625,305 5,116,958 69,767,565




6.2 PG&E Custom ProjectsIn this section, we explore the specific changes made to PG&E’s reported claims.

6.2.1 Cumulative Changes from Database and Sample ReviewAs we described in sections 3.2 and 5.2, we reviewed reported savings claims and made several types of changes that modified the lifetime net savings for a claim. The extent of those modifications often depended on the order in which the changes were applied. For example, a change in the NTGR has a different effect if it is applied after a change in EUL than it does if it is applied before the EUL change. Table 8 shows the adjusted savings as the changes were applied in a specific order. The reported claims are shown at the top of the table, followed by the savings and the percent change from the original claim as the differences accumulate over the course of five changes made during our database review and four more changes made during our sample review.

Table 8: Cumulative Changes Due to Database and Sample Review for PG&E

Cumulative Lifetime Net Savings

MWH MW Therms

Reported Claims 1,364,619 214 63,614,914

Database Review

Installation Date 1,359,000 212 63,398,774

% change from claim 0% -1% 0%

Net-to-Gross Ratio 1,281,941 146 58,262,675

% change from claim -6% -31% -8%

EUL or RUL 1,296,740 147 56,386,449




MWH MW Therms


Gross Realization Rate 1,296,740 147 56,386,449


Installation Rate 1,296,740 147 56,386,449


Database and Sample Review

Eligibility 1,321,884 210 63,053,039


Gross savings 1,356,978 216 66,858,245

% change from claim -1% 1% 5%

Net-to-Gross Ratio 1,287,871 151 65,529,328

% change from claim -6% -29% 3%

EUL or RUL 1,284,843 150 62,787,667


All Changes All Changes 1,284,843 150 62,787,667


6.2.2 Eligibility-Related Adjustments for Sampled ProjectsOur sample review explored how each sampled claim complied with CPUC eligibility policies and guidance. Some eligibility-related adjustments resulted in savings being set to zero, while others caused us to recalculate savings. Table 9 shows the number of sampled reported claims in the top row, followed by the number of claims that were subject to each of the eligibility-related adjustments. For any claim that qualified for one or more of the first set of six adjustments, we set savings to zero.

Table 9: Number of Sampled Claims with Eligibility-Related Savings Adjustments for PG&E

Claims

Reported Claims 107

Savings Set to Zero

Install date before 2017 without M&V justification 5

Application signed after install date 0

No evidence that required permits were obtained 0

Did not pass fuel switching 3-prong test 0



Claims Savings not adjusted for on-site generation 0

Measure efficiency did not exceed that of existing equipment 0

Other Savings Adjustments

No evidence of program influence 0

Changed measure application type

No justification provided for early replacement 3

RUL less than one year 1

Did not exceed all applicable codes and regulations 0

Did not exceed Industry Standard Practices 0

If a claim qualified for one or more of the second set of other savings adjustments, we recalculated savings for the first or second baseline period or for both periods as appropriate. If the files documenting a claim did not justify early retirement, then we set the savings for the second baseline period (EUL years minus RUL years) to zero and recalculated the savings for the first baseline period (RUL years) using the appropriate baseline characteristics.

If there was no evidence of program influence (see section 5.2.1), we set NTGR to zero. However, as specified in CPUC policies, our estimate of lifetime net savings included a market effects adder that reflects other actions, by the participant or non-participants, unrelated to the specific claims we reviewed. This market effect is implemented by adding 0.05 to all NTGR values. So, following this adjustment, the smallest NTGR is 0.05.

6.3 SCE Custom ProjectsIn this section, we explore the specific changes made to SCE’s reported claims.

6.3.1 Cumulative Changes from Database and Sample ReviewAs we described in sections 3.2 and 5.2, we reviewed reported savings claims and made several types of changes that modified the lifetime net savings for a claim. The extent of those modifications often depended on the order in which the changes are applied. For example, a change in the NTGR has a different effect if it is applied after a change in EUL than it does if it is applied before the EUL change. Table 10 shows the adjusted savings as the changes are applied in a specific order. The reported claims are shown at the top of the table, followed by the savings and the percent change from the original claim as the differences accumulate over the course of five changes made during our database review and four more changes made during our sample review.



Table 10: Cumulative Changes Due to Database and Sample Review for SCE

Cumulative Lifetime Net SavingsMWH MW Therms

Reported Claims 894,335 117 1,305,975

Database Review Installation Date 635,038 80 611,448


Net-to-Gross Ratio 623,323 80 771,272


EUL or RUL 601,916 77 638,853


Gross Realization Rate 601,916 77 638,853


Installation Rate 601,916 77 638,853


Database and Sample Review Eligibility 657,419 84 606,518


Gross savings 676,219 85 641,640


Net-to-Gross Ratio 660,205 84 661,182


EUL or RUL 633,559 81 627,179


All Changes

All Changes 633,559 81 627,179


6.3.2 Eligibility-Related Adjustments for Sampled ProjectsOur sample review explored how each sampled claim complied with CPUC eligibility policies and guidance. Some eligibility-related adjustments resulted in savings being set to zero, while others caused us to recalculate savings. Table 11 shows the number of sampled reported claims in the top row, followed by the number of claims that were subject to each of the eligibility-related adjustments. For any claim that qualified for one or more of the first set of six adjustments we set savings to zero.



Table 11: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SCE

Claims

Reported Claims 79

Savings Set to Zero





Savings not adjusted for on-site generation 0









If a claim qualified for one or more of the second set of other savings adjustments, we recalculated savings for the first or second baseline period or for both periods as appropriate. If the files documenting a claim did not justify early retirement, then we set savings for the second baseline period (EUL years minus RUL years) to zero and recalculated the savings for the first baseline period (RUL years) using the appropriate baseline characteristics.

If there was no evidence of program influence (see section 5.2.1), we set the NTGR to zero. However, as specified in CPUC policies, our estimate of lifetime net savings includes the market effects adder that reflects other actions, by the participant or non-participants, unrelated to the specific claims we reviewed. This market effect is implemented by adding 0.05 to all NTGR values. So, following our review, the smallest NTGR is 0.05.

6.4 SDG&E Custom ProjectsIn this section, we explore the specific changes made to SDG&E’s reported claims.

6.4.1 Cumulative Changes from Database and Sample ReviewAs we described in sections 3.2 and 5.2, we reviewed reported savings claims and made several changes that modified the lifetime net savings for a claim. The extent of those modifications often depended on the order in which the changes are applied. For example, a change in the



NTGR has a different effect if it is applied after a change in EUL than it does if it is applied before the EUL change. Table 12 shows the savings as the changes are applied in a specific order. The reported claims are shown at the top of the table, followed by the savings and the percent change from the original claim as the differences accumulate over the course of five changes made during our database review and four more changes made during our sample review.

Table 12: Cumulative Changes Due to Database and Sample Review for SDG&E


MWH MW Therms

Reported Claims 164,824 24 7,671,599

Database Review

Installation Date 154,120 23 7,671,599


Net-to-Gross Ratio 154,637 23 8,840,817


EUL or RUL 153,695 23 7,625,305


Gross Realization Rate 153,695 23 7,625,305


Installation Rate 153,695 23 7,625,305



Eligibility 154,843 23 7,671,599


Gross savings 146,651 29 8,132,986




EUL or RUL 143,116 27 7,932,992


All Changes

All Changes 143,116 27 7,932,992




6.4.2 Eligibility-Related Adjustments for Sampled ProjectsOur sample review explored how each sampled claim complied with CPUC eligibility policies and guidance. Some eligibility-related adjustments resulted in savings being set to zero, while others caused us to recalculate savings. Table 13 shows the number of sample reported claims in the top row, followed by the number of claims that were subject to each of the eligibility-related adjustments. For any claim that qualified for one or more of the first set of six adjustments, we set savings to zero.

Table 13: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SDG&E

Claims

Reported Claims 22

Savings Set to Zero














If a claim qualified for one or more of the second set of adjustments, we recalculated savings for the first or second baseline period or for both periods as appropriate. If the files documenting a claim did not justify early retirement, then we set the savings for the second baseline period (EUL years minus RUL years) to zero and recalculated the savings for the first baseline period (RUL years) using the appropriate baseline characteristics.

If there was no evidence of program influence (see section 5.2.1), we set the NTGR to zero. However, as specified in CPUC policies, our estimate of lifetime net savings includes a market effects adder that reflects other actions, by the participant or non-participants, unrelated to the specific claims we reviewed. This market effect is implemented by adding 0.05 to all NTGR values. So, following our review, the smallest NTGR, is 0.05.



6.5 SCG Custom ProjectsIn this section, we explore the specific changes made to SCG’s reported claims.

6.5.1 Cumulative Changes from Database and Sample ReviewAs we described in sections 3.2 and 5.2, we reviewed reported savings claims and made several types of changes that modified the lifetime net savings for a claim. The extent of those changes often depended on the order in which the changes are applied. For example, a change in the NTGR has a different effect if it is applied after a change in EUL than it does if it is applied before the EUL change. Table 14 shows the savings as the changes are applied in a specific order. The reported claims are shown at the top of the table, followed by the savings and the percent change from the original claim as the differences accumulate over the course of five changes made during our database review and four more changes made during our sample review.

Table 14: Cumulative Changes Due to Database and Sample Review for SCG


MWH MW Therms

Reported Claims 31,689 10 18,648,295

Database Review

Installation Date 16,272 5 5,116,958




EUL or RUL 16,272 5 5,116,958


Gross Realization Rate 16,272 5 5,116,958


Installation Rate 16,272 5 5,116,958



Eligibility 16,272 5 7,605,813


Gross savings 15,140 4 7,399,727







MWH MW Therms

EUL or RUL 15,140 4 8,026,293


All Changes

All Changes 15,140 4 8,026,293


6.5.2 Eligibility-Related Adjustments for Sampled ProjectsOur sample review explored how each sampled claim complied with CPUC eligibility policies and guidance. Some eligibility-related adjustments resulted in savings being set to zero, while others caused us to recalculate savings. Table 15 shows the number of sampled reported claims in the top row, followed by the number of claims that were subject to each of the eligibility-related adjustments. For any claim that qualified for one or more of the first set of six adjustments we set savings to zero.

Table 15: Number of Sampled Claims with Eligibility-Related Savings Adjustments for SCG

Claims

Reported Claims 30

Savings Set to Zero














If a claim qualified for one or more of the second set of adjustments, we recalculated savings for the first or second baseline period or for both periods as appropriate. If the files documenting a



claim did not justify early retirement, then we set the savings for the second baseline period (EUL years minus RUL years) to zero and recalculated the savings for the first baseline period (RUL years) using the appropriate baseline characteristics.

If there was no evidence of program influence (see section 5.2.1), we set the NTGR to zero. However, as specified in CPUC policies, our estimate of lifetime net savings includes a market effects adder that reflects other actions, by the participant or non-participants, unrelated to the specific claims we reviewed. This effect is implemented by adding 0.05 to all NTGR values. So, following our review, the smallest NTGR, is 0.05.



7 Conclusions and RecommendationsThis section provides our conclusions and recommendations for improvements in claim reporting, review of reported claims, ex ante review, and ex post evaluation for custom projects.

7.1 Applicable to All Reported Claims7. Prohibit projects installed in other years. The largest change in savings was due to claims

with installation dates before 2017, for which the delays were not justified by the PA’s measurement and verification of savings. CEDARS should be configured to only allow the submission of claims installed in prior years if the PA certifies that the delay was due to M&V.

No claims should be allowed for future years.

8. Improve measure descriptions. We found many measure descriptions that were incomplete and ambiguous. For example, “system-HVAC” does not provide any information on what was changed to improve energy efficiency. Measure descriptions should be concise, describe the system, equipment or practice that was changed, explain why the change would be more efficient, and indicate whether the measure was early retirement. For example, “Process – early retirement of gas, standard efficiency, hot water boiler and replacement with a gas, condensing hot water boiler.”

Measure descriptions can be more concise if a consistent set of categorical parameters are required for claims, and those categories are comprehensive and nonoverlapping. One example would be to require descriptions to specify a sector using mutually exclusive parameters such as residential, commercial, industrial or agricultural.

There is no need to embed information in measure descriptions if it is reliably captured by a standardized set of categorical parameters.

The CPUC should work with stakeholders to develop and disseminate a best-practices guide for writing measure descriptions.

9. Connect EAR dispositions to CEDARS claims. We were unable to determine which of the reported 2017 claims were subjects of ex ante reviews. As a result, we could not determine whether the CPUC’s policy regarding ex ante gross realization rate had been followed. EAR dispositions should be associated with unique project identifiers, and PAs should be required to maintain the same project identifier throughout the custom project’s lifecycle from initial application to final submission to CEDARS.

10. Require reproducible savings claims. Even after we received supplemental information for some of the sampled projects, we were not able to reproduce the reported savings. The CPUC should require that all custom projects must be documented with a set of files that an appropriately trained reviewer can use to reproduce the claimed savings. This set of files must contain all input values used in calculating the savings, along with either the model



used in that calculation or a citation for the exact version of a publicly available model that can be used to reproduce the savings. Publicly available models may be subject to license or usage fees, but they must be readily available for use by a reviewer.

11. Improve DEER NTG and EUL tables. In many instances, we found it difficult or impossible to definitively associate a claim with a unique row in the DEER NTG or EUL tables. Some of the descriptions provided for each row of these tables are ambiguous and overlapping. This was even true when we combined the information available from the categorical parameters associated with the rows of these tables. Some of the confusion arises from the ambiguous values used for the categorical variables, such a non-residential vs. commercial.

To improve these tables, a set of categorical parameters must be developed that when taken together form a unique identifier for each row. If such a unique identifier can be developed and then required for all PAs who submit claims to CEDARS, then the ID columns of the tables can be removed. Although the values in the ID columns are unique, they do not provide enough information to definitively assign a NTG or EUL value to a claim. Two sets of categorical parameters need to be developed; one to identify NTG and the other to identify EUL. These two identifying parameters would make it easier to develop a measure description that fits with the suggestion for a best practices guide mentioned in recommendation 2 above.

In a separate but related suggestion, one of the NTG categorical parameters should indicate whether a claim is associated with a hard-to-reach customer. In general, as the CPUC creates policies that distinguish the ex ante treatment of any class of custom projects, these categorical parameter values, or the parameters themselves, must be updated so that the claims data completely and accurately characterizes each custom measure.

12. Confirm claims are nonoverlapping and accrue to an IOU. We found some cases where an electric-only utility claimed therm savings and other cases where a gas-only utility claimed electric savings. These could be interactive effects, where a gas or electric measure causes changes in the other fuel, or they could be projects developed through the collaboration of two utilities. The key requirement should be that the savings claims do not overlap. That is, the savings claims must be additive and not duplicative.

Further, we found one case where savings occur outside the service territories of the CPUC. In other words, the claimed project did not affect energy supplied by an IOU. To avoid cases like this, documentation should be required for each project to confirm the claimed savings do not overlap with another PA’s claim and that all savings accrue to one of the four California IOU energy suppliers.

7.2 Applicable to Specific PAsIn this section, we recommend specific improvements each PA should implement in preparing their claims.



7.2.1 PG&EIn preparation for submitting claims, PG&E should:

Enter the NTG_ID and EUL_ID for all claims. This will document the source of the NTG, EUL, and RUL values.

Use approved NTG values. Most notably, LED lighting comprised a large percentage of submitted claims, however reported NTG values were higher than what was approved in the e x ante database for 2017.

Confirm that the claimed values are consistent with the final project documents.

Ensure that calculation methods and selected baselines are correct.

7.2.2 SCEIn preparation for submitting claims, SCE should:

Better track installation date to remove non-program year claims or provide justification when claims required extensive M&V that spanned multiple program years.


Use approved NTG values. Most notably, LED lighting comprised a large percentage of submitted claims, however reported NTG values were higher than what was approved in the ex ante database for 2017.


7.2.3 SDG&EIn preparation for submitting claims, SDG&E should:


Provide more descriptive elements in the measure description for claims that can be interpreted multiple ways, such as Whole Building, or Systems-HVAC.

Confirm that the claimed values are consistent with the final project documents.




7.2.4 SCGIn preparation for submitting claims, SCG should:

Better track installation date to remove non-program year claims or provide justification when claims required extensive M&V that spanned multiple program years.


Provide more descriptive elements in the measure description for claims that can be interpreted multiple ways, such as Whole Building, or Systems-HVAC.

Do not claim savings for non-IOU energy such as electric savings for LADWP Savings by Design projects.

Maintain complete project documentation for gas claims resulting from SCE projects.




8 Data ProductsWe have assembled all the data collected in this review. Excel workbooks containing the claims for each PA are available for PA-only review in the directory titled “2017 Custom Claims Review Results” in the Evaluation, Measurement, and Verification (EM&V) platform on the website www.deeresources.info



Appendix A: Stakeholder Comments on Work PlanPA Section Topic Page Comment SBW Response

PG&E 1 Gross Savings

2 Can the evaluation team please define "within the scope" and explain how exogenous variables will be accounted for?

Scope refers to the scope of the population represented by the last evaluation. For example, if it did not include high bay LED lighting fixtures, we would pass thru the savings claim

PG&E n/a Individual Project Review Results

n/a As the evaluation team concludes evaluation of individual projects, can they make those results immediately available to the IOUs so that PAs have sufficient time to draft comments or search for any additional information that could help inform the results?

Schedule does not allow for this.

PG&E 5.2.3 Gross Savings Models

11 The evaluation team states that a "replacement model" may be used. In this case, could the model be included in the report so PAs can observe what parameters are used and replicate results?

These could be provided on CMPA with CPUC approval.

PG&E 1 Use of Findings

1 Given that this research will serve as a proxy for 2017 ex post results, are the findings limited to affecting only the 2019 ESPI awards? What steps are being taken to ensure that proxy evaluation results are not used in a way that would be reserved for full evaluation activity?

CPUC determines how the results and report are applied.

PG&E 5.2.1 Project Eligibility

10 If the evaluation team believes that a project may be ineligible, can they please contact the PAs first so additional information, if available, can be provided?

We will ask for supplemental information if we cannot find evidence of eligibility.

PG&E 5.2.4 NTGR 12 Due to idiosyncrasies in how PG&E tracks projects internally it is possible that one of the three conditions that would cause NTGR to be set to zero may appear to have occurred. In any case where the evaluation team believes NTGR should be set to zero can they please contact the PAs immediately so additional information, if available, can be provided? The evaluation plan stipulates this will happen for missing documentation referenced in criterion three, but we make this comment with regard to all three cases.

Insufficient time to make this change.



PA Section Topic Page Comment SBW Response

PG&E 5.2.4 NTGR 12 The Net To Gross Working Group set forth standards upon which net-to-gross ratios should be calculated in California. Much of the NTGR is determined by participant interviews. Since such interviews are not in the scope of this proxy evaluation, PG&E is concerned with the latitude afforded the evaluation team when they state "we will characterize how well the PAs document program influence." Can the evaluation team please share their scoring methodology with the PAs and remain open to any refinements that become apparent? PG&E believes that a simplified NTGR calculation methodology could actually result in an equally accurate and more cost-effective evaluation and looks forward to being a stakeholder in this alternate NTGR approach.

Program influence is only to be considered as an eligibility criterion, it does not affect the NTGR.

PG&E 2 Eligibility 7 Custom projects often require a year or more to be implemented. Some projects claimed in 2017 were reviewed and were issued incentive agreements in 2016 or prior. PG&E believes that the policy and guidance that was in effect at the time of the incentive agreement should be the policy that is applied to a project, not the policy in effect at the time of the claim. What will be the evaluation team’s approach to these projects?

A data request was issued allowing PAs to cite the projects whose claim was delayed by M&V. These will be allowed.

PG&E 3.1 Claim Year 6 Many custom projects installed in 2016 will be claimed in 2017 because several months are required for the project implementer to conduct M&V, revise energy savings, and prepare reports, and for PG&E to review these reports and revised calculations before the savings for these projects can be finalized. If there are projects other than those addressed in the CPUC's follow-up data request on March 29, 2019, can the evaluation team please reach out to the PAs to request further information that would denote an M&V period took place?

Schedule does not allow for this.




PG&E 3.6 GRR 7 PG&E would like to note that if the CPUC selects a project for ex ante review and then subsequently waives the review, the GRR was set to .9 per PG&E’s understanding of CPUC guidance. Is SBW’s understanding the same?

It was not possible to match EAR dispositions with 2017 claims, so no changes will be made to claimed GRR.

SCE - General - Since baseline accuracy impacts gross savings, and since it may be very difficult to determine them without site visits, how this potential gap is addressed may impact the accuracy of the 2017 impact evaluation if there are substantial performance variations from historical trends. While there is no clear remedy to this problem, SCE would appreciate the team’s thoughts on how to mitigate this challenge. Perhaps the census tier or novel projects could have more time for data discovery.

Our project file reviews will only make changes that can be supported by CPUC policies and by documentation provided by the PAs. We agree that the absence of site visits limits this work. We do not see any way to mitigate the absence of sites visits.

SCE - General - The research team can address some of the challenges posed by this issue by focusing on items the 2015 IALC impact team found key to driving gross and net savings

2015 IALC impact results will only be applied when information critical to eligibility, gross/net saving estimation is not made available for a sample project. We are not investigating the processes by which the PAs developed and documented these projects.




SCE - Net - SCE is concerned about setting NTG to zero if there is a “lack of documentation of program influence or equipment order/installation date being earlier than the application date.” Program influence is often challenging to establish or for parties to agree to. There are good projects with poor documentation and vice versa, which is why NTG surveys and site visits with document discovery are typically important to help fairly determine program influence. Since surveys and site visits are not being deployed, SCE suggests the research team allow for flexibility on what documentation is provided in the initial project files since 2017. To reduce NTG to zero is an unreasonable discount given that no surveys or site visits will be initiated. SCE would greatly appreciate (as has been done historically) the ability to review these “zero saver” projects.

We will be looking for any documentation of program influence. If none is found, we will ask the PA to provide such documentation.

SCG-SDG&E 3.1 Install date 6 The methodology in section 3.1 should be modified to state “savings will be excluded for the purposes of ESPI calculation for any projects that were not installed in program year 2017.”

PAs have been asked to list those projects whose installation date is prior to 2017 due to M&V. These projects will be treated the same as projects with installation in 2017. The purpose of this review is to support ESPI, so that proposed language is not necessary.

SCG-SDG&E 5.2.4 NTGR 12 If documentation of program influence exists within the project file, no further adjustments to NTGR value should occur based on review of program file information alone.

The three criteria cited determine whether the project was eligible for a savings claim. Even for eligible claims, the claim may have assigned the wrong NTGR based on the measure description and other factors such as delivery type. So, the review may result in selecting a more appropriate NTGR from the DEER values adopted for use in 2017.
