appendix h of garage lighting retrofit 85 energy … · 2016-07-22 · analysis team in the...

Energy Efficiency on an Urban Scale Global Institute of Sustainability66

Back to Main Table of Contents

APPENDIX D: CASE STUDY: LEVEL 2 ANALYSIS OF GARAGE LIGHTING RETROFIT 85

D1. BACKGROUND 85

D2. OBJECTIVE 85

D3. METHODOLOGY 85

D4. RESULTS OF SAVINGS MEASUREMENTS 86

D5. COMPARISON WITH UTILITY BILL ANALYSIS 87

D6. CONCLUSION 87

APPENDIX HENERGY SAVINGS EVALUATION OF COMMERCIAL UPGRADE MEASURES THROUGH INDIVIDUAL PROJECT ANALYSIS AND UTILITY BILL MODELING

TABLE OF CONTENTS

ABSTRACT 67

1. INTRODUCTION 67

2. OBJECTIVES 67

3. OVERALL APPROACH 68

4. LEVEL 1 ANALYSIS METHODOLOGY 69

4.1 DATA SCREENING AND BINNING 69

4.2 AUTOMATION OF SAVINGS CALCULATION 70

4.3 ANALYSIS RESULTS 71

4.4 POSSIBLE CAUSES FOR DISCREPANCY 74

5. LEVEL 2 ANALYSIS 75

6. LEVEL 3 ANALYSIS 77

6.1. BACKGROUND 77

6.2. UPGRADES SUGGESTED AND PREDICTED ENERGY SAVINGS 78

6.3. ANALYSIS AND SAVINGS VERIFICATION 79

6.4. CONCLUSION 79

7. PAYBACK ANALYSIS 80

8. CUMULATIVE SAVINGS OVER TIME 80

9. SUMMARY AND SUGGESTIONS FOR FUTURE ENERGY CONSERVATION PROGRAMS 81

REFERENCES 81

APPENDIX A: BASELINE MODEL DEVELOPMENT AND UNCERTAINTY 82

APPENDIX B. MODEL IMPROVEMENT AFTER ADJUSTING FOR UTILITY BILL READ DATES 83

B1. PROBLEM STATEMENT 83

B2. COMPARISON OF MODEL GOODNESS-OF-FIT 83

B3. ENERGY SAVINGS COMPARISON 83

B4. CONCLUSION 83

APPENDIX C: ENERGY CREEP ANALYSIS 84

C1. BACKGROUND 84

C2. ISSUE INVESTIGATED 84

C3. CALCULATION METHODOLOGY 84

C4. RESULTS AND CONCLUSION 84

This report summarizes the efforts and findings of the commercial team during the three year course of the Energize Phoenix program. We wish to acknowledge useful contributions from all the members of the team, especially, Alex Castelazo and Prof. Patrick Phelan. An undergraduate student, Hara Kumar, was also involved in the last stages of this work, especially related to the creep analysis. The contribution of former graduate students, Shreya Agnihotri and Sadiq Jubran is also acknowledged.



Appendix H Table of Contents

ABSTRACT

Energize Phoenix (EP) was a three year energy efficiency program conducted by a joint collaboration between the City of Phoenix, Arizona State University and a large electricity provider. The intent was to improve energy efficiency in residential, multi-family, commercial/industrial buildings located in a portion of the city of Phoenix around the light rail corridor. There are several facets to the EP program with engineering-based verification of the energy savings due to commercial buildings upgrades being one of them, and the focus of this report.

Various issues had to be addressed such as incomplete data, spurious data behavior, multiple upgrade projects in the same facility, weather normalization using well-accepted change point models applied to utility bill data, and baseline model uncertainty. Considering the nature and characteristics of utility bill data, the evaluation methodology initially adopted was labor intensive and involved a manual data screening procedure of projects on an individual basis, and then one-by-one baseline modeling and savings assessment. An automated process was developed in order to reduce the labor required to analyze and update energy savings in hundreds of buildings on a periodic basis. This report presents results of analyzing close to 560 completed upgrade projects with the primary objectives of measuring savings from Energize Phoenix commercial projects and comparing evaluated savings with those predicted by the contractors prior to the upgrades. Reasons for these differences are discussed, and follow-up investigations into this discrepancy are also documented. The uncertainty in the savings determined of all the projects has also been computed and reported. This report ends with conclusions and suggestions for further investigation needed to improve the accuracy and reliability of determining energy savings in allied large scale energy efficiency programs. Four appendices describe specific allied investigations undertaken in support of some of the issues identified during the course of this study. A separate technical report assembles, in further detail, the complete analysis work done by the non-residential analysis team in the framework of the EP project.

1. INTRODUCTION

Energize Phoenix (EP) was a three year energy efficiency program led by a joint collaboration of three major institutions –the City of Phoenix, Arizona State University (ASU) and Arizona Public Service (APS) – the state’s largest electricity provider. The main goal of the program was to improve the energy efficiency in the buildings located around the Phoenix light rail corridor and to create jobs. The participating

buildings included residential, multi-family, commercial/industrial, etc. The program is one of 41 from across the United States which are supported by the U.S Department of Energy’s Better Buildings Neighborhood Program and the American Recovery and Reinvestment Act of 2009 in order to test new models for scaling energy efficiency and to create jobs. Historically, energy efficiency programs have faced a trio of interconnected forces — technical, economic, and socio-behavioral — which continue to hinder mass-market scaling. Inter-disciplinary research within the framework of EP is aimed at understanding and helping resolve these barriers. Research projects cover numerous facets (such as behavioral and attitudinal differences between participating and non-participating homeowners and business owners, contractor marketing methods, the effects of energy feedback devices coupled with other education or budgeting information, spatio-temporal trends in participation rates, econometric modeling of savings, and economic impact analysis), all of which were meant to study the above influences and at assuring that energy efficiency programs realize their full potential.

EP was a contractor-driven program in which participants receive incentives to encourage energy efficiency projects. Prospective buildings were not pre-selected by ASU, APS or the City of Phoenix; instead, the task of initiating contact and convincing customers to join the EP program rested with the contractor. Customers utilizing Energize Phoenix were able to match the incentive provided by APS through its ongoing Solutions for Business energy efficiency program with an additional incentive from EP. EP restricted incentives such that the sum of the incentives could not exceed 100% of the incremental project cost to the customer. This report is narrowly focused in its scope on research work performed to better understand the energy savings achieved by Energize Phoenix in commercial projects and the accuracy of the contractor predictions of those savings.

2. OBJECTIVES

The primary objective of the EP commercial team was to analyze the data and to quantify the energy savings achieved in the commercial buildings which underwent upgrades incentivized by the EP program. These savings were then compared to the savings predicted by the energy contractors during the project sales process. Note that the contractors used either custom audits or prescriptive guidelines that rely on equipment counts (such as lights) to predict savings. In addition, contractors utilizing the Small Business program, a streamlined, direct-install program meant for small businesses and schools, employed standard software and




other proprietary tools supplied by a third party contractor. The energy savings comparison between predicted and evaluated by the EP team helps to assess the overall effectiveness of the upgrades and the accuracy of the savings projections of the program as a whole. Other allied issues were also investigated; for example, whether certain contractors tended to consistently over-predict savings as compared to others, and the reasons for doing so.

There were hundreds of commercial buildings which underwent energy upgrades through the 3-yr duration of the program. These projects varied vastly in characteristics like business type, size, etc. Four different approaches are suggested in standard protocol documents such as IPMVP (2010) and in ASHRAE Guideline 14 (2002). However, the monthly utility bill analysis approach was deemed to be the only realistic method to determine savings in a program this big with limited personnel involved in the measurement and verification (M&V) process. Further, since EP was an ongoing program and contractors often specialized in certain types of retrofits, buildings underwent upgrades on a continuing basis, and so savings calculations had to be redone at frequent intervals. Utility bill data for all projects (old as well as recent) was provided by the utility on a quarterly basis, and it was logical to recalculate and update the savings for all buildings every three months. This prompted an additional objective, namely to simplify and automate the savings analysis methodology as far as possible so that future energy conservation programs similar to EP could reduce M&V analysis costs.

3. OVERALL APPROACH

Because of time constraints, the baseline electricity consumption prior to the implementation of energy upgrades could not be determined by in-situ measurement. Hence, the whole building analysis approach, which is one of the four general M&V approaches widely followed by the professional M&V community (see for example, ASHRAE Guideline 14, 2002 or IPMVP, 2010) was adopted. The approach involves relying on a whole year of utility bill data prior to the upgrade to establish a baseline model of energy use against monthly mean outdoor temperature. Such monthly utility bill data was made available from the APS customer billing database. The model was then applied to measured outdoor temperature during the post-upgrade period, and the sum of the monthly differences between these model predictions and the actual measured utility bills during the post-upgrade period were taken to be the “evaluated” upgrade energy savings. Note that energy savings cannot be directly measured but are often inferred from measurements performed before and after the

upgrade. Hence, the term “evaluated savings” is used in this report when such an M&V path is followed.. The commercial team’s entire analysis process is described in more detail in Appendix A.

Figure 1 depicts these three levels of analysis in a succinct manner. This report presents the results of our Level 1 and Level 2 analyses along with Level 3 results for one of the two buildings studied, while the technical report (Reddy et al., 2013) includes the Level 3 results of the other building.

Because of the error introduced in such a general approach (called Level 1 analysis) which does not involve inspecting the buildings individually, it was decided to conduct a limited number of in-depth analyses in buildings where large discrepancies were found between evaluated and contractor-predicted savings. This approach (referred to as Level 2) was meant to provide some degree of credibility in our speculation as to the observed differences, and allow us to correct the data as appropriate. Due to the large number of projects, our approach was to sample a sub-set of the completed upgrade projects and verify the savings predicted by the contractor through follow-up field visits, installing in-situ equipment and monitoring for a relatively short period of time. The degree of over- or under- prediction of the savings could then be determined more accurately, and the causes for any such discrepancies identified. This would provide useful feedback to APS and to the contractor, and suggest ways by which future upgrade savings estimations can be improved.

FIGURE 1. THE THREE LEVELS OF ANALYSIS APPROACHES UNDERTAKEN IN THE EP PROGRAM BY THE COMMERCIAL TEAM




Finally, Level 3 involved an in-depth energy analysis of a few selected projects so as to evaluate energy savings and to provide recommendations for additional potential energy conservation measures. Level 3 analysis involved developing a calibrated detailed simulation model of the energy use in the building based on owner-provided architectural drawings, energy audit reports, usage data and project applications. This is consistent with another standard M&V approach described in such documents as ASHRAE Guideline 14 (2002) and IPMVP (2010). Sub-monitoring the energy use and indoor environment was also done to calibrate the model. The primary objective was to determine quantitatively the effect of individual energy efficiency upgrades on overall energy consumption and to identify other possible energy conservation measures (ECMs).

It is important to note that APS currently utilizes a third party contractor to conduct the measurement and verification of energy savings for each of the APS energy efficiency programs, rather than relying on contractor predictions. The results of that M&V work are integrated into the annual Energy Efficiency report that is submitted each year to the Arizona Corporation Commission (ACC) to document the claimed savings from APS’s energy efficiency programs. The savings estimates made by the third party M&V contractor are based on M&V industry standard practices involving extensive measurement and verification activities employing statistical sampling techniques. Activities include field metering of the specific equipment installed, on-site inspection of the equipment, customer surveys to estimate run times, engineering models to simulate electricity use, and comparison to estimated usage by standard efficiency equipment. The EP commercial team’s analysis was conducted independently of that of the third party contractor at APS’ request.

4. LEVEL 1 ANALYSIS METHODOLOGY

4.1 Data screening and binning

The first step involved ascertaining consistency of energy use over the years. This was conveniently done by simply generating time series plots (see Figure 2) of historic utility bills, and looking at them visually. Some of the projects showed considerable variation in usage pattern which made it necessary to manually screen all individual projects for data quality. This also led to the decision to use only one year of data immediately prior to the upgrade as the baseline period since, as is well known, energy use patterns in commercial buildings tend to change over time.

FIGURE 2. TIME SERIES PLOT OF MONTHLY UTILITY DATA FOR OVER THREE YEARS BEFORE THE UPGRADE AND ONE YEAR AFTER UPGRADE FOR A SPECIFIC EP BUILDING

The data visualization step allowed identification of anomalous behavior and grouping of buildings into bins, as illustrated in Figure 3. Bin A consisted of buildings where there were missing or inadequate pre-upgrade data (i.e., less than twelve utility bills). Buildings with abnormal data patterns were placed in Bin B. Three of the common generic cases encountered are illustrated in Figure 3. Some buildings exhibited an increase in energy use after the upgrade, some had abnormal spikes, and others had markedly different seasonal variation patterns. Bin C consisted of buildings which did not have at least six months of post-upgrade data, in which case the calculation of energy savings was deferred until more utility bill data was forthcoming. Finally, those buildings which did not fall in any of the above three bins were placed in Bin D, for which the savings were determined. Additional manual screening criteria for data quality had to be empirically framed as shown in Table 1. For example, if predicted savings were less than 1% of the energy use, our analysis procedure was deemed to be unsuitable. An example of anomalous behavior which warranted placing a project in Bin B was a case in which the contractor-predicted savings exceeded the total energy use of the building. This screening process made the whole analysis labor intensive, but it needed to be done only once per project.

FIGURE 3. ILLUSTRATIVE EXAMPLES OF ABNORMAL DATA BEHAVIOR PERTINENT TO BIN A AND BIN B IDENTIFIED DURING THE VISUAL SCREENING PROCESS




Since there were several buildings which fell into Bin B, phone calls to the facility managers or owners of a subset of these buildings were also undertaken in order to identify possible reasons and to reconcile the odd behavior. If the behavior could be explained convincingly or if appropriate rectifying action could be taken, these buildings were moved to Bin D, otherwise they were moved to Bin A. A possible factor causing some of the anomalous behavior could be attributed to the fact that the team was unable to perform account matching with the master meter of the facility. Such data was not made available due to privacy reasons.

TABLE 1. DATA SCREENING AND BINNING CRITERIA EMPLOYED FOR SCREENING

4.2 AUTOMATION OF SAVINGS CALCULATION

Savings were evaluated by comparing the energy consumption between corresponding months of pre and post upgrade periods. A large number of projects showed weather dependency where the total energy consumption was influenced by cooling and heating loads of the building. Thus, the influence of the weather had to be taken into account for these projects in order to properly evaluate the upgrade savings. The billing cycle does not usually match with calendar months, and since read dates were available, the utility bills of each project were adjusted to match calendar months as this would simplify the analysis considerably. A detailed evaluation of analysis results with and without such an adjustment is described in Appendix B.

FIGURE 4. A HYPOTHETICAL BUILDING SHOWING MULTIPLE-UPGRADE PROJECTS. ENERGY SAVINGS WERE SIMPLY DETERMINED USING PRE-UPGRADE AND POST-UPGRADE PERIODS AS SHOWN

There were several buildings which qualified for EP incentives involving multiple energy upgrades. These were treated as single projects using the simple approach illustrated in Figure 4. The data period in-between the first and the last upgrades was simply excluded from the analysis since, in most cases, these multiple upgrades were done within a few months of each other. All the upgrades were treated as one single upgrade with the post upgrade period assumed to start after the last upgrade was completed. The sum total of all the contractors’ savings predictions for the building was taken to be the overall predicted savings.

As the number of projects increased and since savings had to be recalculated at quarterly intervals as more data was forthcoming, it was critical to automate the process as much as possible. The automation scheme which evolved is shown in Figure 5. Note that there are still two steps which require manual screening.

FIGURE 5. FLOWCHART OF THE AUTOMATED ROUTINE DEVELOPED TO DETERMINE ENERGY SAVINGS FROM NUMEROUS UPGRADED BUILDINGS IN THE FRAMEWORK OF EP PROGRAM

To facilitate the screening in the automation process, a visual template (shown in Figure 6) was developed. This involved generating scatter plots of monthly energy use versus outdoor temperature as well as annual time series plots superimposed on each other for energy use both prior to and after the upgrades.




FIGURE 6. PLOTS FROM THE VISUALIZATION TEMPLATE MEANT AS AN AID TO PERFORM MANUAL SCREENING OF DATA

which the results were analyzed, and are reported here. Over time, as more data is forthcoming, the 171 projects under Bin C will also be moved to Bin D and will become part of the pool of buildings where energy savings will be determined. Nonetheless, 150 projects which fell into Bins A and B had to be kept aside due to data quality and inadequate data issues, and savings from these projects are not part of the results reported here.

Figure 7 is a pie chart showing the relative distribution (as percentages) of specific ECMs implemented (lighting, controls, pumps/motors, HVAC, windows, food refrigeration and solar water heating). Lighting upgrades were by far the most frequent, accounting for 70% of the total projects, primarily because of the relative low effort required in both estimating and installing the upgrades, as well as the relatively high incentives and the minimal disruption to participants’ operations.

FIGURE 7. PIE CHART DEPICTING THE PERCENTAGES OF SPECIFIC ENERGY CONSERVATION MEASURES (ECMS) PERFORMED BY ECM-TYPE FOR THE 557 PROJECTS. LIGHTING ACCOUNTED FOR 70% OF THE TOTAL NUMBER OF ECMS EMPLOYED IN PROJECTS

The methodology for developing the baseline model is described in Appendix A. It is consistent with the modeling procedures advocated in the engineering literature involving identifying the best change point regression model among several different model formulations with outdoor temperature as the independent variable. A FORTRAN program was developed specifically for the purpose of the EP commercial building analysis effort which incorporated the widely used Inverse Modeling Toolkit (IMT) computer code (Kissock, Haberl and Claridge 2002) as a subroutine. The program reads the utility bill data for a specific building along with outdoor temperature, and assigns it to the pertinent bin. If the building falls into Bin D, the program then identifies the best change point model among several possible functional forms, calculates savings for that building, and does this for all the buildings in the database. The total savings are then determined along with the contractor-predicted savings. Finally, the automated routine generates pertinent summary statistics and graphics of the entire program savings.

4.3 ANALYSIS RESULTS

As of end of April 2013, 557 non-residential upgrade projects were completed. Pertinent statistics are assembled in Table 2. Of all these projects, only 236 projects fell into Bin D for

TABLE 2. SUMMARY TABLE OF RELEVANT OVERALL STATISTICS (AS OF END OF APRIL 2013).




The savings fraction can be calculated in two ways, which would yield different results because of large differences in the sizes of the upgrade projects. The two definitions are given below, and analysis results following both methods are presented.

(a) Aggregated savings fraction determined as the ratio of the aggregated energy savings summed across all projects divided by the aggregated baseline energy use:

Aggregated Savings Fraction =

Equation 1where N is the total projects for which the savings are determined. This is tantamount to considering all the upgrades as having happened in one large hypothetical building. This analysis may introduce some error as some of the projects with large energy consumption can bias the analysis results.

(b) Average of individual fractional savings of individual projects calculated as:

Average of Individual Fractional Savings =

Equation 2

In addition, the percentile values of individual fractional savings at 10%, 25%, 50%, 75% and 90% were also calculated since this would yield insights into the distribution of savings across the numerous projects (which is a non-gaussian distribution).

The aggregated savings fraction for the 236 Bin D projects was 7.2%. The average of fractional savings of individual projects amounted to 10.0%.

A data audit of contractor predictions revealed that 201 of the 236 analyzed projects had usable data regarding final contractor savings predictions. Using the 201 projects, researchers re-calculated total and average evaluated energy savings and contractor-predicted energy savings for those projects. The results are summarized in Table 3. It should be noted that many of the 35 projects dropped were multiple-upgrade projects, some were large projects and all utilized the prescriptive application of the Business program.

TABLE 3. SUMMARY RESULTS OF SAVINGS FRACTIONS COMPUTED FOLLOWING EQUATIONS (1) AND (2)

The results of the Level 1 analysis are summarized in the following section.

(i) There is a major discrepancy between the total savings predicted by the contractors and those determined from weather normalized savings calculations, referred to as evaluated savings (see Figure 9 and Table 3). Both savings calculation methods (namely, the total fractional savings method and average of the individual project method) reveal this discrepancy. While the total fractional method suggests a fractional savings of 5.5% of the baseline energy use, the contractor-predicted savings fraction was 9.8%. Similarly, the average of individual fractional savings method resulted in evaluated fractional savings of 10.4% while the contractor-predicted savings turned out to be 22.4%. Thus, both methods point to a considerable over-estimation of savings by the energy contractors.

FIGURE 9. COMPARISON OF EP EVALUATED AND CONTRACTOR-PREDICTED FRACTIONAL SAVINGS FOR 201 PROJECTS ANALYZED COMPUTED TWO DIFFERENT WAYS (EQS. 1 AND 2)

∑Ni = 1 Annual Savings (kWh)i

∑Ni = 1 Annual Baseline (kWh)i

Savings (kWh)Annual Baseline (kWh) i

N




(ii) The comparison of contractor-predicted savings with aggregated evaluated savings by both weather corrected and non-weather corrected analysis methods is provided by Figure 10. Note that the weather corrected and non-weather corrected (direct month to month comparison) methods yielded results which were relatively close. With weather correction, the total fractional savings came out to 5.5 % whereas the non-weather corrected resulted in a 5.7% savings. These values are close, but we still recommend the use of weather correction as a general approach since it would be of general relevance, i.e., in places where outdoor temperature is an influencing factor of energy consumption in a building.

FIGURE 10. COMPARISON OF CONTRACTOR-PREDICTED TOTAL ANNUAL SAVINGS WITH EP EVALUATED AGGREGATED SAVINGS FRACTION (BY BOTH WEATHER CORRECTED AND NON-WEATHER CORRECTED METHODS). N=201

savings uncertainties are relatively small. This is better illustrated by Fig. 12 which is a sorted distribution of 201 projects. Only 2 projects have uncertainties greater than 200 % (the y-axis scale has been cut off at 40% to provide greater resolution for most of the other projects which had lower uncertainties).

(v) The percentiles reported in Table 3 also provide some insights. Firstly, the contractor predictions are never negative for the 10% percentile (as one would expect) while the evaluated savings are about -0.9% (negative meaning that energy use increased after the ECM was installed). This could be caused by increases in energy use in the building unrelated to the upgrade. However, it is likely that energy use in other buildings could also have declined as a result of dynamic changes in the way the building is operated unrelated to the upgrade. One would then expect these trends to compensate each other to some extent when a sample size consisting of 201 buildings is analyzed. The differences between the contractor-predicted and EP- evaluated savings are roughly of the order of 2:1 for all other percentiles. The contractors over-predict savings by 100% relative to evaluated savings.

FIGURE 11. PLOT DEPICTING DIFFERENCES IN EP EVALUATED ENERGY SAVINGS PERCENTAGE AND THE ASSOCIATED RELATIVE UNCERTAINTY FOR INDIVIDUAL PROJECTS. THE RELATIVE UNCERTAINTIES ARE GENERALLY SMALL THOUGH THERE ARE SOME EXCEPTIONS. THIS PLOT APPLIES FOR 201 PROJECTS SORTED INTO BIN D(iii) Table 3 also assembles results of analyzing the lighting-

only projects. Again, the differences between EP-evaluated and contractor-predicted savings fraction are significant, with the difference being very similar to those from all projects since lighting upgrade is by far the most dominant type of energy upgrade.

(iv) Figure 11 depicts the evaluated savings percentage (i.e. energy savings divided by baseline energy use) on an annual basis for all individual projects along with the associated fractional uncertainty (i.e. energy savings uncertainty divided by energy savings). The uncertainties of the change point models, characterized by their coefficient of variation of the root mean squared error (CV-RMSE) are generally large. However, the baseline model is used to predict energy use each month for the 12 months of the year, and so the uncertainty of the summed values is lower. The relevant formulae are given in various publications (Reddy and Claridge, 2000 or ASHRAE 14, 2002). Except for a small number of buildings, the




FIGURE 12. PLOT DEPICTING THE RELATIVE UNCERTAINTIES IN INCREASING ORDER. THIS PLOT IS SHOWN FOR 199 PROJECTS OUT OF THE 201 PROJECTS AS 2 PROJECTS HAD UNCERTAINTIES GREATER THAN 200%. THE RELATIVE UNCERTAINTIES ARE GENERALLY SMALL FOR THE MAJORITY OF THE PROJECTS

fraction for the 201 projects with usable contractor predictions falling in Bin D. It can be seen that most of the projects are below the expected ratio line of 1 with a few projects assuming large negative values. These fractional and negative projects drive the noted discrepancy. Figure 15 is the same plot redrawn with a narrower y-axis range so as to provide better resolution.

FIGURE 14. SORTED DISTRIBUTION PLOT OF THE SAVINGS RATIO FOR ALL THE 201 PROJECTS ANALYZED

Possible causes for the discrepancy stated under (i) above were investigated. Figure 13 is a plot of the distributions in annual energy savings fraction for the 201 projects predicted by the contractors and those determined by the weather normalized approach. The predicted savings fractions have a noticeably wide distribution across the various projects, exhibiting a long positive tail. On the other hand, the evaluated savings have a tighter distribution, and peak around 15% savings. However, many of the projects show negative savings which is a contributor to why the total EP evaluated energy savings fraction turns out to be so much lower than those predicted by the contractors.

FIGURE 13. COMPARISON OF THE FREQUENCY DISTRIBUTIONS OF ANNUAL SAVINGS PERCENTAGE AS A PERCENTAGE OF BASELINE CONSUMPTION PREDICTED BY THE CONTRACTOR AND AS PER EP EVALUATED SAVINGS DATA ANALYSIS. DATA IS FROM 201 PROJECTS WITH WEATHER CORRECTION

Another way of presenting this discrepancy is shown in Figure 14 as a distribution where the ratio of evaluated to contractor-predicted savings are sorted by magnitude of the savings

FIGURE 15. SAME AS FIG. 14 BUT WITH NARROWER Y-AXIS SCALE (WITHOUT OUTLIER POINTS) FOR BETTER VIEWING OF THE DISTRIBUTION

4.4 Possible Causes for Discrepancy

We also investigated possible causes of differences between predicted and evaluated savings distributions. The following potential causes were identified.

(a) The way visual binning was performed

A possible reason for the discrepancy could be due to bias introduced in the analysis due to the visual manner by which buildings are sorted into the various bins (as described earlier). The savings analysis was repeated without such binning before the audit of contractor savings predictions. Both the contractor-predicted savings and the evaluated savings fractions decreased. However, this analysis could not be repeated post-audit due to data constraints, so it is not possible to make conclusions. It is recommended that, though time consuming, screening and binning be conducted. It is




also recommended that analysis be run on the data both with binning and without binning to investigate if any possible bias has been introduced.

(b) Difference due to single ECM versus multiple ECM upgrades

In another effort to isolate the cause for the discrepancy between contractor-predicted and EP evaluated savings projects were divided into single ECM and multiple ECM categories. Figure 16 provides a direct comparison of the contractor-predicted and evaluated savings percentages. For single projects, the savings predictions by the contractors amounts to 9.2% of baseline energy use, while analysis suggests 3.4% evaluated, a larger discrepancy than for all projects combined. Of the 162 single-ECM projects analyzed, 161 were lighting upgrade projects, suggesting a possible connection specifically between lighting projects and estimation accuracy. . Further investigations were undertaken as part of Level 2 analysis, and are reported later on in this document. For projects adopting multiple ECMs, the discrepancy between contractor predictions and evaluated savings were much less than for all projects combined, (11.0% versus 9.8%).

FIGURE 16. COMPARISON OF CONTRACTOR-PREDICTED SAVINGS AND EP EVALUATED SAVINGS WITH WEATHER CORRECTION FOR SINGLE PROJECTS (162 PROJECTS) AND MULTIPLE ECM PROJECTS (39 PROJECTS).THE PERCENTAGE SAVINGS COMPARED TO THE BASELINE IS ALSO SHOWN IN THE GRAPH

and #7, performed a high number of projects, and consistently over-predicted savings greatly. Figure 18 shows the same plot sorted by the number of projects done by the contractor (the contractor id is consistent with Figure 17).

FIGURE 17. DIFFERENCE BETWEEN PREDICTED AND EVALUATED SAVINGS FOR 12 CONTRACTORS SORTED BY THE PREDICTED SAVINGS. THE NUMBER OF PROJECTS IS INDICATED ABOVE THE INDIVIDUAL BARS

(c) Contractor bias in savings estimation

Another investigation involved determining whether certain

contractors tended to consistently over-predict energy savings.

While there were 36 different contractors in total, there were

12 who undertook numerous projects or projects with large

energy savings. The results of this study are summarized in

Figure 17. We note that most of the contractors over-predicted

savings as compared to evaluated savings with varying

degrees. In particular, four contractors, contractor #1, #5, #6

FIGURE 18. DIFFERENCE BETWEEN CONTRACTOR-PREDICTED AND EP EVALUATED SAVINGS FOR 12 CONTRACTORS SORTED BY THE NUMBER OF PROJECTS. THE NUMBER OF PROJECTS IS INDICATED ABOVE THE INDIVIDUAL BARS

5. LEVEL 2 ANALYSIS

In the case of lighting projects, the contractor-predicted

savings were calculated as the kW reduction multiplied by

the number of hours of operation. The calculation of the

kW reduction appears fairly straightforward since it entails

counting the number of fixtures and using engineering

formulae to account for ballast and other effects. The number

of hours, on the other hand, is an estimate, often supplied by

the building owner.

Level 2 analysis was meant to resolve any noticed discrepancy

with the aid of direct measurements. In the Level 1 analysis,

any lighting projects were found to have contractor kWh saving

predictions which amounted to more than 70% of the baseline

consumption for the building per utility data provided by APS.




One primary reason for such high percentages may have been the lack of a proper framework for matching an account with the master meter at the facility, but the differences could be due to other causes as well.

FIGURE 19. LEVEL 2 ANALYSIS PERFORMED ON VARIOUS SITES FOR ASCERTAINING DISCREPANCY IN ESTIMATED HOURS OF OPERATION BY THE CONTRACTOR AND THE ACTUAL MEASURED HOURS OF OPERATION. THE NUMBERS ABOVE THE BARS ARE THE NUMERICAL VALUES OF THE WEEKLY HOURS OF OPERATION

of the baseline energy use and it was found that the utility bills provided were only from one electric meter while the facility had four electric meters. As mentioned, there was no framework to match master meters with participant accounts.

Another potential source of contractor estimation error is inaccurate assessment of pre-upgrade equipment conditions. Field measurements and internal inspection of fixtures on one project determined that, while the owner and/or contractor had assumed that all existing ballasts consisted of older, inefficient magnetic technology, at least some of the ballasts had been replaced with newer electronic versions during regular maintenance as ballasts had burned out. While a 100% pre-upgrade audit is not a cost-effective solution, this example illustrates the importance of pre-upgrade inspections with appropriate sampling. (APS has a statistical sampling-based inspection process in place for its programs and, for Energize Phoenix, APS agreed to have 100% of projects inspected. The percentage of fixtures/equipment inspected at each project and the depth of inspection is not known.)

Other identified sources of contractor estimation error revealed during the data audit process include simple math errors, reading data from the wrong cells on worksheets (such as kW savings rather than kWh savings) and varying quality of estimation tools. The tools used by contractors to generate savings predictions varied widely in sophistication, usability and utility. Some involved room by room detailed equipment counts, specs and operating hours, and also included predicted HVAC savings from individual lighting heat load reductions. Some were little more than summary word processing documents. Some contractor spreadsheets and the contractor savings reporting spreadsheet provided by EP for prescriptive projects would have benefitted from having formula-driven calculations and cross-checks to minimize math errors.

The causes of discrepancies discussed above could greatly skew either contractor predictions or analysis results, and so it is important for programs to put procedures in place to ensure proper quality control. Other potential causes include the “rebound effect” (using savings to buy more energy-using equipment or being less diligent on behavioral conservation habits) and “energy creep” (a gradual increase in installed plug loads over time) in a facility after the upgrades were completed. The latter issue has been investigated and found to be unimportant for the Energize Phoenix project (reported in Appendix C).

Even though the customer confirms in writing on the incentive application that the operating hours estimated by the contractor are correct, inaccurate operating hours were hypothesized to be a major cause of savings prediction discrepancies. The procedure adopted to test this hypothesis was to install data loggers to determine hours of operation, and then compare them to the values submitted by the contractors. Out of numerous sites visited, twelve sites granted permission to install monitoring devices. Out of the twelve sites, eleven generated usable data. The data loggers were installed for a minimum of two weeks. The position of the data logger was carefully selected to avoid interference by human element or sunlight. The results of the analysis are shown in Figure 19. The results show that eight of the sites visited exhibited differences greater than 10% between reported and measured values of operating hours. Five of the sites had major discrepancies.

For sites #6, 7, 9, 10 and 11, the cause for the observed discrepancies could not be determined. For two of these projects, the actual operating hours were less than half of the value used to predict energy savings. Since the predicted savings depend upon a direct multiplication by this value, inaccuracies lead to a directly proportionate overestimation of the savings.

Another source of discrepancy in the savings calculations could be the quality of the utility bill data itself and how it was designated in the database. A field visit was made to another facility where predicted savings were over 100%




FIGURE 22. COMPARISON OF THE ACTUAL BASELINE MONTHLY UTILITY BILLS AND THE ENERGY CONSUMPTION OF BASELINE COOLING PLANT CALCULATED BY THE ENERGY MODEL FOR THE AZRS BUILDING. NOTE, THAT THE ENERGY MODEL ONLY CONSIDERED A FRACTION OF THE TOTAL ENERGY CONSUMPTION OF THE BUILDING TO ESTIMATE THE SAVINGS.

6. LEVEL 3 ANALYSIS

6.1. Background

The objective of Level 3 analysis was to perform a detailed energy analysis of a few selected projects involving the use of a detailed building energy simulation program. Two sites were selected:

a. The Arizona State Retirement System Building (AZRS), and

b. Mercado D building of Arizona State University

This report only describes analysis work done on the first one, while that for the second building can be found in the Technical Report under preparation (Reddy et al., 2013). Though the latter was a relatively small building, there were several similar buildings on the same campus which suffered from HVAC units which reached their end of life status. Further, these buildings were repurposed several times, and so the rooftop HVAC units were added on incrementally resulting in excess cooling capacity, improper ducting runs and poor air distribution in spaces. The effect of a complete redesign of the HVAC units and of the air distribution system on the energy efficiency of the entire building was studied and is documented in Reddy et al (2013).

Arizona State Retirement System (AZRS) was one of the first projects, and also one of the largest, to be completed using EP incentives. Table 4 assembles some pertinent facts about the building. AZRS is part of the 3300 tower building in Downtown Phoenix (see Fig. 20.). The project involved replacement of chillers and converting to a chilled water variable speed pumping system. These upgrades were predicted to save about 1.26 million kWh of energy annually.

FIGURE 20. PHOTOGRAPH OF THE ARIZONA STATE RETIREMENT SYSTEM BUILDING (AZRS)

TABLE 4. AZRS BUILDING

FIGURE 21. EQUEST MODEL OF THE AZRS BUILDING.

TABLE 4. AZRS BUILDING.

The analysis reports and procedures used by the contractors during the design phase were requested, acquired and used to reanalyze and verify both the energy model and the savings, using actual monitored data. For predicting the savings,




the contractor had developed an energy model using eQuest simulation software (2010) (see Figure 21). However, the model was specifically developed for the cooling plant and for the HVAC system with little effort put into capturing the building loads accurately, which is a rather lengthy and tedious process. The time series plots shown in Figure 22 clearly indicate that the actual utility bills of the building during the baseline period are much higher than that of the cooling plant energy use generated from the eQuest model simulation. However, it is interesting to note that the patterns of variability are quite similar for both plots. The objective of this analysis was specifically to ascertain whether such an approach did yield the savings actually observed once the upgrades were completed.

6.2. Upgrades Suggested and Predicted Energy Savings

The following upgrades were identified and implemented by the contractors:

• Replace the two (2) existing 900 ton Trane Centrifugal Chillers with four (4) new 350 Ton McQuay Frictionless Centrifugal Chillers (see Figure 23). Piping can be easily modified to accept the four new chillers. Existing power supply is adequate with minor modifications. Existing pad size is adequate with minor pad addition to accept an additional chiller per pad. The new chillers will primarily operate below 50% of existing kW/Ton conditions.

• Re-pipe the central plant from a variable primary system to a primary secondary system. This new piping configuration will provide chilled water throughout the building in the most efficient manner possible; meaning water will be pumped only as require by cooling demand.

• Relocate HX within the piping system to optimize efficiency and utilize excess capacity of cooling Towers.

• Reprogram existing Alerton BACNET controls to maximize system staging and operate central plant in the most efficient manner, while maintaining tenant comfort.

FIGURE 23. PHOTO OF THE NEW CENTRIFUGAL CHILLER

The eQuest model (2010) for the water side plant equipment is shown in Figure 24. Table 5 assembles the month-by-month disaggregated baseline consumption and the annual energy savings as predicted by the simulation model. Note that the predicted annual savings are 4,996,000-3,730,000 = 1,266,000 kWh, which is about 25.3% of the baseline value before the upgrades.

FIGURE 24. WATER SIDE PLANT EQUIPMENT AS MODELED IN EQUEST SOFTWARE

FIGURE 25. DISAGGREGATED END USE ENERGY BEFORE AND AFTER THE UPGRADES AS PREDICTED BY THE EQUEST SIMULATION MODEL DEVELOPED

FIGURE 26. PREDICTED ENERGY SAVINGS CONTRIBUTIONS FOR THE AZRS BUILDING




Figure 25 shows the same data in a more visually meaningful manner. As expected from the nature of the upgrades, the space cooling, pumps and auxiliary and the vent fans energy uses before and after the upgrades are distinctly different. The others (area lights, miscellaneous equipment and heat rejection) show no change. Figure 26 depicts the savings from the three individual end-uses. Note that space cooling energy reduction accounts for 76% of the total savings with pumps and auxiliary accounting for much of the remaining savings contribution.

TABLE 5.BASELINE AND PROPOSED ENERGY CONSUMPTION CALCULATED USING ENERGY MODEL

difference is less than 10%, which is quite close in view of the other uncertainties present.

FIGURE 27. THE 5P CHANGE POINT BASELINE MODEL FITTED FOR THE AZRS BUILDING

6.3. Analysis and savings verification

As stated earlier, the eQuest model developed by the contractors was not built to reflect the whole building energy use data. The reasoning was that this was not necessary since only cooling plant and HVAC related upgrades were being performed, and it would be adequate to model them properly to reflect energy use prior to the upgrade. Rather than trying to retune the baseline eQuest model to fit actual data, it was decided to simply evaluate the energy savings as a result of the upgrades and determine whether evaluated savings were consistent with predicted savings.

A change point model (with five parameters) was found to best capture the monthly variations of the whole building energy use prior to the upgrades. This is consistent with the fact that the building had both electric heating and cooling sources. Figure 27 shows how well the 5P model fits actual utility bills. Figure 28 allows comparison of the energy savings as predicted by the contractors and that determined from our analysis (termed evaluated savings). Note that there is a small difference (about 0.9%), with the contractor estimating a 9.7% savings while we found 8.8%. In relative terms, this

FIGURE 28. COMPARISON OF CONTRACTOR PREDICTED AND MEASURED ENERGY SAVINGS FOR THE AZRS BUILDING

6.4. Conclusion

Whenever possible, developing a detailed calibrated model is the best way to analyze energy savings in a building which undergoes multiple upgrades. However, the process is tedious (which translates to labor costs) and requires some amount of expertise both in the use of the software program and in the calibration process. Further, the needed monitoring data may not be available. The norm is to calibrate the model as accurately to the whole building energy use as possible and only then can one obtain an accurate prediction of the effect of different ECMs. The AZRS building model, though it was not well calibrated to the whole building, was modeled accurately enough in terms of the performance of the cooling plant and the distribution system that the energy savings predicted matched quite closely with those determined from actual utility bills before and after the upgrades.




7. PAYBACK ANALYSIS

A simple payback analysis was also performed in order to obtain a conservative projection of the effectiveness of the Energize Phoenix program. Two types of simple payback values were calculated: one without EP rebates, and another with EP rebates. From Table 6, note that for the 236 Bin D projects the projected payback is 9.5 years without rebates and 7.4 years with rebates. Note that there is an additional rebate given by Arizona Public Service on top of the Energize Phoenix rebate, which could not be taken into account since the rebate amounts were not available for our analysis.

TABLE 6. PAYBACK ANALYSIS FOR 236 COMPLETED PROJECTS

FIGURE 29. PROJECTED YEARS FOR PAYBACK DETERMINED BY VARIOUS METHODS

In essence, the payback periods are very high, especially given that the great majority of the EP upgrades were lighting upgrades which tend to have 1-2 year payback. If it had not been for the APS and EP rebates provided and/or for over-predicted savings, it is very likely that these lighting upgrades with total project cost payback periods of 9.5 years would not have been acceptable to the building owners. These calculations, however, do not include what the industry refers to as Non-Energy Benefits (NEBS), such as carbon reduction, increased property valuations and environmental benefits. Other NEBS, such as comfort, durability, indoor air quality, and safety and their resulting impacts on health and productivity may alter the financial payback equation substantially. Since many of these benefits are very complex to estimate financially and some are broader societal benefits that are not captured directly by the building owner, they are unlikely to drive the building owner’s decision-making process. The behavioral team separately analyzed organizational attitudes and motivations for participating in the program.

These differences between payback periods using evaluated and contractor-predicted savings directly attributed to the under-performance of lighting projects. The results are also plotted in Figure 29 for better visualization.

8. CUMULATIVE SAVINGS OVER TIME

Another aspect studied related to how energy savings accrued over the course of the three year EP project as a result of energy upgrades. These savings would depend on the number of projects implemented over time and how they performed subsequently. Figure 30 shows the cumulative plot based on contractor-predicted savings for the 236 projects from bin D. The cumulative energy savings for each month was determined by prorating annual energy savings predicted by the contractor, and adding the total historical savings up to that month. The plot reveals an exponential rise as one would expect. The corresponding number of projects implemented is also shown.

The plot serves to emphasize the ever-increasing disparity between projected and evaluated energy savings.

FIGURE 30. THE CUMULATIVE ADDITION OF CONTRACTOR-PREDICTED SAVINGS AND THE EVALUATED SAVINGS FOR THE 236 BIN D PROJECTS




using utility bills would generally have relative uncertainties much lower than the savings themselves at the individual building level. So, this study reinforces the prevalent outlook in the professional energy service industry that utility bill analysis is a simple and inexpensive way to evaluate energy savings and provide the necessary risk alleviation safeguard which financing entities may stipulate as a condition for providing funding.

In closing, the direct sources of discrepancy between the contractor-predicted savings and evaluated savings could come from:

(i) Contractors’ estimates of operating hours being inaccurate even though the customer is required to sign off on the contractor’s estimate of the number of operating hours at the facility;

(ii) Instances of high burnout in the pre-upgrade case for lighting upgrades while energy savings predictions presume fully operational lighting even when it is known that burnout has occurred;

(iii) Discrepancies in input wattage between installed fixtures & database values. The database values reflect averages for each particular lighting technology and provide conservative estimates for consumption, and

(iv) Errors from inadequate contractor prediction processes and tools, and lack of training of contractor staff in the proper use of those processes and tools,

Pertinent to (ii) above is the inaccurate assessment of pre-upgrade equipment conditions.

If energy efficiency programs are to be scaled substantially, large portfolio financing is one enabler to reach scale. Financing sources need predictable returns in order to invest without requirements for the high risk premiums warranted by uncertainty. A possible means of reconciling predicted versus actual performance, either by installing data loggers or by field visit surveys has been suggested. The proliferation of interval data from smart meters also opens up new possibilities for increasing estimation accuracy at the individual building level and through analysis of “Big Data” at the program level.

References

ASHRAE Guideline 14 (2002), ASHRAE Guideline 12-2002. Measurement of Energy and Demand Savings. American Society of Heating, Refrigeration and Air-Conditioning Engineers, Atlanta, GA.

9. SUMMARY AND SUGGESTIONS FOR FUTURE ENERGY CONSERVATION PROGRAMS

In summary, the major conclusion of the EP commercial analysis effort is that contractor Predicted savings are much higher than those actually determined from analysis of actual utility billing data. While the former was around 9.8% of the baseline energy use for the entire program till end of April 2013, the EP evaluated savings were only 5.5%. The discrepancy in average savings of individual projects was 22.4% versus 10.4%. Higher discrepancy was found in lighting-only projects than in projects with multiple ECMs.

Contractor bias accounts for some, if not much, of the observed differences in savings. One suggestion is that contractors be provided with utility bills or smart meter data of the facility at the time of estimating savings. This would better inform the contractor and minimize inappropriately high savings predictions. Another suggestion is that contractors be provided with a suite of savings estimation tools and incentives to use them (such as faster processing time for projects that utilize the provided tools). This could not only reduce estimation errors but also reduce program administration costs analyzing savings predictions from a wide range of tools of varying quality. Also, contractors should be provided feedback on how their predicted savings stack up against evaluated savings, i.e. inform them of the performance-based evaluated savings. This education would enable them to make quality control corrections to their audit process so as to produce more realistic savings predictions.

From an evaluation standpoint, all building data should be screened in order to identify and remove spurious data spikes and patterns. The discrepancy between evaluated and contractor-predicted savings fraction increases greatly when binning is not undertaken. This leads to a tentative conclusion: in order to have sufficient confidence in savings analysis results, it is strongly advisable to first screen the data even though it is a time consuming process. The process could be somewhat automated, but this effort was deemed to be beyond the limited resources and time commitment of the commercial team of the EP program.

Further, even though the differences were not large in the EP program, it is advisable when evaluating savings using pre- and post-upgrade utility bills to perform weather normalization in a routine manner. The savings estimations would likely be more realistic and program managers would tend to view these corrections as warranted.

Finally, change point models used for weather normalization




eQuest (2010). Building energy simulation software program, http://www.doe2.com/equest/

Haberl J.S. and Culp, C. (2007). “Measurement and Verification of Energy Savings,” in Energy Management Handbook (Edited by W.C. Turner), Fairmont Press.

IPMVP (2010). International Performance Measurement and Verification Protocol, Volume 1. Efficiency Valuation Organization, U.S. Department of Commerce, Springfield, VA.

Kissock, J. K.,Haberl, J. S. and Claridge, D. E (2002) “Development of a Toolkit for Calculating Linear, Change-Point Linear and Multiple-Linear Inverse Building Energy Analysis Models,” ASHRAE RP 1050, American Society of Heating, Refrigeration and Air-Conditioning Engineers, Atlanta, GA.

Kissock, J.K., Reddy, T.A. and Claridge, D.E. (1998). “Ambient-Temperature Regression Analysis for Estimating Retrofit Savings in Commercial Buildings,” ASME Journal of Solar Energy Engineering, vol. 120, no. 3, p. 168.

Reddy, T.A., Claridge, D.E. (2000), Uncertainty of “Measured” Energy Savings from Statistical Baseline Models. HVAC&R Research, Journal Vol. 6, no. 1, pp. 3-20, American Society of Heating, Refrigeration and Air-Conditioning Engineers, Atlanta, GA.

Reddy, T.A., K. Thalapully, M. Myers and O. Nishizaki (2013). Final Technical Report by the Non-Residential Analysis Team of the Energize Phoenix Program, in preparation, Arizona State University, Tempe, AZ.

APPENDIX A: BASELINE MODEL DEVELOPMENT AND UNCERTAINTY

The savings methodology adopted is consistent with the one suggested in the professional literature (see for example, Haberl and Culp 2007).The process includes the following steps:

1. Acquire monthly energy use data (from utility bills) and data on influential variables (limited in this study to outdoor dry-bulb temperature) during the pre-upgrade period.

2. Develop a regression model of pre-upgrade energy use as a function of influential variables - this is the “baseline model”.

3. Acquire date of energy use (from utility bills) and influential variables during post upgrade period.

4. Use the values of influential variables from the post upgrade period (from step 3) in the pre upgrade model (from step 2) to predict how much energy the building would have consumed on a monthly if it had not been upgraded.

5. Subtract measured post upgrade energy use (step 3) from the predicted pre-upgrade energy use (step 4) to estimate savings on a monthly basis.

6. Sum the individual monthly savings to determine cumulative (or annual) savings and percentage savings.

7. Compare the model goodness-of-fit (using the coefficient of variation of the root mean square error or CV-RMSE) with the percentage savings determined.

The model approach is statistical in nature, involving identifying a regression model of monthly energy use against monthly mean outdoor temperature using the monthly mean temperature model (Kissock, Reddy and Claridge 1998). The ambient temperature is chosen as the only independent variable because of the easy availability of the data, the difficulty in acquiring other data, and to avoid statistical difficulty arising from a small data set (only 12 data points) and multi-collinearity with environmental indices such as ambient humidity and solar radiation.

Another significant parameter to be considered is the uncertainty in the baseline model for a specific site characterized by the CV-RMSE (coefficient of variation of the root mean square error) of the model. This allows direct insights into the statistical soundness of the associated savings deduced. The CV-RMSE is a rough measure of the fractional (or percentage) uncertainty in the baseline model compared to the mean baseline energy use. A 10% CV-RMSE would imply that model uncertainty is 10% of the mean annual pre-upgrade energy use. If the savings fraction is less than about twice the CV-RMSE (corresponding to approximately 95% confidence level), then one is unjustified statistically in placing too much confidence in the associated savings estimated at that site. Adding this filter criterion to the analysis would have further reduced the total number eligible projects within the EP program. So for a single project all models were evaluated as shown in Fig. A1, and the model with the least CV-RMSE was chosen as the best fit baseline model.

FIGURE A1. PROCESS OF DETERMINING THE BEST FIT REGRESSION MODEL FOR A SPECIFIC PROJECT INVOLVES FITTING ALL FORMS OF CHANGE POINT MODELS AND IDENTIFYING THE ONE WITH THE LEAST ROOT MEAN SQUARE ERROR (RMSE)




APPENDIX B. MODEL IMPROVEMENT AFTER ADJUSTING FOR UTILITY BILL READ DATES

B1. Problem Statement

Utility bills are generally read every month, but the exact dates can vary by a few days from one month to the next. In a large project such as Energize Phoenix involving hundreds of commercial buildings, the read dates for each building are spread out throughout the month. Taking this into consideration while automating the baseline modeling, and then calculating savings for each building, is rather error-prone and tedious especially in the calculation of the associated monthly mean ambient dry-bulb temperatures needed for weather normalization as described in Appendix A. A simplified approach is to assume the utility bills to correspond to the calendar month and use the corresponding monthly mean ambient dry-bulb temperatures which will then be the same for all projects. A better approach, and the one adopted in the Energize Phoenix program, is to weight the utility bill reading during adjacent months with the number of days in each consecutive month, and thereby, obtain adjusted utility bills corresponding to the calendar months. This appendix reports on the differences between the adjusted utility bill results and those based on the raw or non-adjusted ones.

B2. Comparison of Model Goodness-of-Fit

Figure B1 assembles the results of analyzing 151 projects by both methods. The CV_RMSE (coefficient of variation root mean square error) of the individual projects is determined from both data sets. It is evident the CV_RMSE, which is a direct indicator of the goodness-of-fit of the model, has reduced considerably from the unadjusted data set (termed ‘From actual utility bills’). A better way to illustrate this improvement is provided by Figure B2 where the projects have been sorted by increasing CV_RMSE values of the unadjusted utility bill results. Except for 5 projects, the improvement in the resulting change point models for the adjusted data set is very striking.

B3. Energy Savings Comparison

Table B1 assembles the results of the energy savings by both methods. The results using adjusted utility bills are obviously the ones where one would place more confidence. It is clear that there is an important bias in savings between both data sets, and so, the extra step in adjusting the utility bills to correspond to calendar months is warranted.

TABLE B1. ENERGY SAVINGS FRACTIONS DETERMINED BY ADJUSTED AND NON-ADJUSTED UTILITY BILL DATA (151 PROJECTS)

B4. Conclusion

From the analysis results obtained, it is recommended that utility bills adjusted to match calendar months should be used to calculate energy savings. The change points models identified are clearly superior, while the energy saving results from both methods are different enough to warrant the extra step of utility bill adjustment. Hence, it is highly recommended that energy conservation programs, akin to Energize Phoenix, adopt this methodology.

FIGURE B1. COMPARISON OF CHANGE POINT MODEL CV_RMSE SORTED BY PROJECT ID BETWEEN THE RAW OR UNADJUSTED UTILITY BILL DATA AND THE REVISED UTILITY BILLS ADJUSTED TO CALENDAR MONTHS)

FIGURE B2. SAME AS FIG. B1 BUT SORTED BY CV_RMSE OF THE UNADJUSTED UTILITY DATA. THE IMPROVEMENT IN MODEL FIT WHEN USING ADJUSTED DATA IS CLEARLY NOTICEABLE




APPENDIX C: ENERGY CREEP ANALYSIS

C1. Background

Energy creep of a building can be defined as a gradual increase in the energy consumption in the building over time due to increase in plug loads (computers, lights…) or other operational parameters such as changing thermostat set points, increased number of occupants. This factor does not (rather, should not) include changes in other operational parameters such as longer operating hours of the building, or installing additional HVAC equipment.

The study of energy creep is important in a program such as the Energize Phoenix (EP) program. It can mask some of the predicted energy savings due to implemented energy conservation measures, and help undermine the credibility of energy conservation programs. In other words, if a building had shown a continuous energy increase in the years prior to the implementation of the upgrades, then the effect of the upgrades would be under-predicted if the effect of energy creep were neglected.

C2. Issue Investigated

We wanted to determine whether the commercial buildings which underwent upgrades under the EP program suffered from this problem or not. This could have a direct bearing on the evaluated savings determined from the analysis methodology.

C3. Calculation Methodology

Since the EP program started in 2011, the baseline year was chosen to be 2010, and all projects in Bin D which had complete data were taken as the population set. The objective was to simply determine whether energy use during 2010 was statistically different from 2009 or not. This would involve developing a baseline change point model for 2010 for each building and using it to predict energy use for 2009. The difference between the energy use between the two years was attributed to energy creep in that building. Though we realize that random effects would introduce bias and uncertainty in our analysis at the individual building level, taken over the whole population of buildings, this random effect is likely to get smoothened out.

C4. Results and Conclusion

The total number of projects analyzed here is 204 as opposed to 236 which was the total number of projects in the final analysis (see main body of report). This discrepancy is because of data abnormality in the other 32 projects which led us to reject these projects.

Figure C1 shows the results of the analysis for the 204 projects. Here, the projects with positive increase from 2009 would suggest the presence of energy creep while those with negative creep would show a decrease in energy consumption. The graph is almost symmetric indicating that there is no observable energy creep during this period across all the projects.

Figure C2 plots summary results across all 204 projects analyzed. All three average indices are negative suggesting in fact that 2010 energy use is very slightly lower than that of 2009. The median difference between 2009 and 2010 for all the projects is only 0.9%, which is well within the uncertainty range of the models, and can be effectively interpreted as zero. The same conclusion applies to the mean and the total savings fraction also. Thus, we would conclude that though there are differences in individual projects, taken as a whole, there is no statistical evidence to conclude that creep is a factor which needs to be considered in the commercial building data set analyzed within the framework of the Energize Phoenix program.

APPENDIX D: CASE STUDY: LEVEL 2 ANALYSIS OF GARAGE LIGHTING RETROFIT

D1. Background

A lighting upgrade which involved replacing 378 T12 light fixtures 8’ long with 756 T8 light fixtures 4’ long was performed in the three levels of the underground Parking Garage of the Arizona State University Nursing and Health Innovation (NH1) Building. Electrical and luminance measurements, both before and after the upgrade, were conducted. The main intent was to physically measure and study the effects of energy upgrades on both illumination levels and energy consumption.

FIGURE C1. DISTRIBUTION OF PERCENTAGE ENERGY CHANGE FROM 2009 TO 2010 FOR THE 204 PROJECTS ANALYZED FOR CREEP




D2. Objective

The objective of this study was to sub-meter the energy savings and the lighting quality improvement due to retrofits. This building had numerous advantages which made it ideal for measurements. Normally, getting access to one of the commercial projects for performing measurements was a difficult task. Being an ASU building, the team had easier access to this structure. The nature of the upgrade was also another advantage. The parking garage had lights which operated 24 hours a day throughout the year and so the measurements could be taken for a small period which could then be extrapolated for the whole year. The study was

D3. Methodology

The power savings determined by the measurements were directly compared with the value predicted by the contractor. The contractor-predicted savings were determined with feedback from the customer about the details of individual lamps, such as quantity, type, operating hours, etc. This information was then entered into a standard spread sheet to compare it with the upgraded equipment ratings to determine the energy savings. This method is typical of the methodology adopted in most of the EP projects of such type.

The commercial team noted that this simple spreadsheet method also added cooling load savings (resulting from the decrease in the heat output of the lights), with predicted cooling load savings of approximately 15,000 kWh/year. This is incorrect since the parking garage was not air-conditioned, and so the credit associated with cooling load reduction does not apply in this case. It is also worth mentioning that this discrepancy was rectified by the contractor itself and a more accurate value, which didn’t include the cooling load savings, was provided to us with which to perform the utility bill analysis. Such estimation errors can be avoided with fairly simple documentation tools and procedures and, if possible, by physical inspection by the concerned authorities to avert such mistakes.

The lumen levels were also measured at various locations in the garage both before and after the upgrade. The measurement was made using a standard light meter and the measurements were done using procedures prescribed in IES standards (Refer to full technical report for more information).

Finally the savings for this particular project were determined from a utility bill analysis. The results were compared with the contractor predictions and the measurement results made by the EP team to validate the results. A few concerns and inferences are reported below.

D4. Results of Savings Measurements

From Table D1 it is evident that there has been a reduction in the power consumption in both measured as well as the contractor predictions due to the lighting upgrades. The garage level 1 upgrades were not fully complete at the time of the post-upgrade site measurements and the lamps upgraded were wired to different electrical circuits. Hence, these values were intentionally not used in our analysis. The overall savings were thus determined for the levels 2 and 3 of the parking garage assuming year-round operation. The savings result as a percentage decrease is shown in Table D2.

FIGURE C2. COMPARISON OF THE FRACTIONAL CHANGE IN ENERGY CONSUMPTION FROM 2009 TO 2010 FOR THE 204 PROJECTS ANALYZED. NEGATIVE SAVINGS IMPLY A REDUCTION IN ENERGY USE OR A NEGATIVE CREEP

FIGURE D1. THE ARIZONA STATE UNIVERSITY NURSING AND HEALTH INNOVATION (NH1)

conducted in year 2 of the Energize Phoenix project and has

been documented in detail in year 2 report. In year 3, an

additional analysis was done using the utility data to verify

and compare with the earlier results. This is the main focus of

this report.




TABLE D1. MEASURED POWER AND CONTRACTOR-PREDICTED POWER AT VARIOUS LEVELS

TABLE D2. THE ANNUAL SAVINGS ESTIMATED BY MEASUREMENT AND CONTRACTOR PREDICTIONS

The lumen levels increased after the lighting upgrade. The readings for one of the floors is assembled in Figure D2 for illustrative purposes. The figure also indicates the lumen levels for four locations marked (shown in red on the grid map). The pre and post upgrade lumen levels for these locations are shown in the adjacent bar chart (one for each location). For instance, the location A/B had an increase in lumen levels in all three orientations in which lumen levels were measured. The result is also visually obvious from the pre and post upgrade photographs of Figure D3 corresponding to garage

The utility data for the project was obtained in year 3 of the Energize Phoenix project. This allowed re-evaluation of the savings predictions by the contractor against analysis results. Note that the data obtained was for the whole building and the consumption of the garage could not be isolated from it. The contractor-predicted savings fraction was about 4% of

level 1.These results are typical of all the three garage levels where lighting values where measured. We had 8 points for

each floor where the readings were measured. The complete details of the measurements can be found are in the EP summative report for year 2.

FIGURE D2. PRE AND POST UPGRADE LIGHTING LEVELS FOR FOUR LOCATIONS AT A SPECIFIC GARAGE LEVEL. THE PLAN IS SHOWN ON THE LEFT WITH THE RED DOT INDICATING THE LOCATIONS WHERE THE MEASUREMENTS WERE TAKEN. THE READINGS FOR THESE LOCATIONS ARE SHOWN IN THE HISTOGRAMS ON THE RIGHT.

FIGURE D3. PHOTOGRAPHIC COMPARISON OF LUMEN LEVELS BEFORE AND AFTER RETROFIT

D5. Comparison with Utility Bill Analysis

FIGURE D4. ENERGY CONSUMPTION HISTORY OF THE NH1 BUILDING

baseline consumption. This suggests that the total energy consumption of the parking garage was a minor fraction of the

Pre-upgrade picture of garage parking level 1

Post-upgrade picture of garage parking level 1




whole building. So, it is highly likely that the savings value we would determine may not be sensitive enough to discern such a small difference. We also studied the three years of utility

between the energy savings predicted by the contractor and that from our analysis. This may be due to one or more of the following reasons.

a) In the initial pre upgrade measurements, two of the individual fixtures were inspected and it was found that

bills prior to the retrofit, and found good consistency in the usage pattern (see Figure D4). We also made sure that there have been no other major energy upgrades in the building. Thus, the savings determined from a utility bill analysis can be assumed to be solely due to the parking garage energy upgrades.

Figure D5 is a scatter plot of the utility bills versus outdoor temperature. The scatter is quite consistent, and is well fit by a 5 parameter change point model as shown.

FIGURE D5. SCATTER PLOT OF THE UTILITY BILLS VERSUS OUTDOOR TEMPERATURE TO WHICH A 5P CHANGE POINT REGRESSION MODEL HAS BEEN IDENTIFIED

Finally, the savings results obtained by different analysis methods are summarized in Figure D6. It is clear that energy savings determined by our analysis (1.8%) are only half of that predicted by the contractor. The effect of weather normalization is relatively minor.

FIGURE D6. ENERGY SAVINGS BY CONTRACTOR PREDICTIONS AND WEATHER CORRECTED AND NON-WEATHER CORRECTED METHODS

D6. Conclusion

The measured power consumption decreased, as is evident

from measurements and utility bill analysis. There is also a considerable increase in the lumen levels from pre upgrade levels. However, there was a large (about 50%) discrepancy

one of them had an electronic ballast and the other one had a magnetic ballast (which consumes considerably more power). Since there were no data available on the actual numbers of electronic and magnetic ballasts present in the circuit before the retrofit, the contractor had assumed (or the owner had indicated to the contractor) that all the lamps had magnetic ballasts.

b) A few lighting circuits had emergency lamps connected to them. These lamps run on batteries and only use power to charge the batteries when the battery charge is below a threshold level. It was assumed that the batteries were fully charged all the time, since these lamps turn on only when there is a power outage. So their consumption was assumed to be negligible when doing the measurements. But, perhaps not coincidentally, the circuits with emergency lamps showed much less reduction in post retrofit consumption than expected. Since the power consumption of these lamps could not be separated out in our measurements, this might be another reason why evaluated savings are lower than those of the contractor.

c) Our measurement of electrical power did not involve a 3-phase power measurement but, rather, separate readings of current and voltage. Such an approach gives only an approximate estimate of the actual power consumption value and is not always accurate. This might also be a reason for the difference between the values predicted by the contractor and the measured ones.

d) Finally, recall that the lighting upgrades proposed by the contractor were only 4% of the baseline energy use of the




building. The savings results predicted by the utility bill analysis are likely to have uncertainties larger than the amount of savings being evaluated, and so we cannot place too much confidence on the utility bill analysis results.

In summary, the overall conclusions subject to the caveats stated above are:

1) Post-upgrade illumination levels are consistently and significantly higher than those prior to the retrofit.

2) The post upgrade illumination levels exceed IES recommended parking garage lighting levels. (A large percentage of the pre lighting levels were below the recommended levels)

3) The evaluated savings were much lower than those predicted by the contractor. Using direct measurements, post-upgrade power consumption savings were evaluated at 5% (for two parking garage levels) whereas the savings predicted by the contractor were 30 %.

4) A utility bill analysis suggests that the upgrades saved only half of what was predicted by the contractor for the facility (3.9 % to 1.8%).

appendix h of garage lighting retrofit 85 energy … · 2016-07-22 · analysis team in the...

Documents