natural resources canada · 2020. 11. 9. · natural resources canada office of energy efficiency...

NATURAL RESOURCES CANADA OFFICE OF ENERGY EFFICIENCY

INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM PROCESS AND IMPACT EVALUATION REPORT

QuickTime™ et undécompresseur TIFF (LZW)

sont requis pour visionner cette image.

JULY 2005

ADEC-Zariffa Consortium 3640 McCarthy Ville Saint-Laurent, QC H4K 2A6 Telephone: (514) 332-7606 Fax: (514) 331-4692 E-mail: [email protected]


ADEC-Zariffa Consortium i

TABLE OF CONTENTS PAGE

ACKNOWLEDGMENTS ................................................................................................... IX

EXECUTIVE SUMMARY ...................................................................................................X

1. PROGRAM DESCRIPTION .........................................................................................1

1.1. PROGRAM HISTORY....................................................................................3 1.2. FINDINGS...................................................................................................6

2. SCOPE OF STUDY ...................................................................................................7

3. IMPACT EVALUATION METHODOLOGY ......................................................................8

3.1. DATA TRACKING SYSTEM............................................................................8 3.1.1. Participant Data ...........................................................................8 3.1.2. Non-Participant Data....................................................................9 3.1.3. Auditor Data.................................................................................9

3.2. PARTICIPANT METHODOLOGY......................................................................9 3.3. NON-PARTICIPANT METHODOLOGY............................................................13 3.4. AUDITOR METHODOLOGY..........................................................................14

4. PROCESS EVALUATION METHODOLOGY .................................................................15

5. PROCESS EVALUATION RESULTS...........................................................................19

5.1. PROGRAM THEORY ..................................................................................19 5.1.1. Program Justification and Design ..............................................19 5.1.2. Program Objectives ...................................................................20 5.1.3. Evaluation Issues.......................................................................20 5.1.4. Staffing.......................................................................................22


ADEC-Zariffa Consortium ii

5.2. PROGRAM MARKETING .............................................................................22 5.3. PROGRAM IMPLEMENTATION .....................................................................23 5.4. PROGRAM TRACKING................................................................................23 5.5. AUDIT EVALUATION ..................................................................................25

6. AUDITOR SURVEY RESULTS ..................................................................................29

6.1. STATISTICS..............................................................................................29 6.2. AWARENESS AND FIRST CONTACT.............................................................30 6.3. SATISFACTION..........................................................................................31 6.4. PROGRAM IMPACT....................................................................................31 6.5. PROGRAM COMPARISON...........................................................................32 6.6. PROGRAM EXPECTATIONS ........................................................................33 6.7. AUDITOR SURVEY FINDINGS AND CONCLUSIONS.........................................34

7. NON-PARTICIPANT SURVEY RESULTS ....................................................................36

7.1. STATISTICS..............................................................................................36 7.2. AWARENESS ............................................................................................37 7.3. NON-PARTICIPANTS’ REASONS..................................................................38 7.4. NON-PARTICIPANTS’ RECOMMENDATIONS AND SUGGESTIONS .....................39 7.5. NON-PARTICIPANT SURVEY CONCLUSIONS ................................................39

8. PARTICIPANT SURVEY RESULTS ............................................................................41

8.1. DATA PROCESSING ..................................................................................41 8.1.1. Second Level Selection Filter ....................................................43

8.2. NON IMPACT ISSUES RELATED TO PARTICIPANTS’ RESULTS.........................43 8.3. VALIDATION .............................................................................................44 8.4. FREE-RIDERSHIP ISSUES...........................................................................44 8.5. REASONS FOR NON-IMPLEMENTATION.......................................................47 8.6. SPILLOVER ISSUES ...................................................................................48 8.7. PRIOR EXPERIENCE..................................................................................48 8.8. REASONS FOR PARTICIPATION ..................................................................49 8.9. PAYBACK ISSUES .....................................................................................49 8.10. AUDITOR SELECTION ................................................................................49


ADEC-Zariffa Consortium iii

8.11. SATISFACTION WITH PROGRAM COMPONENTS ............................................50 8.12. AWARENESS ISSUES.................................................................................50 8.13. COMMENTS AND RECOMMENDATIONS ........................................................51 8.14. ANALYSIS OF PARTICIPANTS’ NON-IMPACT ISSUES......................................52

9. ENERGY IMPACT EVALUATION RESULTS.................................................................54

9.1. EXTRAPOLATION GJ/Y..............................................................................54 9.2. GROSS SAVINGS POTENTIAL .....................................................................55 9.3. NET SAVINGS OF IMPLEMENTED MEASURES ................................................55 9.4. NET SAVINGS FROM INTENTION TO IMPLEMENT...........................................56 9.5. SAVINGS RATIOS......................................................................................58

10. GREENHOUSE GAS REDUCTION ...........................................................................60

10.1. IMPLEMENTED MEASURES.........................................................................60 10.2. INTENTION TO IMPLEMENT .........................................................................61

11. CONCLUSION .....................................................................................................62

11.1. IMPACT ISSUES ........................................................................................62 11.2. PROCESS ISSUES .....................................................................................64 11.3. TRACKING ISSUES ....................................................................................64 11.4. AUDITOR’S ISSUES ...................................................................................65 11.5. NON-PARTICIPANTS’ ISSUES .....................................................................66 11.6. PARTICIPANTS’ ISSUES .............................................................................66

12. RECOMMENDATIONS ...........................................................................................69

APPENDIX A: PROPOSED ENERGY AUDIT SUMMARY FORM ............................................71

APPENDIX B: AUDITOR SURVEY STATISTICS .................................................................72

APPENDIX C: NON-PARTICIPANT SURVEY STATISTICS ...................................................73


ADEC-Zariffa Consortium iv

APPENDIX D: PARTICIPANT SURVEY STATISTICS ...........................................................74

APPENDIX E: EXPERT AUDIT APPRAISAL #1 .................................................................75

APPENDIX F: EXPERT AUDIT APPRAISAL #2 .................................................................76

APPENDIX G: EXPERT AUDITS APPRAISAL #3 ...............................................................77

APPENDIX H: AUDITOR QUESTIONNAIRE .......................................................................78

APPENDIX I: NON-PARTICIPANT QUESTIONNAIRE ..........................................................86

APPENDIX J: PARTICIPANT QUESTIONNAIRE .................................................................91


ADEC-Zariffa Consortium v

TABLES AND GRAPHS PAGE

TABLE 1.1: GENERAL FEATURES OF THE PROGRAM AND MAIN RESULTS OF THE ANALYSIS........................... XIII

TABLE 1.1: PROGRAM BUDGET $ K ...........................................................................................................4

TABLE 1.2: PROGRAM EXPENDITURES $ K..................................................................................................4

TABLE 1.3: PARTICIPANTS – HISTORY BY FISCAL YEAR ...............................................................................4

TABLE 1.4: PARTICIPANTS – HISTORY BY INDUSTRIAL SECTOR ....................................................................5

TABLE 1.5: PARTICIPANTS HISTORY BY PROVINCE ......................................................................................6

TABLE 5.1: PROGRAM PERFORMANCE INDICATORS ...................................................................................21

TABLE 5.2: AUDIT REPORT EVALUATION CRITERIA ....................................................................................25

TABLE 5.3: AUDIT EVALUATION COMPOUND RESULTS ...............................................................................27

TABLE 5.4: AUDIT EVALUATION VS. AUDIT APPRECIATION .........................................................................28

TABLE 6.1: DISTRIBUTION OF POPULATION AND RESPONDENTS BY PROVINCE............................................29

TABLE 6.2: Q1. HOW DID YOU LEARN ABOUT THE INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM? ..........30

TABLE 6.3: Q02. IN GENERAL, WHO INITIATED THE FIRST CONTACT BETWEEN YOUR FIRM AND THE CLIENT FOR WHOM YOU HAVE DONE THE AUDIT?...................................................................................................30

TABLE 6.4: Q02A-B. HOW DID THESE CLIENTS FIND YOU?..........................................................................31

TABLE 6.5: Q03A-F. ON A SCALE FROM ONE TO FIVE, ONE BEING "UNSATISFACTORY" AND 5 BEING "VERY SATISFACTORY", HOW WOULD YOU EVALUATE THE INDUSTRIAL ENERGY AUDIT INCENTIVE IN TERMS OF THE FOLLOWING? ....................................................................................................................................31

TABLE 6.6: Q04. WOULD YOU SAY THAT THIS PROGRAM HAS ….................................................................32

TABLE 6.7: Q05. WOULD YOU SAY THAT THE INCENTIVE LEVEL LIMITS YOU IN THE SCOPE OF THE ENERGY AUDIT?....................................................................................................................................................32

TABLE 6.8: Q06. HAVE YOU PARTICIPATED IN A SIMILAR ENERGY AUDIT INCENTIVE PROGRAM BEFORE (I.E., UTILITY OR PROVINCIAL PROGRAM)? .................................................................................................32

TABLE 6.9: Q06A. HOW WOULD YOU RATE NRCAN’S INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM FOR INDUSTRIAL ENERGY INNOVATORS COMPARED TO THE OTHER PROGRAM(S)? ......................................32

TABLE 6.10: Q06B CAN YOU PLEASE GIVE US ONE OR TWO REASONS FOR YOUR ASSESSMENT?..................33

TABLE 6.11: Q07. DOES THE PROGRAM MEET YOUR EXPECTATIONS? ........................................................33


ADEC-Zariffa Consortium vi

TABLE 6.12: Q08 DO YOU HAVE ANY GENERAL COMMENTS ABOUT THE PROGRAM OR SUGGESTIONS ABOUT HOW IT COULD BE IMPROVED? .......................................................................................................34


TABLE 7.2: Q01. AS AN INDUSTRIAL ENERGY INNOVATOR, ARE YOU AWARE THAT THERE IS A FINANCIAL INCENTIVE AVAILABLE FOR YOUR COMPANY FROM NRCAN TO HELP YOU DO AN ENERGY AUDIT? .................37

TABLE 7.3: Q02. WOULD YOU LIKE TO RECEIVE DOCUMENTATION ABOUT THIS PROGRAM OR BE CONTACTED BY THE PROGRAM AUTHORITY TO OBTAIN INFORMATION ABOUT THIS PROGRAM? ......................37

TABLE 7.4: NRCAN RECORDS SHOW THAT YOUR COMPANY HAS NOT YET PARTICIPATED IN THE INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM OFFERED TO ALL INDUSTRIAL ENERGY INNOVATORS SUCH AS YOUR COMPANY. IS THIS CORRECT?............................................................................................37

TABLE 7.5: Q03 FROM THE LIST OF REASONS I WILL READ YOU, PLEASE TELL US WHY YOU HAVE NOT YET PARTICIPATED IN NRCAN'S AUDIT INCENTIVE PROGRAM?...........................................................................38

TABLE 7.6: Q04 FROM THE FOLLOWING POSSIBLE CHANGES TO THE PROGRAM, WHICH ONE(S) WOULD BE STRONG ENOUGH TO ENSURE YOUR COMPANY'S PARTICIPATION IN THE PROGRAM? ....................................39

TABLE 8.1: PARTICIPANT DATABASE ANALYSIS .........................................................................................42

TABLE 8.2: RESULT OF SECOND FILTER APPLICATION ...............................................................................43


TABLE 8.4: QINT04. ACCORDING TO NRCAN RECORDS, YOUR COMPANY HAS RECEIVED AN INCENTIVE TO UNDERTAKE AN ENERGY AUDIT AT YOUR COMPANY'S (CITY) FACILITY. IS THIS INFORMATION CORRECT?........44

TABLE 8.5: Q01. PRIOR TO PARTICIPATING IN NRCAN'S INDUSTRIAL ENERGY AUDIT PROGRAM, DID YOUR COMPANY HAVE PLANS TO DO AN ENERGY AUDIT THAT WOULD HAVE TAKEN PLACE WITHIN A YEAR OF WHEN YOUR NRCAN-FUNDED AUDIT WAS ACTUALLY DONE? ......................................................................44

TABLE 8.6: Q02. IF YES, WOULD IT HAVE BEEN SIMILAR IN SCOPE TO THE ONE THAT WAS EVENTUALLY FUNDED BY NRCAN?...............................................................................................................................45

TABLE 8.7: Q04. TO YOUR KNOWLEDGE, HAS YOUR COMPANY ACCESSED AND/OR PARTICIPATED IN ONE OR MORE OF THE FOLLOWING NRCAN INITIATIVES... ..................................................................................45

TABLE 8.8: Q04A. WOULD YOU SAY THAT YOUR INVOLVEMENT IN THIS/THESE INITIATIVE(S) INFLUENCED YOU TO PARTICIPATE IN THE INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM?.........................................45

TABLE 8.9: Q04B. WHICH NRCAN INITIATIVE WOULD YOU SAY HAS INFLUENCED YOU THE MOST?................46

TABLE 8.10: Q04C. DID ONE OR MORE OF THESE INITIATIVES ENCOURAGE YOU TO SPECIFICALLY ASK TO STUDY OPPORTUNITIES TO SAVE ENERGY AND/OR DOLLARS WHEN UNDERTAKING YOUR ENERGY AUDIT?......46

TABLE 8.11: Q04D. DID ONE OR MORE OF THESE INITIATIVES ENCOURAGE YOU TO LOOK INTO THE COST-EFFECTIVENESS OF ENERGY EFFICIENCY MEASURES WHEN EVALUATING DIFFERENT OPTIONS?....................46

TABLE 8.12: Q06A. AMONG THE REASONS THAT I AM GOING TO LIST, WHICH ONE(S) EXPLAINS WHY YOU HAVE NOT IMPLEMENTED ANY OF THE MEASURES RECOMMENDED IN YOUR ENERGY AUDIT REPORT?...........47


ADEC-Zariffa Consortium vii

TABLE 8.13: Q06B. AMONG THE REASONS THAT I AM GOING TO LIST, WHICH ONE(S) EXPLAINS WHY YOU HAVE NOT IMPLEMENTED SOME OF THE MEASURES RECOMMENDED IN YOUR ENERGY AUDIT REPORT? .......47

TABLE 8.14: Q07. DOES YOUR COMPANY OPERATE OTHER PLANTS? .........................................................48

TABLE 8.15: Q07A. DID YOU USE THE RESULTS OF THIS ENERGY AUDIT TO IMPLEMENT THE SAME MEASURES IN YOUR OTHER FACILITIES WITHOUT DOING ANOTHER ENERGY AUDIT?......................................48

TABLE 8.16: Q08. DID YOU HAVE AN ENERGY AUDIT (EITHER COMPREHENSIVE OR LIMITED IN SCOPE) DONE AT YOUR FACILITY PRIOR TO PARTICIPATING IN NRCAN'S INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM?.............................................................................................................................................48

TABLE 8.17: Q09A-E. ON A SCALE FROM ONE TO FIVE, ONE BEING "UNIMPORTANT" AND FIVE "VERY IMPORTANT", HOW WOULD YOU ASSESS THE IMPORTANCE OF THE FOLLOWING FACTORS IN YOUR DECISION TO UNDERTAKE AN ENERGY AUDIT AT YOUR FACILITY? ................................................................49

TABLE 8.18: Q10. WHAT IS THE MAXIMUM PAYBACK PERIOD ACCEPTABLE TO YOUR COMPANY FOR THE IMPLEMENTATION OF ENERGY EFFICIENCY MEASURES? (IN YEARS).............................................................49

TABLE 8.19: Q11. DID YOU SELECT AN ENERGY AUDITOR FROM NRCAN'S ENERGY MANAGEMENT SERVICE DIRECTORY, WHICH IS AVAILABLE ON THEIR WEBSITE? ................................................................49

TABLE 8.20: Q12A-G. ON A SCALE FROM ONE TO FIVE, ONE BEING "UNSATISFACTORY" AND FIVE BEING "VERY SATISFACTORY", HOW WOULD YOU ASSESS THE FOLLOWING ASPECTS OF THE INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAM?...................................................................................................................50

TABLE 8.21: Q13 HOW DID YOU FIND OUT ABOUT THE PROGRAM? .............................................................50

TABLE 8.22: Q14. FINALLY, DO YOU HAVE ANY COMMENTS OR RECOMMENDATIONS ON HOW TO IMPROVE THIS PROGRAM?......................................................................................................................................51

TABLE 9.1: EXTRAPOLATION OF GROSS SAVINGS TO ADMISSIBLE PARTICIPANTS WHO DID NOT RESPOND TO THE SURVEY BASED ON PAYBACK RATIOS............................................................................54

TABLE 9.2: GROSS SAVINGS POTENTIAL IDENTIFIED BY AUDITS IN GJ/Y (N=502 VALID MEASURES 179 VALID AUDITS) ........................................................................................................................................55

TABLE 9.3: NET SAVINGS OF IMPLEMENTED MEASURES IN GJ/Y BY MEASURE TYPE, PAYBACK PERIOD AND INDUSTRY TYPE................................................................................................................................56

TABLE 9.4: INTENTION SAVINGS IN GJ/Y BY MEASURE TYPE, PAYBACK PERIOD AND INDUSTRY TYPE .........57

TABLE 9.5 TOTAL NET ENERGY IMPACT FROM IMPLEMENTED MEASURES AND INTENTION TO IMPLEMENT MEASURES..............................................................................................................................................57

TABLE 9.6: IN SAVINGS RATIOS, PERCENTAGE OF ENERGY CONSUMPTION (GJ/Y).....................................58

TABLE 9.7 IN SAVINGS RATIOS, PERCENTAGE OF ENERGY CONSUMPTION ($) ...........................................58

TABLE 9.8: SAVINGS RATIOS FOR IMPLEMENTED MEASURES AND INTENTIONS............................................59

TABLE 9.9: MEASURES IMPLEMENTATION RATIO .......................................................................................59


ADEC-Zariffa Consortium viii

TABLE 10.1: GHG REDUCTION TONNES EQUIVALENT CO2 BASED ON NET SAVINGS FOR 179 AUDITS ........60

TABLE 10.2: GHG REDUCTION TONNES EQUIVALENT CO2 BASED ON NET SAVINGS FOR 179 AUDITS ........61

TABLE 10.3: TOTAL GREENHOUSE GAS REDUCTION .................................................................................61

TABLE B.1: RESPONSE RATE CALCULATION FOR AUDITORS ......................................................................72

TABLE C.1: RESPONSE RATE CALCULATION FOR NON-PARTICIPANTS........................................................73

TABLE D.1: RESPONSE RATE CALCULATION FOR PARTICIPANTS................................................................74


ADEC-Zariffa Consortium ix

ACKNOWLEDGMENTS{ TC "ACKNOWLEDGMENTS" \F C \L "1" }

The ADEC-Zariffa Consortium wishes to acknowledge and express its gratitude and appreciation to Ms. Melanie Phillips, Chief in the Industrial Program Division of the Office of Energy Efficiency, Natural Resources Canada (NRCan), as well as to her team for their precious collaboration and time in providing all necessary information to carry out this impact evaluation study.


ADEC-Zariffa Consortium x

EXECUTIVE SUMMARY{ TC "EXECUTIVE SUMMARY" \F C \L "1" }

The Industrial Energy Audit Incentive Program is sponsored by NRCan’s Office of Energy Efficiency and offers Industrial Energy Innovators the opportunity to share the cost of undertaking an energy audit of their facilities. NRCan’s contribution represents 50% of the audit cost, up to a maximum of $5,000.

The program was launched in fiscal year 2001/2002 for a period of 5 years. The program objective is to fund some 350 to 500 audits by the end of fiscal year 2005/2006. At the time of the present evaluation, 216 audits reports had been completed and hence constituted the population to be evaluated.

This evaluation has two dimensions. The first is an impact evaluation to estimate the net energy savings reduction of greenhouse gases related to the implementation of measures identified in the energy audits. The second is a process evaluation of the program.

The methodology used by the ADEC-Zariffa Consortium for the energy and environmental net impact evaluations consisted of surveys of three groups: participants, non-participants and the consultants who did the energy audits. For the process evaluation, the methodology involved a series of appraisals of all relevant documents related to program theory, marketing, support tools, implementation and tracking system and administrative procedures.

The distortion effects considered in the calculation of net energy savings in this study are free-ridership (using a sophisticated methodology that includes time as a factor), cost and influence factors related to other NRCan programs and spillover to other facilities.

Net energy savings were calculated for both implemented measures and intentions to implement. The calculation of extrapolations to participants who could not be reached are based on the implementation-to-payback ratio of participants who were interviewed.

GHG emission reductions are calculated by energy type and province, since the role of electricity generation differs from one province or territory to the next.

The most important study finding is related to the quality of the audit in terms of both content and reporting, which has an impact on the process involved in gleaning information from the program tracking database.


ADEC-Zariffa Consortium xi

With respect to content, a panel of experts appraised nine randomly selected audits, giving them an average score of 55%. However, from the participants’ perspective, audit quality does not seem to be an issue.

As for reporting, the difficulty of retracing some of the essential data in the audit reports limited the amount of valid data concerning energy efficiency measures to be addressed in the participant surveys, resulting in underestimated energy and GHG impacts.

Of the 216 audits available for evaluation, 37 were eliminated, for a remaining 179 or 82.8%. The remaining 179 audits include 29 valid audits classified as “don’t contact” due to inaccurate contact information. These 29 audits were nevertheless included in the impact extrapolation calculations. The 179 audits represent 842 measures, but only 502 measures were valid for impact calculations. This particular problem can easily be corrected in the future by introducing a mandatory audit summary form to be filed by the consultants and included in the audit reports.

In terms of program appraisal by all parties, including the evaluator, participants, non-participants and consultants, the results indicate a high level of satisfaction, with proposals for only minor modifications.

In terms of energy impact, the study estimates gross energy savings of 6,995,055 GJ/Y for a net savings of 2,655,561 GJ/Y and a net-to-gross ratio of 37.9%. The net savings are comprised of a solid 2,197,418 GJ/Y from implemented measures and a less solid 458,143 GJ/Y from intentions to implement within the next 12 months. The precision factor for the savings is ± 9.3%.

In terms of proportions, gross savings represent 2.85% of total energy consumption. The trimmed average raises this percentage to almost 7% by eliminating a few outliers with very large energy consumptions. This 7% savings potential is quite normal since it is related to total energy consumption and not to the specific type of energy saved by a particular measure. In other words, annual energy consumption is not a real base line.

Free-ridership is low, due mainly to the influence of other NRCan activities that reduce the level of free-ridership. The spillover effect is relatively high, especially since participants can apply for another audit for a different facility, giving a positive spin to the program.

In terms of GHG reduction, the study estimates the savings at 131,000 tonnes of CO2

equivalent per year. It is unfortunate that the database did not include the duration of the


ADEC-Zariffa Consortium xii

recommended measures for calculating both energy and GHG cumulative effects. As a result, we were unable to compare the present impact evaluation to the program objective of 500,000 tonnes of CO2 equivalent per year, by the year 2010.

If, by the end of the program, the higher target level of 500 audits is reached and we apply the same correction factor of 179/216 and the same mix of savings by type of energy and province, we can anticipate a GHG reduction of at least 300,000 tonnes of CO2 equivalent.

Participants saved a minimum of $15,345,000 in one year from net implemented measures. Some participants compound these savings through measures that have no impact on energy consumption, such as water savings or correction of power factors but generated financial savings. The program’s cumulative expenditures for the fiscal years 2001/2002 to 2004/2005 were $3,800,000 for 339 audits, indicating a very comfortable payback period if one uses a minimum and conservative 5-year life span for these measures.

Finally, in terms of recommendations, the ADEC-Zariffa Consortium recommends a series of simple measures to improve program performance and help undertake future evaluations.

The following table presents the general features of the program and the main results of the analysis.


ADEC-Zariffa Consortium xiii

Table 1.1: General features of the program and main results of the analysis

Main features

Program objectives (2001/02 – 2005/06) 350 to 500 audits

Audit evaluated 216 audits

Program cumulative expenditures (end of January 2005) $3,796,000

Program total budget $5,000,000

GHG reduction target to 2010 (tonnes) 500,000

Main results for 179 audits

Gross energy savings GJ/Y 6,995,055

Net energy savings GJ/Y

Solid GJ/Y :Intentions GJ/Y:

2,655,561

2,197,418 458,143

Gross savings to total energy consumption 2,85%

GHG reductions achievement in tonnes/year 131,102

GHG reduction potential in tonnes/year 300,000

Total energy bill $727,124,300

Gross savings potential /year $26,574,600

Net savings implemented measures /year $15,345,000

Trimmed average consumption GJ/Y 46,297

Trimmed average gross savings potential GJ/Y 3,219

Trimmed average net implemented savings GJ/Y 938

Percentage of gross potential savings 7.0 %

Percentage of net implemented savings 2.1 %

Ratio net to gross 29.1 %


ADEC-Zariffa Consortium 1

1. PROGRAM DESCRIPTION

The Canadian Industry Program for Energy Conservation (CIPEC) is a unique, voluntary alliance between Canadian industry and the federal government through Natural Resources Canada's Office of Energy Efficiency.

For more than 25 years, CIPEC has been helping Canadian industry boost its bottom line by using energy more efficiently. It offers a number of services to help each of Canada's industrial sectors develop energy efficiency goals and action plans.

Industrial Energy Innovators (IEI) is a voluntary, company-based initiative that turns CIPEC sector commitments into actions at the company level. The goal of the IEI is to help Canadian industry stay competitive and to limit greenhouse gas emissions by improving energy efficiency. Industrial Energy Innovators gain access to a variety of products and services from Natural Resources Canada (NRCan) that are designed to help them become more energy-efficient. Participants file annual action plans and progress reports that outline their energy efficiency initiatives and achievements.

To become an Industrial Energy Innovator, a company's President and/or Chief Executive Officer submits a letter of commitment to the Chair of CIPEC's Executive Board agreeing to:

• Develop and implement an energy efficiency improvement target and action plan;

• Nominate an energy efficiency champion;

• Track and report the results of their energy efficiency activities annually.

Many products and services are available to help Industrial Energy Innovators achieve their energy efficiency objectives. Innovators are regularly updated on new technologies, operating practices, an international technical information network and the latest Dollars to $ense series of three energy efficiency workshops.

Recruiting for the IEI began in mid-1995 through CIPEC and its extensive industry network. By early 2004, more than 500 companies had become part of this initiative. Recruitment continues and participants from all industrial sectors are becoming involved.



Industrial Energy Innovators have access to a variety of energy efficiency products and services developed by Natural Resources Canada's Office of Energy Efficiency (OEE). The Industrial Energy Audit Incentive Program (IEAIP) is among these products.

The Industrial Energy Audit Incentive Program is designed to help defray the cost of hiring a professional energy auditor to conduct an on-site audit at an industrial facility. Financial support is available exclusively to industrial companies in Canada that are registered as Industrial Energy Innovators. The audit incentive, administered by NRCan's Office of Energy Efficiency, covers up to 50% of the cost of an energy audit, to a maximum of $5,000, to help industrial companies identify ways to increase energy efficiency, improve production processes and cut costs.

Companies that receive a Natural Resources Canada (NRCan) audit incentive may hire the professional contractor of their choice to conduct the energy audit. Typically, this will be an engineering consulting firm with experience in conducting industrial energy audits. NRCan has an Energy Management Services Directory available on its Web site to help companies choose a contractor.

Companies must submit a short technical proposal along with a completed application form when applying for an incentive. The technical proposal must be prepared by the contractor who was selected to conduct the audit.

Optional guidelines for preparing a technical proposal, which describes the scope of the energy audit, deliverables, cost and schedule, are also available. After the application is approved and the audit is completed, companies are reimbursed by NRCan for a portion of the audit cost.

By using energy more efficiently in the extraction, refining and delivery of natural resources and in the manufacture of products, industry can become more competitive and help reduce greenhouse gases (GHG) that contribute to climate change.

Companies are encouraged to work closely with their contractor throughout the audit process because a successful audit depends on the contractor having a thorough knowledge of the company’s energy systems, utility costs and energy management practices.

If a company has facilities at more than one location and is interested in conducting separate audits at these facilities, it may apply for a separate incentive for each facility. Also, if a



utility agrees to co-fund the audit, NRCan will contribute only as much as the company, to a maximum of $5,000.

The administrative procedure for participating in the program is as follows:

• Arrange for a professional energy audit contractor to develop a technical proposal for the audit.

• Complete NRCan’s Industrial Energy Audit Incentive Application and Contribution Agreement form, available on-line or in print copy.

• Submit the application form and the technical proposal for the audit to NRCan. NRCan will review your application and, if it is acceptable, establish a contribution agreement. You will be notified when your application is approved.

• Arrange for the contractor to perform the energy audit.

• Pay the contractor for the energy audit when the work has been completed to the client’s satisfaction.

• Submit a claim for partial reimbursement of the audit to NRCan, along with a completed copy of the Industrial Energy Audit Incentive Claim form supplied by NRCan. Participants must submit a copy of the paid invoice and the energy audit report. NRCan will send the participant a check covering up to 50 percent of the invoiced audit cost, to a maximum of $5,000.

1.1. PROGRAM HISTORY

The program was launched in 2001 for a period of 5 years ending in 2006. The initial objective was to perform 350 to 500 audits during that period. The expected GHG emissions avoided were estimated at 0.5 Mt CO2 equivalent per year by the year 2010.

The initial program budget authorized by the Treasury Board was set at $5 million. The following tables present a breakdown of the budget and expenditures.



Table 1.1: Program Budget $ K

Expenses 2001/02 2002/03 2003/04 2004/05 2005/06 Total

Salaries 70 120 125 130 135 580

Benefits 14 24 25 26 27 116

Operation & Maintenance 291 581 900 894 888 3,554

Grants & Contributions 100 150 150 150 150 700

Corporate Communication 25 25 0 0 0 50

Total 500 900 1,200 1,200 1,200 5,000

Table 1.2:

Program Expenditures $ K

Fiscal Year Planned Incurred Percentage

2001/02 500 575 115

2002/03 900 921 102

2003/04 1,200 1,100 91

2004/05 1,200 1,200* 100

Total 3,796,000

* End of January 2005.

The following tables present the program participant history.

Table 1.3:

Participants – History by Fiscal Year

Fiscal year Participants

2001-2002 12

2002-2003 76

2003-2004 142

2004 – February 2005* 109

Total = 339

* For the complete fiscal year, the number of participants was 141.



Table 1.4: Participants – History by Industrial Sector

Industry type Participants %

Aluminum 6 2

Brewery 3 1

Cement 4 1

Chemicals 7 2

Construction 2 1

Electricity 8 2

Fertilizer 4 1

Food & Beverage 76 22

Forest 1 0

Foundry 9 3

General Manufacturer 76 22

Mining 2 1

Petroleum Products 3 1

Plastics 15 4

Pulp & Paper 25 7

Rubber 5 1

Steel 3 1

Textiles 16 5

Transportation 35 10

Upstream 28 8

Wood Products 11 3

Total 339 100



Table 1.5: Participants History by Province

Province Participants %

Newfoundland 1 0

Prince Edward Island 2 1

Nova Scotia 5 1

New Brunswick 4 1

Quebec 104 31

Ontario 167 49

Saskatchewan 10 3

Manitoba 4 1

Alberta 32 9

British Columbia 10 3

Total 339 100

1.2. FINDINGS

From tables 1.1 to 1.5 we can conclude that:

Based on the program budget breakdown, it is difficult to assess the actual share of the grants and contributions since it seems that funds from operation and management can be transferred to grants and contributions. Without this transfer, it appears that grants and contributions make up just 14% of the total budget, which is totally disproportionate.

We note that, after starting in a natural and progressive manner, the number of participants levels off in the fourth year of the program.

If we exclude the general manufacturing category, the food & beverage industry has the largest number of participants, followed by transportation and pulp & paper.

The provinces of Ontario and Quebec have the largest number of participants, probably because of the outsourcing of marketing activities to OCETA in Ontario and CETC in Quebec. These provinces are also Canada’s two largest industrialized provinces.

Expenditures incurred in relation to the program appear to be on target when compared to planned expenditures.



2. SCOPE OF STUDY

This study is in response to NRCan’s request for proposal (RFP) soliciting an impact and process evaluation of the Industrial Energy Audit Incentive Program.

Impact evaluation refers to the assessment of net energy and environmental impacts (expressed in terms of avoided GHG emissions) directly resulting from the implementation of the IEAI Program. Net impact refers to the impact related to a strategic intervention, after taking into account the various distortion effects applicable to the program and, if possible, natural savings. In this context, the causal effect has to be established with a reasonable confidence level.

The process evaluation examines aspects such as program theory, administrative procedures, program documentation and implementation tracking activities. Process evaluation usually sheds light on or explains certain discrepancies in impact evaluation results when compared to program objectives.

Process evaluation is also recognized as the most efficient tool for improving program performance. In other words, the two components of the RFP are very closely related and we encourage NRCan to include process evaluation in their other program evaluations as well.

Process evaluation also provides a vehicle for sharing program design and operational improvements with others in order to enhance the effects or cost effectiveness of other programs.



3. IMPACT EVALUATION METHODOLOGY

The impact evaluation assessing net energy savings and avoided GHG emissions was conducted through surveys of three parties: participants, non-participants and professional energy auditors.

The methodology proposed by the Consortium was initially based on the premise that the program implementation tracking system would provide the ADEC-Zariffa Consortium with a database containing the following key information, needed for building the foundation of the evaluation.

3.1. DATA TRACKING SYSTEM

3.1.1. PARTICIPANT DATA

• Contact information for each participant, including the contact person’s name and phone number.

• Industry type.

• Name and contact information of the professional energy auditor who conducted the audit.

For each of the 216 completed audits, a short description of each energy measure identified in each audit along with:

• The participant’s total annual energy consumption with the proper energy units.

• The estimated savings per measure in absolute quantities.

• The source of energy saved per measure.

• The duration of the savings related to each measure.

• The cost of each measure.



• The cost effectiveness of each measure in terms of payback or any other cost effectiveness test performed by the auditor (preferably payback for consistency purposes).

• The audit’s completion date.

• The existence of another financial incentive to cover the audit cost.

• The total cost of the audit.

After an examination of the database, it appeared that some information was not available, either because the database fields did not include specific data (i.e., the duration of each measure used to calculate cumulative impacts), or because the data was missing from the audit report and hence could not be retrieved. The impact of missing data is discussed in the impact section of this report.

3.1.2. NON-PARTICIPANT DATA

NRCan also provided a list of Industrial Energy Innovators that did not participate in the IEAIP, along with contact person information.

3.1.3. AUDITOR DATA

The professional energy auditor identification was included in the participant database.

3.2. PARTICIPANT METHODOLOGY

The Consortium developed three questionnaires that were approved by NRCan authorities before the pre-tests: one for participants, one for the professional auditors and one for non-participants. Some questions in the participant survey were inspired by the results of a mid-term survey of 19 program participants performed by the Ontario Centre for Environment Technology Advancement (OCETA) in October 2004.

To ensure a high level of participation, the Consortium also provided the program authorities (PA) with examples of the survey notification letters sent to the three parties to explain the purpose of the survey. Past experience has demonstrated that this technique fosters good results. The letters were e-mailed to all parties a few days prior to the survey’s starting date.



After an examination of the database, a total of 216 participants with completed audits were identified as eligible to be surveyed, prior to the application of screening filters to identify the measures to be included in the survey.

Since the Consortium and the PA did not expect all participants to answer the survey, some kind of extrapolation to all participants was necessary.

The extrapolation methodology discussed with the PA was based on the payback ratio for implemented measures. In the extrapolation process, we paid special attention to geographical location in order to take into consideration the appropriate GHG conversion factors, especially for electricity generation.

The energy efficiency measures identified for each audit and other information provided by the NRCan database were embedded in the questionnaires so that the questions were related to specific measures for each participant and then translated into annual energy gross and net impact by energy type and GHG avoided emissions.

The questions were related to several aspects of the program, including implementation status, program awareness, participant satisfaction, incentive level, access to other types of incentives for the implementation of measures or the covering of another part of the audit cost (local governments or public utilities), free-ridership and spillover effect. The usefulness of the contractor database was also questioned.

The spillover effect related to specific measures that had been examined through the funded audit and could be implemented in a different facility owned by the participant without the need for an additional audit.

In evaluating the free-ridership and spillover effect, the Consortium applied a methodology based on the Standardized Methods for Free-Ridership and Spillover Evaluation of 5 American electric utilities.1 This methodology has been used before by at least one Canadian public utility. It was developed mainly for programs that fund specific measures but can also be used for funded feasibility studies or audits. This methodology takes into consideration timing, cost, quantity and impact of past participation in other NRCan programs, as well as the level of energy efficiency measure. Nevertheless, since this program does not subsidize

1 National Grid, NSTAR Electric, Notheast Utilities, Unitil, Cape Light Compact: Standardized Methods for Free-Ridership

and Spillover Evaluation – Task 5 final report, June 2003. Pamela Rathbun, Carol Sabo and Bryan Zent.



implementation of individual measures, the level of energy efficiency was not taken into consideration.

With this methodology, free-ridership was established for each participant on a scale from 0% to 100%. The calculated ratio was then applied to the savings associated with each measure to determine the net savings generated by each measure.

The net savings formula for each recommended measure is:

NS= (NMS-FR+SO)

Where

NS = Net savings.

NMS = Net measure savings. NMS is calculated by multiplying the gross savings of the measure as established in the audit and adjusted by the complete or partial implementation factor (based on the % of savings as established by participant compared to audit estimate).

SO= Spillover effect. This effect adds savings to the measure.

Spillover effect was calculated using the following formula:

SO= MGS*NbINS*%S*%FR

Where:

MGS refers to the measure of gross savings as established in the audit and applied to another installation without an audit.

NbINS is the number of installations in which this measure was implemented.

%S is the percentage of savings related to the measure and resulting from total or partial implementation or a different base line (as reported by participants).

%FR is the global free-ridership ratio calculated in the facility where the audit was funded (we assume the same behaviour)

FR: The free-ridership calculation starts with a key question about timing. If the participant declares that he intended to undertake such an audit within 12 months of the funded audit completion date, he is considered a free-rider at a level establish by the following formula:

FR= NMS*C*IF.



Where

C = Cost aspect expressed in terms of % cost the participant was willing to pay if he had not received external funding compared to the full audit cost.

For example, if the participant answered that he would have been willing to pay 75% of the audit cost, the C factor would be 75%.

IF= Influence factor due to past participation in NRCan programs or activities.

The influence factor is based on the number of affirmative answers regarding client behaviour subsequent to participation in previous NRCan programs or activities.

• For 3 “yes” answers, we apply an IF factor of .25

• For 2 “yes” answers, we apply an IF factor of .6625

• For 1 “yes” or none, we apply an IF factor of 1

The FR ratio is applied to intentions and spillover effects.

The initial proposed formula was FR= NMS*C*Q*IF

Where Q was the quantity aspect expressed in % of number of measures that would have been studied, translated in terms of % savings on total savings. The cost factor (C) and the quantity factor are usually used for programs that offer funding for the implementation of a particular measure and not for an audit. Even though we included a question related to Q in this study, results were not conclusive since there was a perfect correlation between scope of study and cost. Hence C and Q were the same.

Free-ridership is a negative distortion effect, while spillover is a positive distortion effect.

The questionnaire for participants took an average of 9 minutes to complete and included 60 variables.

The questions were designed bearing in mind the program’s ultimate goals, stated as follows:

“The goal of NRCan’s audit incentive is to ensure that companies have access to affordable customized audits, so that they can make sound business decisions about how to become energy efficient and reduce GHG emissions at their facilities.”

Also, the audits are expected to benefit the industry by:

• Finding out how to reduce energy costs by a minimum of 5 to 15 percent, often with payback periods of less than two years;



• Establishing a baseline against which to compare future improvements;

• Helping kick-start a company energy management strategy.

NRCan’s program manager must bear in mind that surveys are all subject to self-reporting limitations. Nonetheless, the Consortium had no reason to believe that Innovators would mislead the evaluator for the sake of appearances alone.

Energy savings predicted for each measure by the audit were used as a baseline to calculate the program net savings after applying the various adjustment factors identified in the survey (i.e., full or partial implementation, free-ridership or spillover effect).

SPSS software was used to convert implementation status into impacts. Energy savings were calculated on an annual basis only and not cumulatively since the lifespan of each measure was not available in the database. Intentions were calculated separately since they carry a high level of uncertainty.

Savings were established by measure and payback, as well as by a set of 5 industries. In converting savings into avoided GHG emissions, savings were also linked to province of origin, since electricity generation is different from one province to the other. Lastly, savings were also determined in terms of dollars.

To convert energy savings into GHG, the Consortium used the CO2 conversion factors calculator provided by NRCan.

3.3. NON-PARTICIPANT METHODOLOGY

The Industrial Energy Innovators database contained 174 companies with valid contact information, of a total of 198 companies. The objective of the survey was to complete 50 questionnaires. A short questionnaire of 9 minutes was designed to identify reasons for non-participation in the program. Reasons could be circumstantial or related to aspects of the program itself, to be brought to the attention of the PA manager for adjustment and program improvement purposes.



3.4. AUDITOR METHODOLOGY

From the master auditor database containing 145 names (including multiple listings), we selected 124 auditors who had performed audits for participants and conducted 40 interviews. A short 6-minute questionnaire was designed to establish the auditors’ level of satisfaction with the program’s components and its impact on their business.

Certain questions were also designed to cross-validate some of the participants’ answers, regarding, for example, their intention to undertake the audit without the program.



4. PROCESS EVALUATION METHODOLOGY

The process evaluation took on the challenge of evaluating most, if not all aspects associated with four elements of the program:

1. Program theory

2. Marketing aspects

3. Implementation aspects

4. Tracking aspects

To perform this task, the evaluator interviewed key personnel responsible for these 4 elements. The evaluator also examined 9 randomly selected participant files, including the energy audit reports that had been examined by a panel of energy audit experts. Paper and electronic trails related to program implementation and tracking were also examined.

Lastly, the evaluator established a consistency factor between the PA’s process description and the participants’ assessment of the process, based on the survey questionnaires, in order to obtain both sides of the story.

In respect to program theory, the document submitted to the treasury board to justify the program and request funding authorization was examined and discussed with the program manager. The evaluator looked at the rationale behind the program as well as its justification, its quantitative objectives and the manner in which they were established.

In respect to program marketing, paper and electronic program documentation, marketing techniques and activities, as well as the marketing partner’s identification and roles were examined and discussed with the PA.

In respect to program implementation, the full administrative procedure involved in processing a request to participate was examined and discussed with the PA to determine effectiveness, response time, complaints, feed-back, etc.

In respect to program tracking activities, the data collection, validation and entry processes were thoroughly examined, since they are the basis for any impact evaluation calculation. The evaluator examined the effectiveness, uniformity and usefulness of the



tracking system. In order to assess the adequacy of the tracking system, the evaluator examined a number of participant audits and compared them with the data recorded in the database.


ADEC-Zariffa Consortium

Figure 1. Methodology Logic Model

Review program data and program theory. Collect available data and program information.

Survey pre-test for participants

1. Assess program theory with program manager

2. Interview program staff

3. Examine program documentation

4. Examine randomly selected participant files and audits

5. Examine database

1. Survey data analyses 2. Determination of energy savings (gross, net,

yearly, cumulative, extrapolation) 3. Environmental impact 4. Process evaluation 5. Report and recommendations

Process evaluation issues

• Participant survey • Non-participants survey

Participant respondents

Non-participant respondents

Presentation of findings

Design questionnaires 1. Participants 2. Professional energy

auditor 3. Non-participants

Draw sample for non-participants

Draw sample for professional energy auditor from participant respondents
Survey professionalenergy auditor
17

Professional auditor respondents


ADEC-Zariffa Consortium

Figure 2. Evaluation Team

Project Administrator

Gilles Joubert

Project Manager and Head Evaluator Sohel Zariffa

Statistician and Head Surveyor

Serge Marcil

Call Center Géocom Communication

Guy Mercier

Audit Advisory CommitteeAndré Chalifour

Yves Blanc

18

Réal Olivier



5. PROCESS EVALUATION RESULTS

5.1. PROGRAM THEORY

Technically speaking, an examination of program theory focuses on the premises or hypotheses used to justify the program, its components and its objectives. These hypotheses are then validated with participant profiles or data collected during implementation in order to assess the soundness and rigorousness of the design. If a program is relatively new in age or content, it is not unusual to adjust program theory in order to improve forecasting. Programs that don’t need major adjustments are those with a long participant history allowing for better forecasts and, accordingly, fewer in-course adjustments.

Program hypotheses and objectives are often drawn from prior market research, published literature or simply the benchmarks provided by the performance records of similar programs.

The document used to study the program theory was a request to the Treasury Board, dated October 9, 2001, for the funding of a series of six measures, identified as “Industry Cross-cutting Measures”.

5.1.1. PROGRAM JUSTIFICATION AND DESIGN

The program is one of six industry cross-cutting measures identified in the Memorandum to Cabinet, Climate Change First Federal Business Plan and Strategy. Hence, it is part of the Federal Government action plan to reduce GHG as part of Canada’s commitment to the Kyoto Protocol.

The program was conceived and developed in cooperation with Canadian industry through consultation with different forums such as the Climate Change Industry Issue Table, the CIPEC Executive Board and Task Forces, and the National Advisory Council on Energy Efficiency of the Office of Energy Efficiency. A special consultation session involving representatives from more than 15 sectors was held in January 2001.



Feedback from the consultation sessions concerning the audit measure was also positive. Suggestions were raised about the funding ratio (i.e., 50/50 versus some other split), the synergy between the audit activity and the benchmarking program and the need for flexibility for delivery through either a third party regional agency or directly through trade associations. Audit take-up and follow-up action by companies was also of interest. There were differing opinions on “uptake”.

Our conclusion on the issue of justification and design is that the program has a very solid base that should ensure its success.

5.1.2. PROGRAM OBJECTIVES

The overarching objective of the six measures is to encourage Canadian industry to become more energy efficient and, in so doing, reduce greenhouse gas (GHG) emissions by 5.8 Mt per year in 2010. The share of the IEAIP in the GHG emissions reduction was established at 0.5 Mt by 2010. To achieve this goal, the program expects to fund 350 to 500 audits within its 5-year lifespan.

The specific objective of the IEAIP was to:

Provide support for companies to audit their facilities in order to identify potential energy efficiency measures and encourage them to implement these measures.

It is not clear from the Treasury Board request how the quantitative objective of 0.5 Mt per year was calculated. According to program authorities, the objective was derived from similar programs that had been examined, but no documentation is available on the matter. If the 0.5 Mt is somehow related to what the program brochure indicates as possible savings from the 350 to 500 audits in the magnitude from 5 to 15% with a payback period of less than 2 years, we believe that the 0.5 Mt objective seams reasonable.

5.1.3. EVALUATION ISSUES

Although the TB document does not include an evaluation plan for the program as such, it does establish the following performance indicators.



Table 5.1:

Program Performance Indicators

Activities Outputs Indicator Data Sources Targets

Short-term Long-term

Company audit reports on energy efficiency opportunities

Number of establishments audited Program files 50 audits by 2002 350 to 500 audits by 2006

National database of audit results

Comprehensiveness of database (i.e., number of variables monitored; number of audited companies included)

Audit reports and program files

Establish a database to house audit results by 2002

Maintain and use database to evaluate extent to which companies implement audit recommendations and analyze the energy and emissions potential of industrial establishments

Energy Efficiency Audits:

Manage delivery of audit service, including the negotiation of funding arrangements with partners

Establish and implement audit funding process

Establish audit database

Promote audit initiative, including publicizing audit results

Case studies Number of case studies Program files Case studies of 2% of establishments audited by 2002

Case studies of 2% of establishments audited by 2006



After analyzing different aspects of the program process, we believe that the performance indicators mentioned in Table 5.1 are available for program monitoring. The one aspect we would recommend is the design of an evaluation plan incorporated in the program theory to avoid any unpleasant surprises during evaluation. In this particular program, we are referring to a database quality control problem we encountered during examination. We will discuss this subject in the tracking section of the process evaluation.

5.1.4. STAFFING

NRCan’s staffing of the program seems adequate (about 3 person/years). With outsourcing resources such as OCETA and CETC, it is somewhat difficult to track total resources involved in managing the program. Outsourcing is very effective for this type of program and we do not recommend a change of strategy. However, we do recommend that the total value of staffing be determined in order to identify the relative importance of efforts (program expenses) and deliverables (grants and contributions).

5.2. PROGRAM MARKETING

Paper and electronic program documentation, marketing techniques and activities and the identification and role of marketing partners were discussed with the program authorities. Extensive bilingual documentation exists to promote this program and resources such as workshops, information sessions, newsletters and case studies.

Three organizations are involved in marketing the program in three different regions: the Ontario Centre for Environmental Technology Advancement (OCETA) in Ontario, the Energy Technology Centre (CETC) in Quebec and the Petroleum Technology Alliance of Canada (PTAC) in Alberta. In Quebec and Alberta, both CTEC and PTAC are involved in processing the various participant forms and examining the audits.

Except for the reference to 5 to 15% savings with a payback of often less than 2 years, which we feel is a bit high and could foster unrealistic expectations on the part of participants, we believe the marketing aspects of the program are adequate and well orchestrated.



5.3. PROGRAM IMPLEMENTATION

The full administrative procedure, from the participants’ application at the outset to the final claim and disbursement process, was examined and discussed with the PA to identify aspects such as effectiveness, delays, complaints and feedback.

All program forms and checklists were examined, as were the application processing and response time. The time required to process applications and issue a response to participants is 10 working days, which seems entirely adequate, while the time required to process a claim and send the contribution to the participant seems a bit high at 45 working days.

Some of the administrative documents examined are listed below:

• Industrial Energy Audit Incentive Application and Contribution Agreement Review Checklist

• Industrial Energy Audit Incentive Application and Contribution Agreement

• Technical Review Checklist

• Industrial Energy Audit Incentive Claim

• Industrial Energy Audit Incentive Claim Revue Checklist

• Search for a Contractor tool

• Guidelines for the Preparation of Energy Audit Reports

• Industrial Energy Audit Incentive Program Frequently Asked Questions

• Canadian Industry Program for Energy Conservation portfolio

• Sample Summary Tables for the Executive Summary

Also, program form design dates and revision dates should be printed on each form to avoid confusion. It should, however, be kept in mind that too many in-course modifications could affect database consistency and complicate analysis.

5.4. PROGRAM TRACKING

The process of data collection, validation and entry was thoroughly examined, since it provides the basis for any impact evaluation calculation. The evaluator looked at the



effectiveness, uniformity and usefulness of the tracking system. To do so, the evaluator randomly selected 9 audits with their respective administrative files and examined the database recording process.

The findings of this examination indicate that the gleaning process is somewhat deficient. The level of missing or inadequate data is relatively high, as reflected by an examination of the complete database. Indeed, an initial examination of the database revealed a 38% level of inadequate data (according to program authorities).

We have traced the problem to two sources:

The first is the quality of some audits, in which basic and important information is simply missing.

The second is the gleaning process, in which information may not be clearly available in the audit and a little research or calculation is required to find the missing data. In other cases, information was available but not recorded, for reasons that are not known.

Before using the database for the survey, program authorities reviewed it to reduce the level of inadequate data, insofar as it was not related to the first reason. This reduced the inadequate data ratio to 33.8%, or 285 inadequate measures of a total of 842 measures for 216 audits, indicating that the second reason is far less important than the first (audit quality).

In order to resolve such problems, we recommend that the auditors fill out a mandatory form summarizing the audit findings and attach it to their audits. An example of such a form is presented in appendix A. Use of the form would not only solve this problem, but also introduce a level of uniformity to the data collection process, improving process efficiency. Such a form could also be more effective for use in future evaluations than a database that includes transcription errors or interpretations.

Our examination of the various fields in the database showed that only the duration of each measure is missing. Without information on duration, one cannot establish cumulative impacts and undertake retention studies in the future. This information is included in the proposed energy audit summary form.



5.5. AUDIT EVALUATION

The consortium advisory committee examined nine randomly selected audit reports and employed an evaluation matrix used to evaluate other audits. Each member examined the nine reports and scored them according to 6 criteria and 13 sub-criteria. Notes and observations regarding each audit were transmitted to the program manager on a confidential basis.

The evaluation criteria and sub-criteria, their weight and maximum score are presented in Table 5.2.

Table 5.2:

Audit Report Evaluation Criteria

Evaluation Criteria Weight % Maximum Score

Conformity with the program guidelines 10%

Degree of compliance with the study guidelines 5

Comprehensiveness of the study* 20%

Presentation of global energy picture is included 5

Main energy consuming systems are analyzed 5

Recommended improvements are significant on energy costs 5

Degree of innovation 10%

Innovative solutions have been proposed 5

Clarity of report 10%

Clear description of the existing situation 5

Clear description of improvements 5

Clear economic evaluations 5

Reliability 25%

Reliability of the technical solutions 5

Reliability of investment estimates 5

Reliability of savings calculations 5

Value, i.e., cost to benefit ratio 25%

Usefulness of the study for the client 5

Potential savings justify the cost of study 5

Total 100%

* Comprehensiveness of the study is not to be expected from all audits since the level of funding doesn’t allow a full-scope audit in some large installations and not all clients want a full-scope study. As a result, the scoring for this particular criterion has to be carefully interpreted.



Table 5.3 presents the consolidated results of the advisory committee’s review, excluding participant identification for confidentiality purposes. A final score of 55% indicates room for improvement from the point of view of industrial energy experts, providing some cross-validation of the inadequate level of data in the database. The evaluations submitted by the members of the advisory committee are presented in appendix E, F and G.

To improve audit report quality, we recommend training in the form of a workshop for both auditors and participants, in order to establish a strong link between what participants should expect and what auditors should deliver. Pre-qualification of auditors by industry experts could be another way to enhance audit quality.



Table 5.3: Audit Evaluation Compound Results

Expert committee audit evaluations: Industrial Energy Audit Incentive Program - NRCanBased on a review of 9 randomly selected audits

1 2 3 4 5 6 7 8 910%

5 4 4 4 3 1 2 2 4 2 2,920%

5 5 5 5 5 1 1 2 3 2 3,05 3 4 4 4 1 2 2 4 2 2,95 4 2 4 3 3 2 1 3 3 2,7

10%5 1 2 3 2 1 2 2 2 1 1,7

10%5 4 5 5 4 3 2 1 4 2 3,45 3 4 4 4 4 2 2 4 2 3,15 3 4 3 2 3 2 1 3 1 2,6

25%5 4 4 4 3 4 2 2 4 3 3,35 3 4 3 2 4 2 2 3 0 2,55 3 4 3 2 3 3 2 3 1 2,6

25%5 3 4 4 3 4 2 1 4 1 3,05 2 2 3 2 3 2 2 3 3 2,5

100% 63% 69% 72% 56% 53% 42% 32% 66% 37% 55%

Potential savings found justify the cost of the study

TOTAL (%)

Reliability of investment estimatesReliability of savings calculations

Value i.e. cost to benefit ratioUsefulness of the study for the client

Clear description of improvementsClear economic evaluations

ReliabilityReliability of technical solutions

Degree of innovationInnovative solutions have been proposed

Clarity of the reportClear description of the existing situation

Comprehensiveness of the studyPresentation of global energy picture has been included

Main energy consuming sectors have been analysedommended improvements are significant on energy costs

Average

Conformity with the guidelinesDegree of compliance with the study guidelines

Evaluation criteria Weight Maximum score

Project no.



Of the 9 companies whose audits were evaluated, 5 were included in the completed surveys. The following table presents the scores participants gave when asked to rate the quality of the audit, with 5 being the highest score.

Table 5.4:

Audit Evaluation Vs. Audit Appreciation

Project number

Advisory Committee Score (%)

Participant score

1 63 5

2 69 3

3 72 4

8 66 1

9 37 5



6. AUDITOR SURVEY RESULTS

6.1. STATISTICS

The master auditor database contained 145 names. After eliminating auditors who had not performed any of the 216 audits and those with multiple listings (in several cases, auditors did more than one audit), the final population eligible for survey was 124 auditors. The survey completed 40 interviews. Appendix B presents additional statistical details concerning response rate calculations.

Our sample provides an adequate reflection of the provincial distribution of the population of auditors, even though the samples are very small for each province. For example, Ontario is represented by a number of contractors almost equal to the proportion of the population concerned (44% for the population and 45% for the sample). The auditor surveys lasted an average of 5 42’ minutes.

Table 6.1:

Distribution of Population and Respondents by Province

Province Population Respondent

Frequency % Frequency %

Alberta 17 14 4 10.0

British Columbia 5 4 0 0.0

Idaho* 1 1 0 0.0

Manitoba 3 2 0 0.0

New Brunswick 6 5 2 5.0

Nova Scotia 3 2 1 2.5

Ohio* 1 1 0 0.0

Ontario 51 41 18 45.0

Quebec 32 26 14 35.0

South Carolina* 1 1 0 0.0

Saskatchewan 3 2 1 2.5

Northern Territories 1 1 0 0.0

Total 124 100 40 100

*American auditors were also eligible to perform audits.



For a confidence level of 95%, the sampling error for a proportion varies from 9.4% to 13.2%.

6.2. AWARENESS AND FIRST CONTACT Table 6.2:

Q1. How did you learn about the Industrial Energy Audit Incentive program?

Frequency %

NRCan Website 14 33

Program brochure 3 7

Other participants 11 26

Contractor 3 7

NRCan personnel 4 9

Information session 4 9

Dollar to $ense workshop 1 2

Heads up CIPEC newsletter 2 5

Gaz Métropolitain 1 2

Total 43 100

The contractors found out about the program from eight sources, with a clear majority informed by two sources: the NRCan Website (33%) and other participants (26%). The high percentage recorded for the “other participants” variable is a very good indicator of program appreciation since word-of-mouth is considered the most credible form of referral or publicity.

Table 6.3:

Q02. In general, who initiated the first contact between your firm and the client for whom you have done the audit?

Frequency %

My firm 36 90

The client 4 10

Total participants 40 100

From the above table, we can conclude that the market push by firms is much stronger than the market pull from clients.



Table 6.4: Q02a-b. How did these clients find you?

Frequency %

Through NRCan's Energy Management Service Directory on NRCan 1 25

Reference from another client 2 50

Reference from one of the organizations involved in marketing: PTAC (Petroleum Technology Alliance Canada) 1 25

Total 4 100

The numbers recorded in this table are too small to be of any significance.

6.3. SATISFACTION

Satisfaction with various program elements, rated on a scale from 1 to 5 (one being "unsatisfactory" and 5 being "very satisfactory"), varies from 3.11 to 4.00. The element with the lowest score is the “time required to process and approve the applications”, while the element with the highest score is customer service (e.g., my questions are answered promptly and in sufficient detail).

Table 6.5:

Q03a-f. On a scale from one to five, one being "unsatisfactory" and 5 being "very satisfactory", how would you evaluate the Industrial Energy Audit Incentive in terms of the following?

Score

The application process and documentation required? 3.76

The time required to process and approve applications? 3.11

The level of incentive? 3.45

The program brochure and instructions? 3.92

Source of payment (i.e., the client, not NRCan, pays you)? 3.68

Customer service (e.g., my questions are answered promptly and in sufficient detail)? 4.00

6.4. PROGRAM IMPACT

For auditors, the principal impact of the program is an increase in their business for more energy audits (72%).



Table 6.6: Q04. Would you say that this program has …

Frequency %

Increased your business for more energy audits 29 72

Increased your business for more engineering work 9 23

Increased your business for the installation of new equipment 2 5

Total 40 100

The contractors are divided with respect to the impact of the incentive level on the scope of the energy audit (Table 6.7).

Table 6.7: Q05. Would you say that the incentive level limits you in the scope of the energy audit?

Response Frequency %

Yes 18 45

No 20 50

Don’t know 2 5

Total 40 100

6.5. PROGRAM COMPARISON Table 6.8:

Q06. Have you participated in a similar energy audit incentive program before (i.e., utility or provincial program)?


Yes 23 57

No 17 43

Total 40 100

More than half of the contractors (57%) have participated in a similar program. See .

Table 6.8

Table 6.9: Q06a. How would you rate NRCan’s Industrial Energy Audit Incentive Program for Industrial Energy

Innovators compared to the other program(s)?

Frequency %

Better 5 22

Same 10 43

Worse 6 26

Don’t know 2 9

Total 23 100



When comparing the IEAIP to the other programs, the majority of auditors (43%) consider the IEAIP of the same calibre, while 22% consider the program better and 26% consider it worse.

Table 6.10: Q06b Can you please give us one or two reasons for your assessment?

Better Worst


Effective and professional work 3 60 1 17

Application process too long 0 0 1 17

Satisfied service with the customers 1 20 0 0

Satisfied with level of subsidy 1 20 0 0

Red tape 0 0 1 17

Not enough incentive 0 0 2 32

Ineffective, loss of money 0 0 1 17

Other (SPECIFY) 0 0 0 0

Total 5 100 6 100

Even though the frequencies are too small to be conclusive, it seems that the main reason for the better evaluation is the “effective work and professional” variable.

6.6. PROGRAM EXPECTATIONS

A very large majority of the contractors consider that the program meets their expectations. See Table 6.11.

Table 6.11: Q07. Does the program meet your expectations?


Yes 35 87

No (explain: process too long (1), cost too high (1)) 3 8

Don’t know 2 5

Total 40 100



Table 6.12: Q08 Do you have any general comments about the program or suggestions about how it could be

improved?


Very good service with the customers, effective work. 4 10

Process is too slow. 6 15

Organization and coordination flows. 4 10

Level of incentive is insufficient. 8 20

No incentive for the implementation of measures. 1 2

Other (SPECIFY) 8 19

None 10 24

Total 41 100

If we exclude the “other” category, the contractors have five general comments and suggestions about the program; see Table 6.12.

6.7. AUDITOR SURVEY FINDINGS AND CONCLUSIONS

The auditors’ assessment of the program seems very good in general. Program spin-off for the contractors is normal in terms of additional audit work but could be somewhat higher in terms of engineering work, which would be an indicator of more implementation. In the participant survey, we found an implementation ratio of 58%, including intentions. The only room for improvement we can detect is related to the variable “level of incentive is insufficient.” Technically speaking, this type of answer is to be expected with regard to incentive levels. But in fact there is some truth to this, since a brief analysis of the level of the subsidy in proportion to the cost of the audits points toward the same conclusion.

In fact, for 179 audits, the average audit cost is $12,268 while the average annual energy bill is $4,062,146. NRCan’s funding share is 35% and that of other sources of funding is 16%, while the participant’s share accounts for the remaining 49%.

In other words, the 50% limit for audit funding, to a maximum of $5,000, is not reached but is offset by the existence of other programs. Auditors generally believe that the investment in an audit is worthwhile if the annual energy bill is over $100,000, in which case the audit cost would be about $10,000 and the gross savings potential around 10%.



Although more than half the auditors believe that the level of subsidy did not affect the scope of the audit, we believe that without the other subsidies, the program would have attracted fewer participants. This is confirmed by other answers and comments indicating that the incentive level is not high enough. As such, the continuance of other sources of funding must be carefully monitored. If there is any relaxation on the part of these programs and no increase in the NRCan funding level, fewer participants should be expected. Seventy-two (72) of the 179 audits received other funds, mostly provided by Gas Metro (26 audits), Enbridge (25 audits) and ACC (10 audits).

Another issue that should be addressed among the various organizations that offer incentives for such audits is the attribution of savings and GHG reductions. No matter who attributes what to which organization, the key is to avoid double or even triple accreditation, otherwise the accounting of GHG emissions, if considered as an effort within the Kyoto Protocol, will be overestimated.



7. NON-PARTICIPANT SURVEY RESULTS

7.1. STATISTICS

The database of the Industrial Energy Innovators who did not participate in the IEAIP contained 174 companies. From that population, we completed 50 valid interviews. Appendix C presents additional statistical details concerning the response rate calculations.

Table 7.1:

Distribution of Population and Respondents by Province

Province Population Respondent


Alberta 22 13.6 5 10.0

British Columbia 9 5.2 4 8.0

Manitoba 2 1.1 1 2.0

New Brunswick 6 3.4 1 2.0

Newfoundland 1 0.6 0 0.0

Nova Scotia 8 4.6 2 4.0

Ontario 90 51.7 23 46.0

Prince Edward Island 1 0.6 1 0.5

Quebec 28 16.1 11 22.0

Saskatchewan 7 4.0 2 4.0

Total 174 100 50 100

Our sample adequately reflects the provincial distribution of the non-participant population, even though the samples are very small for each province. For example, Ontario is represented by a number of non-participants that approximates the proportion of the population concerned (52% for the population and 46% for the sample). The non-participant surveys lasted an average of 8 50’ minutes.



7.2. AWARENESS

Before we present the data for the 50 valid and completed interviews, we would like to provide information about program awareness on the part of all 71 admissible contacts, after eliminating those who thought they had participated but in fact, after verification, had not participated and were probably confusing the IEAIP with some other NRCan program.

Table 7.2:

Q01. As an Industrial Energy Innovator, are you aware that there is a financial Incentive available for your company from NRCan to help you do an energy Audit?

Frequency %

Yes 50 71

No 21 29

Don’t know 0 0

Total 71 100

Table 7.3:

Q02. Would you like to receive documentation about this program or be contacted by the program authority to obtain information about this program?

Frequency %

Yes 20 95

No 1 5

Don’t know 0 0

Total 21 100

The following tables and questions apply to the 50 admissible interviews.

Table 7.4:

NRCan records show that your company has not yet participated in the Industrial Energy Audit Incentive program offered to all Industrial Energy Innovators such as your company. Is this correct?

Frequency %

Yes 49 98

No 0 0

Don’t know 1* 2

Total 50 100

* This non-participant agreed to be interviewed.



7.3. NON-PARTICIPANTS’ REASONS Table 7.5:

Q03 From the list of reasons I will read you, please tell us why you have not yet participated in NRCan's audit incentive program?

Better

Frequency %

Your company intends to participate in the near future 12 14

Your company has no time to manage an energy audit project 10 12

Your company doesn't have the financial resources 8 10

Your company doesn't need an energy audit 6 7

Your company has already done an energy audit 19 23

Your company prefers to do an energy audit without participating 4 5

The program's administrative procedures are too constraining 6 7

The program's incentive level is too low to do a comprehensive audit 6 7

Your company cannot find a consultant to do the energy audit 1 1

Energy efficiency is no longer an immediate priority of yours 1 1

Your company no longer has an energy efficiency champion 2 2

Other (SPECIFY) 7 8

Refusal/Does not know 2 2

Total 84 100

Percentages apply to the number of answers and not the number of respondents. The fact that the main reason for non-participation is the recent performance of an energy audit is quite normal, since similar programs and awareness about the benefits of energy audits have been in existence for several years in Canada.



7.4. NON-PARTICIPANTS’ RECOMMENDATIONS AND SUGGESTIONS Table 7.6:

Q04 From the following possible changes to the program, which one(s) would be strong enough to ensure your company's participation in the program?

Better

Frequency %

The level of incentive should be raised to a maximum of $10,000 20 26

The auditor should be certified by NRCan 7 9

The energy audit should be standardized 4 5

NRCan should offer a no-interest loan to finance the remaining audit cost 2 3

100% of the audit cost should be reimbursed if all the measures are implemented 19 25

Financial incentives should be offered for the implementation of measures 17 22

Other (SPECIFY) 6 8


Total 77 100

Percentages apply to the number of answers and not the number of respondents. Raising the level of the subsidy or offering funding one way or another could encourage some industries to do more comprehensive audits and possibly save more energy.

7.5. NON-PARTICIPANT SURVEY CONCLUSIONS

If we exclude non-participants who declared that they have participated in the program, awareness about the program among the others is in the range of 71% and the remaining 29% wish to obtain further information. This ratio is, in our opinion, quite good considering personnel turnover in industry.

About 19 (38%) of the non-participants surveyed have already done an energy audit, while 12 (24%) intend to participate in the future. These percentages seem normal since energy audit incentive programs have been in existence in Canada for some time now and through several organizations, mainly governments and public utilities.

The two main suggestions for increasing future participation are related to the incentive level. Raising the incentive level to $10,000 and/or reimbursing the total cost of the audit if all measures are implemented are the two main suggestions. Based on our experience, we tend to favour the second option since it encourages implementation. Nevertheless, before amending the program incentive level, one should be aware that there are other programs that



provide funding in addition to NRCan and that several participants take advantage of such programs. An overbidding of funding, with a significant reduction in the contribution by companies, is not recommended.

The third suggestion is to provide financial incentives for the implementation of measures. This could be an interesting option, but would require a significant budget increase and more complex management procedures.



8. PARTICIPANT SURVEY RESULTS

8.1. DATA PROCESSING

Before importing the information contained in the database of audit recommendations, some analysis was necessary to make sure all questions asked were relevant to the main objectives of the evaluation, which are the achievement of both an energy impact and an environmental impact. As such, we applied two levels of selection filters.

The first filter level was mandatory for asking a given question, while the second was helpful for calculating other aspects, such as the percentage of energy bill savings, the payback ratio, the relationship between implementation and payback and extrapolations.

The end result of this selection is that out of 216 audits containing 842 measures, 179 were eligible for interviews, containing 502 valid measures. Of the 179 audits, 29 could not be contacted due to insufficient contact person information, representing 123 valid measures. Nevertheless, they were included in the extrapolation exercise.

In the end, out of a population of 150 participants (179-29), 103 were surveyed, representing an answering ratio of 68%. In total, questions were asked concerning 308 measures.

The following table presents the results of this analysis.



Table 8.1: Participant Database Analysis

Audits Measures Total measures with adequate documentation

Total measures with inadequate 3

documentation

Total in database 216 Total in 216 audits 842 557 (66.15%) 285 (33.84%)

Less Less

Audits with no valid EE measures 28

Measures not valid in terms of energy

savings 2 - 289 1

Don’t contact list Valid audits 29 2 Don’t contact list

Valid measures -123

Don’t contact list Invalid audits 8 Don’t contact list

Invalid measures -51

Audit too recent for interview 1

Total eliminated for survey 66 Total eliminated -340

(289+51)

Remaining valid audits for survey 150 Remaining valid

measures for survey 379

Ratio for survey 150/216

69.44%

Valid for extrapolation (150+29)

179

(123 +379)

502

Ratio for extrapolation 179/216

82.8%

502/842

59.61 %

Completed surveys 103

Measures surveyed 308

1 We applied three filters for this selection:

1. Measure description must be available (to ask the question)

2. Energy savings in energy units (to calculate energy impact)

3. Type(s) of energy saved (to calculate GHG)

Of the 340 non-valid measures, more than 50 measures were good measures but had no energy impact, such as the correction of power factor, water savings, scheduling and intangible measures. 2 The 37 “don’t contact” mentions are those with no valid contact name that were identified by NRCan through the bouncing back of the advance notification letter sent by e-mail to participants and that could not be corrected. We believe that this situation is quite normal in the industrial sector, where personnel turnover can be relatively high depending on economic conditions. Accordingly, we applied a filter to distinguish valid audits (29) from invalid audits (8) among these 37 audits and used the valid ones for extrapolation purposes, along with the 47 participants who were contacted from among the 150 audits but could not be reached. As such, extrapolation applies to 76 audits (47+29) for a total of 179 audits. 3 Inadequate documentation was labelled by NRCan personnel as measures that do not have the proper information, such as the correction of power factor resulting in energy savings instead of dollars savings only.



8.1.1. SECOND LEVEL SELECTION FILTER

This table was used to establish the limits of the extrapolation exercise and calculate different ratios, such as percentage of gross and net savings compared to energy consumption, average savings in $ and payback ratios, that form the basis of our extrapolation.

Table 8.2:

Result of Second Filter Application

From the 179 valid audits Available Missing Total

Total annual consumption (GJ) 175 4 179

Total annual energy bill ($) 145 34 179

Estimated energy bill ($) 175 4 179

From the 502 measures

Valid for extrapolation Available Missing Total

Recommendation cost ($) 502 0 502

Savings in ($) 499 3 502

Payback in years 499 3 502

Since the average length of the participant questionnaire was 9 minutes, contact names that appeared more than once for different audits of the same company were contacted and surveyed for only one audit (the oldest) to avoid discomfort. The other audits were included in the non-respondent list and used for extrapolation purposes.

Appendix D presents more statistical details concerning the response rate calculations.

8.2. NON IMPACT ISSUES RELATED TO PARTICIPANTS’ RESULTS

The total number of valid surveys was 103 out of 150. Our sample adequately reflects the provincial distribution of the participant population, even though the samples are very small for each province. For example, Ontario is represented by a number of participants approximately equal to the proportion of the population concerned (49% for the population and 47% for the sample).

An analysis of the following tables ( to ) is provided in section 8.14. Table 8.4 Table 8.22



Table 8.3: Distribution of Population and Respondents by Province

Province Population Respondents


Alberta 16 11 9 8.7

British Columbia 5 3 5 4.9

Manitoba 3 2 2 1.9

New Brunswick 1 1 0 0

Newfoundland 1 1 1 1

Nova Scotia 3 2 2 1.9

Ontario 74 49 48 46.6

Price Edward Island 1 1 1 1

Quebec 40 27 31 30.1

Saskatchewan 6 4 4 3.9

Total 150 100 103 100

8.3. VALIDATION Table 8.4:

QInt04. According to NRCan records, your company has received an incentive to undertake an energy audit at your company's (city) facility. Is this information correct?

Frequency %

Yes 100 97

No 2 2

Don’t know 1 1

Total 103 100

8.4. FREE-RIDERSHIP ISSUES Table 8.5:

Q01. Prior to participating in NRCan's Industrial Energy Audit Program, did your company have plans to do an energy audit that would have taken place within a year of when your NRCan-funded audit was

actually done?

Frequency %

Yes 31 30

No 70 68

Don’t know 2 2

Total 103 100



Table 8.6: Q02. If yes, would it have been similar in scope to the one that was eventually funded by NRCan?

Frequency %

Yes 17 55

No 13 42

Don’t know 1 3

Total 31 100

Answers to questions about free-ridership by measure are not presented in a table but translated into impacts in the impact calculation section.

The question about the cost of the audit was unnecessary given the question about scope, which is closely related, and was therefore eliminated.

Table 8.7: Q04. To your knowledge, has your company accessed and/or participated in one or more of the following

NRCan initiatives...

Frequency %

Dollars to $ense workshops 31 24

Industrial Energy Efficiency information sessions 16 13

CIPEC task force meetings 10 8

Heads Up CIPEC newsletter 24 19

None 44 34


Total 128 100

Percentages are in relation to the number of answers and not the number of respondents.

Table 8.8:

Q04a. Would you say that your involvement in this/these initiative(s) influenced you to participate in the Industrial Energy Audit Incentive Program?

Frequency %

Yes 45 80

No 10 18

Don’t know 1 2

Total 56* 100

* 56 =103-(“None”: 44 + “Refusal”: 3)



Table 8.9: Q04b. Which NRCan initiative would you say has influenced you the most?

Frequency %

Dollars to $ense workshops 25 45

Industrial Energy Efficiency information sessions 8 14

CIPEC task force meetings 4 7


None 4 7


Total 56 100

Table 8.10:

Q04c. Did one or more of these initiatives encourage you to specifically ask to study opportunities to save energy and/or dollars when undertaking your energy audit?

Frequency %

Yes 43 77

No 11 20

Don’t know 2 4

Total 56 100

Table 8.11:

Q04d. Did one or more of these initiatives encourage you to look into the cost-effectiveness of energy efficiency measures when evaluating different options?

Frequency %

Yes 46 82

No 8 14

Don’t know 2 4

Total 56 100



8.5. REASONS FOR NON-IMPLEMENTATION Table 8.12:

Q06a. Among the reasons that I am going to list, which one(s) explains why you have not implemented ANY of the measures recommended in your energy audit report?

Frequency %

Payback period is too long 4 16

Investment is too high 13 52

No significant savings 1 4

Investment funds are not available 1 4

Lack of confidence in audit report 3 12

Poor audit quality 2 8


Total 25 100

The number of respondents who did not implement any measure is 25 out of 103.

Table 8.13: Q06b. Among the reasons that I am going to list, which one(s) explains why you have not implemented

SOME of the measures recommended in your energy audit report?

Frequency %

Payback period is too long 7 17

Investment is too high 21 52

No significant savings 2 5

Investment funds are not available 4 10

Lack of confidence in audit report 1 3

Poor audit quality 2 5

Other opportunities with better investment return 1 3

Uncertain of the results of the modifications 1 3


Total 40 100

The number of respondents who implemented some measures is 78 out of 103.



8.6. SPILLOVER ISSUES Table 8.14:

Q07. Does your company operate other plants?

Frequency %

Yes 74 72

No 29 28

Total 103 100

Table 8.15:

Q07a. Did you use the results of this energy audit to implement the same measures in your other facilities without doing another energy audit?

Frequency %

Yes 18 24

No 53 72

Don’t know 3 4

Total 74 100

8.7. PRIOR EXPERIENCE Table 8.16:

Q08. Did you have an energy audit (either comprehensive or limited in scope) done at your facility prior to participating in NRCan's Industrial Energy Audit Incentive Program?

Frequency %

Yes 38 37

No 62 60

Don’t know 3 3

Total 103 100



8.8. REASONS FOR PARTICIPATION Table 8.17:

Q09a-e. On a scale from one to five, one being "unimportant" and five "very important", how would you assess the importance of the following factors in your decision to undertake an energy audit at your

facility?

Score

Requirement to reduce costs? 4.6

Access to financial incentive from NRCan? 3.7

Access to third party technical expertise? 4.5

Corporate mandate to reduce energy consumption? 4.1

Corporate mandate to reduce GHG (Greenhouse gas) emissions? 3.7

8.9. PAYBACK ISSUES Table 8.18:

Q10. What is the maximum payback period acceptable to your company for the implementation of energy efficiency measures? (In years)

Frequency %

1 year 22 21

2 years 48 47

3 years 20 19

4 years 2 2

5 years 9 9

Don’t know 2 2

Total 103 100

Mean (years) 2.29

8.10. AUDITOR SELECTION Table 8.19:

Q11. Did you select an energy auditor from NRCan's Energy Management Service Directory, which is available on their Website?

Frequency %

Yes 18 17

No 79 77

Don’t know 6 6

Total 103 100



8.11. SATISFACTION WITH PROGRAM COMPONENTS Table 8.20:

Q12a-g. On a scale from one to five, one being "unsatisfactory" and five being "very satisfactory", how would you assess the following aspects of the Industrial Energy Audit Incentive program?

Score

The application process and documentation required? 3.9

The time required to process and approve applications? 3.8

The level of incentive? 3.6

The program brochure and documentation? 3.8

Customer service? (e.g. my questions are answered promptly) 4.1

The program's ability to meet your company's needs? 3.8

Your energy auditor's work and report? 3.8

8.12. AWARENESS ISSUES Table 8.21:

Q13 How did you find out about the program?

Frequency %

NRCan Website 8 8

Program brochure 7 7

Other participants 11 11

Contractor 38 37

NRCan personnel 14 14

Information sessions 12 12

Dollar to $ense workshops 4 4


OCETA: Ontario Centre for Environmental Technology Advancement 4 4

Manitoba Hydro 1 1

BC Hydro 1 1

Union Gas 1 1

Enbridge Gas 3 3

Magazines 2 2

Local economic development officer 1 1

Gas Metro 1 1

Utility company 1 1

CFI Kyoto 1 1

Other 1 1

Total 103 100



8.13. COMMENTS AND RECOMMENDATIONS Table 8.22:

Q14. Finally, do you have any comments or recommendations on how to improve this program?

Frequency %

Auditors should be chosen for their experience and competence 2 2

Change the tax code to allow 100% of the capital write-down in the first year 1 1

Good marketing strategy 34 29

Good program 7 7

Incentive should vary depending on the type and scope of the project 1 1

It should give us more possibilities to work with the USA 1 1

Longer payback period would be appreciated 1 1

More incentives for measures aimed towards the Kyoto protocol 3 3

More incentives for the implementation of the measures 2 2

The program met our expectations 7 7

Prioritize cost efficient project 1 1

Would prefer a more individual approach in workshop 1 1

More support is needed when the company moves 2 2

Project costs too much to get incentive 2 2

The program doesn't deal correctly with companies 1 1

More information should be given on natural resources (solar, hydro,...) 3 3

More publicity 9 9

More recommendations should be made about alternate energy sources 2 2

The audit was a sale pitch 2 2

More incentive for retrofits in the manufacturing industry 3 3

Forces us to evaluate ourselves and our energy system 1 1

The audit was of good quality 2 2

The audit was of poor quality 1 1

The communication between NRCan and auditors should be improved 1 1

The information given was good 1 1

The incentive level should be higher 1 1

The incentive level should vary depending on how big the company is 1 1

The process is too slow 1 1

The program should include other types of companies 2 2

The results of the audit were applied in other areas 1 1

The technical information should be more detailed 1 1

Would like more money for small company 1 1

Would like to be able to meet with people form NRCan 1 1

Would like to have an information session 1 1

Would like to have more time to look at the information 1 1

Written information too long and out of date 4 4

None 38 37


Total 103 100



8.14. ANALYSIS OF PARTICIPANTS’ NON-IMPACT ISSUES

Of the respondents, 30% stated that they had plans to conduct an energy audit within one year of when the NRCan-funded audit was actually done ( ). If the answer to that question was yes, it opened a series of sub-questions related to the scope and cost of the audit and the influence of other programs or activities within the IEAIP to establish the real level of free-ridership.

Table 8.5

Intentions to undertake the same kind of audits of the same scope (hence cost) would tend to increase free-ridership, while influence by other NRCan activities would tend to decrease free-ridership. By applying the methodology described in the participant methodology section of this report, the real level of free-ridership was established at a level much lower than the stated 30% (see impact evaluation section).

The Dollars to $ense Workshops and the Heads Up newsletter seem to be the two activities with the most influence in the program (Table 8.9).

Financial aspects such as a high level of investment and long payback period are clearly the two most important factors for the non-implementation of any measure. Audit quality, however, does not seem to be a factor. This contradicts the findings of the advisory committee members regarding audit quality (Table 8.12).

This could mean several things. First, the sample of studies analyzed by the committee is not representative, even though it was selected randomly. Second, participants do not know what to expect from a quality audit. Lastly, it could simply be due to self-selection bias, with participants unwilling to criticize a study partially funded by a third party. In our opinion, a workshop for both parties along with a mandatory audit summary form should improve the quality of the audit. Based on the impact section that follows, we determined that of 308 measures probed, 147 were implemented and 33 were intentions to implement with the next 12 month for a total of 180 measures, or 58% (Table 9.9).

Of the 103 respondents, 74 operate other plants and 18 stated that they used the result of the NRCan-funded audit to implement measures in other facilities (Table 8.14 and Table 8.15), providing for a positive spillover effect that is small in proportion but significant in terms of energy impact (see impact section).



Of the respondents, 37% had a previous audit report done at their facility prior to participating in the IEAIP ( ). Table 8.16

It appears that the three main factors in the decision to undertake an energy audit are requirement to reduce cost, followed by access to third party technical expertise and corporate mandate to reduce energy consumption (Table 8.17).

For 68% of respondents, a payback period of less than 2 years seems to be an acceptable timeframe for investing in energy efficiency measures (Table 8.18).

The list of auditors posted on NRCan’s Web site does not seem to be of great value. This is quite normal since customers prefer a reference from another customer to choosing a name from a list without further information about the contractor’s credibility (Table 8.19).

In another series of factors influencing the decision to participate in the program, the strongest reasons are related to process issues such as customer service, time required by the process and application documentation required. Surprisingly, incentive level comes last with a score of 3.6 out of 5, although it is the first reason for non-participants. This is a very strong score for the process aspect of the program (Table 8.20).

Program awareness sources are mainly contractors and to a lesser degree NRCan personnel and information sessions (Table 8.21).

In terms of comments and recommendations to improve the program, marketing strategy and publicity are clearly dominant issues among the various suggestions ( ), perhaps explaining the 29% of non-participants who were unaware of the program.

Table 8.22



9. ENERGY IMPACT EVALUATION RESULTS

To extrapolate the results of 103 respondents to the 179 valid audits, we established the savings distribution of the 103 audits by payback period. Then we applied the same distribution ratio to the measure of the remaining 76 audits (179-103). For example if the percentage of savings for a payback period of 2 years was 30 % of all the savings, we would apply that same percentage to the measures having a 2 year period from the measures of the 76 remaining audits. The result is a gross savings potential of 7 million GJ/Y for the 179 audits.

9.1. EXTRAPOLATION GJ/Y Table 9.1:

Extrapolation of Gross Savings to Admissible Participants Who Did Not Respond to the Survey Based on Payback Ratios

N=103 surveyed audits N=179 total valid audits

Payback Period GJ/Y GJ/Y

No investment 1,688,054 2,210,406

Less than 1 year 347,985 633,333

1 to 2 years 763,379 984,759

More than 2 years 2,889,195 3,166,558

Total implemented 5,688,613 6,995,055



9.2. GROSS SAVINGS POTENTIAL Table 9.2:

Gross Savings Potential Identified by Audits in GJ/Y (N=502 Valid Measures 179 Valid Audits)

By Measure Type Frequency GJ/Y %

Buildings and grounds 101 249,713 4%

Combustion systems 70 1,602,046 23%

Electrical power 73 815,890 12%

Industrial design 8 138,365 2%

Motor systems 70 1,361,041 19%

Operations 24 76,083 1%

Operation (waste minimization) 15 95,798 1%

Thermal systems 140 2,656,119 38%

Total 502 6,995,055 100%

By Payback Period GJ/Y %

No investment 114 2,210,406 32%

Less than 1 year 85 633,333 9%

1 to 2 years 103 984,759 14%

More than 2 years 199 3,166,558 45%

Total 502 6,995,055 100%

By Industry Type GJ/Y %

Food & beverage 43 408,285 6%

General manufacturing 43 1,932,078 28%

Transportation 16 125,817 2%

Upstream 10 901,283 13%

Others 66 3,627,591 52%

Total 179 6,995,055 100%

9.3. NET SAVINGS OF IMPLEMENTED MEASURES

Net savings were calculated after applying the different correction factors as shown in .

Table 9.3



Table 9.3: Net Savings of Implemented Measures in GJ/Y by Measure Type, Payback Period and Industry Type

Measure Type Gross savings GJ/Y

Implementationratio (%) - FR + SO = Net %

Buildings and grounds 249,713 28% 2,621 3,822 71,208 3%

Combustion systems 1,602,046 45% 4,575 113,677 830,894 38%

Electrical power 815,890 83% 1,569 24,387 703,741 32%

Industrial design 138,365 8% 0 0 10,907 0%

Motor systems 1,361,041 7% 12,155 47,703 124,314 6%

Operations 76,083 84% 0 117,572 181,713 8%

Operation (waste minimization) 95,798 0% 0 0 0 0%

Thermal systems 2,656,119 9% 1,435 47,407 274,640 12%

Total 6,995,055 27% 22,355 354,569 2,197,418 100%

Payback Period Net GJ/Y %

No investment 944,180 43%

Less than 1 year 110,863 5%

1 to 2 years 607,719 28%

More than 2 years 534,656 24%

Total 2,197,418 100%

Industry Type Net GJ/Y %

Food & beverage 158,408 7%

General manufacturing 822,693 37%

Transportation 74,668 3%

Upstream 361,407 16%

Others 780,241 36%

Total 2,197,418 100%

Clearly, measures concerning combustion systems and electric power generated the highest levels of net savings. By payback measures, behavioural measures that do not need any investments were clearly the first choice for implementation.

9.4. NET SAVINGS FROM INTENTION TO IMPLEMENT

For measures that were not implemented, a series of questions were asked related to intention to implement such measures and in what proportion. The same distortion effects were applied to both intentions to implement and implemented measures. However, the block of savings related to intentions is of course less solid than that related to implemented measures.



Table 9.4:

Intention Savings in GJ/Y by Measure Type, Payback Period and Industry Type

Measure Type Gross savings GJ/Y

Implementation intention ratio (%) - FR + SO = Net %

Buildings and grounds 249,713 8% 793 1,156 21,531 5%

Combustion systems 1,602,046 17% 1,751 43,512 318,039 69%

Electrical power 815,890 3% 61 945 27,275 6%

Industrial design 138,365 0% 0 0 0%

Motor systems 1,361,041 0% 93 364 948 0%

Operations 76,083 0% 0 0 0 0%

Operation (waste minimization) 95,798 0% 0 0 0%

Thermal systems 2,656,119 3% 472 15,596 90,351 20%

Total 6,995,055 6% 3,170 61,573 458,143 100%

Payback Period Net GJ/Y %

No investment 344,776 75%

Less than 1 year 27,692 6%

1 to 2 33,471 7%

More than 2 52,203 11%

Total 458,143 100%

Industry Type Net GJ/Y %

Food & beverage 99,823 22%

General manufacturing 6,503 1%

Transportation 16,170 4%

Upstream 0 0%

Others 335,648 73%

Total 458,143 100%

Table 9.5

Total Net Energy Impact from Implemented Measures and Intention to Implement Measures

Implemented measures 2,197,418 GJ/Y

Intention to implement measures 458,143 GJ/Y

Total 2,655,561 GJ/Y

Precision ± 9.3 %



9.5. SAVINGS RATIOS Table 9.6:

In Savings Ratios, Percentage of Energy Consumption (GJ/Y)

In percentage of energy consumption (GJ/Y) All (179 audits) Trimmed Average (5%)*

Gross energy savings potential (GJ/Y) 6,995,055 3,219

Annual energy consumption (GJ/Y) 245,824,029 46,297

Mean ratio (%) 2.8% 7.0%

Net implemented savings (GJ/Y) 2,197,418 938

Annual energy consumption (GJ/Y) 245,824,029 44,364

Mean ratio (%) 0.9% 2.1%

Net implemented savings (GJ/Y) 2,197,418 938


Ratio net to gross (%) 31.4% 29.1%

The trimmed average excludes outliers of 5% from both extremities of the distribution curve. As such, the more significant percentage to retain is a level of savings of 7% of the total energy consumption, which is in the range of what the program literature estimates (5% to 15%). It is important however to mention that the savings, both in terms of $ and GJ/Y, are audit estimates.

Table 9.7

In Savings Ratios, Percentage of Energy Consumption ($)

In percentage of energy expenditure ($) All (179 audits) Trimmed Average (5%)

Gross energy savings potential ($) 26,574,600 34,030

Annual energy consumption ($) 727,124,300 473,510

Ratio mean (%) 3.7% 7.2%

Net implemented savings ($) 15,345,000 10,050

Annual energy consumption ($) 727,124,300 473,510

Ratio mean (%) 2.1% 2.1%

Net implemented savings ($) 15,345,000 10,050

Gross energy savings potential ($) 26,574,600 34,030

Ratio net to gross (%) 57.7% 29.5%



Table 9.8: Savings Ratios for Implemented Measures and Intentions

All (179 audits) Trimmed Average (5%)


Net savings including intentions (GJ/Y) 2,655,561 1,134

Ratio 38.0% 35.2%

Table 9.9:

Measures Implementation Ratio

Measure status Number

Recommended valid measures for 103 surveyed audits 308

Implemented measures and intentions 180

Implementation ratio 58%



10. GREENHOUSE GAS REDUCTION

Greenhouse gases were calculated using NRCan conversion factors. Several variables were used to establish GHG reduction and extrapolate them to the population as well as to intentions. These variables were the amount of savings per measure in energy units, the source of energy, the origin or province and the payback period.

10.1. IMPLEMENTED MEASURES Table 10.1:

GHG Reduction Tonnes Equivalent CO2 Based on Net Savings for 179 Audits

Province of Origin Electricity Natural Gas Oil Propane Total

Alberta 1,753.1 12,522.1 0 0 14,275.2

British Columbia 1,185.6 2,668.6 0 0 3,854.2

Manitoba 310.7 0 0 0 310.7

New Brunswick 0 0 0 0 0

Newfoundland 0 305.6 0 0 305.6

Nova Scotia 0 0 0 0 0

Ontario 21,326.8 40,241.7 0 0 61,568.5

Prince Edward Island 0 0 0 0 0

Quebec 4.4 12,916.0 1,417.5 0 14,338.0

Saskatchewan 3,148.1 10,149.5 0 0 13,297.5

Total tons CO2 equivalent 27,728.7 78,803.5 1,417.5 13 107,949.7

Total in GJ 882,321.1 1,281,286.6 33,810.3 2,197,418.0



10.2. INTENTION TO IMPLEMENT Table 10.2:

GHG Reduction Tonnes Equivalent CO2 Based on Net Savings for 179 Audits

Province of Origin Electricity Natural Gas Oil Propane Total

Alberta 0 0 0 0 0

British Columbia 0 0 0 0

Manitoba 0 0 0 0

New Brunswick 0 0 0 0 0

Newfoundland 0 0 0 0 0

Nova Scotia 474.4 0 0 0 474.4

Ontario 541.2 724.3 0 0 1,265.5

Prince Edward Island 0 0 0 0

Quebec 21,412.1 0 0 21,412.1

Saskatchewan 0 0 0 0

Total tons CO2 equivalent 1,015.6 22,136.3 0 0 23,151.9

Total in GJ 27,530.6 430,612.4 458,143.0

Table 10.3:

Total Greenhouse Gas Reduction

Implemented measures 107,949 T CO2 equivalent

Intentions to implement measures 23,151 T CO2 equivalent

Total 131,102 T CO2 equivalent

Precision ± 9.3 %



11. CONCLUSION

The Industrial Energy Audit Incentive Program is sponsored by NRCan’s Office of Energy Efficiency and offers Industrial Innovators the opportunity to share the cost to undertake an energy audit of their facilities. NRCan’s contribution represents 50% of the audit cost, up to a maximum of $5,000.

The program was launched in fiscal year 2001/2002 for a period of 5 years. The program objective is to fund some 350 to 500 audits by the end of fiscal year 2005/2006. At the time of the present evaluation, 216 audits reports had been completed and hence constituted the statistical population to be evaluated.

The IEAIP quantitative objective with respect to GHG emissions reduction was established at 0.5 Mt by 2010.

This evaluation has two dimensions. The first is an impact evaluation to estimate net energy savings and GHG reductions related to the implementation of measures identified in the energy audits. The second is a process evaluation of the program.

One of the important findings of the study is related to the audit’s quality in terms of both content and reporting, which has an impact on the process of gleaning information from the program tracking database. In terms of content, a panel of experts appraised nine randomly selected audits, giving them an average score of 55%. From the participants’ perspective, however, audit quality doesn’t seem to be an issue.

In terms of program appraisal by all parties, including the evaluator, participants, non-participants and consultants, the results indicate a high level of satisfaction, with only minor modification proposals.

11.1. IMPACT ISSUES

In terms of energy impact, the study estimates gross energy savings of 6,995,055 GJ/Y for 179 audits and net savings of 2,655,561 GJ/Y for a net to gross ratio of 38.0%. The net savings are comprised of a solid 2,197,418 GJ/Y from implemented measures and a less solid



458,143 GJ/Y from intentions to implement within the next 12 months. The precision factor for the savings is ± 9.3%.

In terms of proportions, the gross savings represent 2.85% of total energy consumption. The trimmed average (exclusion of extreme values) raises this percentage to almost 7% by eliminating a few outliers with very large energy consumptions. This 7% savings potential is quite normal since it is related to total energy consumption and not to a specific type of energy saved by a particular measure. In other words, annual energy consumption is not a real base line.

Free-ridership is low, due mainly to the influence of other NRCan activities that reduce the level of free-ridership. The spillover effect is relatively high, especially since participants can apply for another audit for a different facility, giving the program a positive spin.

In terms of GHG reduction, the study estimates the savings at 131,000 tonnes equivalent CO2 per year. If this figure is extrapolated to 500 audits with the same correction factor for audits with no energy impact found in this study (179/216), the final impact could be around 300,000 tonnes of CO2 equivalent. However, we invite the reader not to draw quick conclusions with respect to final impact, since too many variables could change for the remaining period of the program and extrapolation of this kind carries a high level of uncertainty.

Participants saved a minimum of $15,345,000 in one year from net implemented measures. Some participants compound these savings through measures with no impact on energy consumption, such as water savings or correction of power factors. The program’s cumulative expenditures for the 339 audits were $3,796,000, for a very comfortable payback period if we use a minimum and conservative five-year savings life span.

Finally, it is not clear from the Treasury Board how the quantitative objective of saving some 500,000 tonnes per year was calculated, nor what number of audits are expected to receive funding. According to program authorities, the objective is derived from similar programs that have been examined, but no documentation on this mater is available. If the 500,000 tonnes is somehow related to what the program brochure indicates as possible savings from the 350 to 500 audits in the magnitude of 5 to 15% with a payback period of less than 2 years, we believe that the 500,000-tonne objective is maybe a bit too high, given that the average savings amounted to 7 %. Proper documentation on how the 0,5 Mt was calculated would have helped to establish whether the objective was too high or the results of the audits too low.



11.2. PROCESS ISSUES

After analyzing different aspects of the program process, we believe that the performance indicators mentioned in the program documentation are available for program monitoring. In our examination of the program in question, we encountered a database quality control problem.

All program forms and checklists were examined, as well application processing and response time. The time required to process an application and issue a response to the participant is 10 working days, which seems entirely adequate, while the time required to process a claim and send the contribution to the participant seems a bit high at 45 working days.

11.3. TRACKING ISSUES

The database examination indicates that the gleaning process is somewhat deficient. The level of missing or inadequate data is relatively high, as reflected by an examination of the complete database. Indeed, an initial examination of the database showed a 38% level of inadequate data (according to program authorities).

We have traced that problem to two sources:

The first is the quality of some audits, in which basic and important information is simply missing.

The second is the gleaning process, in which information may not be clearly available in the audit and a bit of research or calculation is needed to find the missing data. In other cases, the information was available but not recorded, for reasons that are not given.

Before using the database for the survey, program authorities reviewed it to reduce the level of inadequate data, insofar as it was not related to the first reason. This reduced the inadequate data ratio to 33.8%, or 285 inadequate measures of a total of 842 measures for 216 audits, indicating that the second reason is far less important than the first (audit quality).

Our examination of the various fields in the database showed that only the duration of each measure is missing. Without such information, one cannot establish cumulative impacts or



undertake any retention studies in the future. Information about duration is included in the proposed energy audit summary form.

11.4. AUDITOR’S ISSUES

The auditors’ assessment of the program seems very good in general. Program spin-off for the contractors is normal in terms of additional audit work but could be somewhat higher in terms of engineering work, which would be an indicator of more implementation. In the participant survey, we found an implementation ratio of 58%, including intentions. The only room for improvement we can detect would be related to the level of incentive, which is considered too low by some auditors. Technically speaking, this type of answer is to be expected with regard to incentive levels. But in fact there is some truth to this, since a brief analysis of the level of the subsidy in proportion to the cost of the audits points toward the same conclusion.

Based on 179 audits, the average cost of the audits is $12,268 for an average annual energy bill of $4,062,146. NRCan’s funding share is 35% and that of other sources is 16%, with participants assuming the remaining 49%.

In other words, the 50% limit for funding the audit, with a maximum of $5,000, is not reached but is offset through other programs. Auditors generally believe that the investment in an audit is worthwhile if the annual energy bill is over $100,000. In this case, the audit would cost about $10,000 and identify about 10% of gross savings.

Even if more than half the auditors felt that the subsidy level did not affect the scope of the audit, we believe that without the other subsidies, the program would have attracted fewer participants. This is confirmed by other answers and comments indicating that the incentive level is not high enough. Most of the other sources of funds are provided by Gas Metro with 26 contributions, Enbridge with 25 contribution and ACC with 10 contributions. In all, 72 of the 179 audits received funding from sources other than IEAIP.

Another issue that should be addressed among the various organizations that offer incentives for such audits would be the attribution of savings and GHG reduction. No matter who attributes what to which organization, the important thing is to avoid double or even triple accreditation, otherwise the accounting of GHG emissions, if considered as an effort within the Kyoto Protocol, will be overestimated.



11.5. NON-PARTICIPANTS’ ISSUES

If we exclude non-participants who declared that they have participated in the program, awareness about the program among the others is in the range of 70% , with the remaining 30% wishing to obtain further information. We feel this ratio is quite good considering personnel turnover in industry.

About 38% of non-participants have already done an energy audit, while 24% intend to participate in the future. These percentages seem normal since energy audit incentive programs have been in existence for a certain time now in Canada through several organizations, mainly governments and public utilities.

The two highest ranking suggestions for increasing participation in the future are related to the level of the incentive. Raising the incentive level to $10,000 and/or reimbursing the total cost of the audit if all measures are implemented are the two major suggestions. Based on our experience, we tend to favour the second option since it encourages implementation. Nevertheless, before amending the program incentive level, one should be aware that other programs in addition to NRCan provide funding, and that several participants take advantage of such programs. However, an overbidding of funding is not recommended.

As for the third-ranked suggestion, with respect to providing financial incentives for the implementation of the measures, it could be an interesting option, but would require a significant budget increase and more complex management procedures.

11.6. PARTICIPANTS’ ISSUES

The screening of the database for valid questions resulted in a net valid balance of 179 of 216 audits (82.8%) and 502 valid measures of a total of 842 measures (59.61%). Naturally, measures that may be good but have no impact on energy consumption, such as water savings or power factor correction, were considered invalid for the purpose of this evaluation.

About 30% of respondents stated that they had plans to do an energy audit within one year of when the NRCan-funded audit was actually done. If the answer to that question was yes, it



opened a series of sub-questions related to the scope and cost of the audit as well as the influence of other IEAIP programs or activities to establish the real level of free-ridership.

Intentions to undertake the same kind of audits of the same scope (hence cost) would tend to increase free-ridership, while the influence of other NRCan activities would tend to decrease free-ridership. By applying the methodology described in the participant methodology section of this report, the real level of free-ridership has been established at a much lower level than 29% which was established by asking the timing questions (see Table 7.2 Q01).

The Dollars to $ense workshops and the Heads Up newsletter seem to be the two activities with the most recruiting influence in the program.

Financial aspects such as a high level of investment and long payback period are clearly the two most important factors for non-implementation of any measure. Audit quality, however, does not seem to be a factor, in contradiction of the findings of the advisory committee members regarding audit qualities.

This could mean several things. First, the sample of studies analyzed by the committee is not representative, even though it was selected randomly. Second, participants do not know what to expect from a quality audit. And lastly, it could simply be due to self-selection bias, with participants unwilling to criticize a study partially funded by a third party. We believe that a workshop for both parties, along with a mandatory audit summary form, would improve audit quality. From the impact section that follows, we established that of 308 measures questioned, 147 were implemented and 33 were intentions to implement within the next 12 months, for a total of 180 measures or 58%.

Of the 103 respondents, 74 operate other plants and 18 stated that they used the result of the NRCan-funded audit to implement measures in other facilities, generating a positive spillover effect that is small in proportion but significant in terms of energy impact.

More than one third of the respondents had an audit report done at their facility prior to their participation in the IEAIP.

Requirements to reduce costs, access to third party expertise and corporate mandate to reduce energy consumption are the three major factors in the decision to undertake an energy audit.

For 68% of respondents, a payback period of less than 2 years seems to be an acceptable timeframe for investing in energy efficiency measures.



The list of auditors posted on NRCan’s Web site does not seem to be a primary reference. This is quite normal since customers prefer referrals from other customers to picking a name from a list without further information about the contractor’s credibility.

In another series of factors influencing the decision to participate in the program, the strongest reasons are related to process issues such as customer service, time required for process and application documentation required. Surprisingly, level of incentive comes last with a score of 3.6 out of 5, while it is the first reason for non-participants. This is a very strong score for the process aspect of the program.

Contractors, NRCan personnel and information sessions are the main sources of program awareness.

In terms of comments and recommendations for improving the program, marketing aspects and publicity are clearly dominant among the various factors, which may explain the 29% of non-participants who are not aware of the program.



12. RECOMMENDATIONS

In light of these findings, the ADEC-Zariffa consortium recommends the following measures to improve IEAIP performance.

1. Introduce a mandatory audit summary form to be filed by the consultants and included in the audit reports. A proposed form is provided in appendix A. This form, which is already used for a similar program by Gas Metro, should increase the quality of the audits as well as any future evaluations.

2. The life duration of the measures should be available in all audits since it is a very important input for the calculation of both energy and GHG cumulative effects. Also, without it one cannot compare impact to the program objectives of 0.5 Mt equivalent CO2 per year by the year 2010. This information should be included in the mandatory form mentioned above since it is an important decision-making factor for participants, especially those who calculate the net present value (NPV) of each measure.

3. In the program budget breakdown, it is difficult to assess the actual grants and contributions share since it seems that funds from operation and management can be transferred to grants and contributions. Without this transfer, it appears that grants and contributions account for just 14% of the total budget, which is totally disproportionate. We recommend that, in future, the budget-planning breakdown be more detailed for performance indicator purposes.

4. It is strongly recommended that, in the future, an evaluation plan be designed along with the program theory to avoid any unpleasant surprises. Evaluation plans usually specify program objectives, the type of evaluations foreseen, the evaluation methodology and the data collection mechanism.

5. As far as staffing is concerned, the total value of staffing should be determined in order to assess the relative importance of efforts (program expenses) and deliverables (grants and contributions).

6. To improve client satisfaction, we recommend delegating final approval authorization for claims to a lower level of authority in order to shorten the 45-day processing time.



7. Also, program form design dates and revision dates should be printed on each form to avoid confusion. It should, however, be kept in mind that too many in-course modifications could affect database consistency and complicate analysis.

8. To improve the quality of audit reports, we recommend training in the form of a workshop for both auditors and participants in order to establish a strong link between what participants should expect and what auditors should deliver.

9. The continuance of other sources of funding to undertake energy audits must be carefully monitored. Should there be any relaxation of these programs and if NRCan does not increase its funding, fewer participants can be expected. The accreditation of energy savings or GHG reductions among various parties should also be addressed to avoid any overlapping effects.

10. We recommend that the next evaluation include non-energy benefits, which could be quite important in some cases and provide a positive spin to the program.

11. Lastly, pre-qualification of auditors by type of expertise, especially in industrial process, would improve audit quality and client satisfaction.



APPENDIX A: PROPOSED ENERGY AUDIT SUMMARY FORM { TC "APPENDIX A: PROPOSED ENERGY AUDIT SUMMARY FORM " \F C \L "1" }

Participant information Annual Energy Consumption : End dateName : Energy source Natural Gas Oil Electricity Coal Other specify) Total TotalCIPEC Sector : Units (m³) (Litres) MWh Tons (GJ) ($)Address : Quantities# and street Energy Bill ($)CityProvince and PCTelephone : Fax : E-mail :

Brief measure description and audit's reference

Mea

sure

use

ful l

ife (y

ears

)

Ener

gy s

ourc

e

Ener

gy u

nits

(m3, K

Wh,

lit

res,

etc

.)

Ener

gy S

avin

gs (u

nits

)

% o

f sav

ings

of s

ame

ener

gy s

ourc

e

Savi

ngs

valu

e ($

)

Mea

sure

cos

t ($)

Pay

back

per

iod

(yea

rs)

1-

2-

3-

4-

Name:Address:Phone: Fax : E-mail: Page 1 of 1 Audit completion date : (Y/M/D)

Instructions:

IEAIP Form # Date

NRCan - INDUSTRIAL ENERGY AUDIT INCENTIVE PROGRAMENERGY AUDIT SUMMARY FORM

NRCanParticipant #

Contractor information

Mea

sure

rela

ted

end-

use



APPENDIX B: AUDITOR SURVEY STATISTICS { TC "APPENDIX B: AUDITOR SURVEY STATISTICS " \F C \L "1" }

Table B.1: Response Rate Calculation for Auditors

Response rate formula %

Response rate at the filter stage (B+D+C1+C2 / B+E+D+C1+C2 ) 81

Response rate to main survey among admissible cases (C / D+C1+C2) 50

Admissibility and incidence rate (AIR) = (D+C1+C2 / B+D+C1+C2) 100

Global response rate to survey (C1+C2 / (D+C1+C2+ (AIR*E) ) 40

Answering ratios

Formula code Respondents %

Starting Selection 124 100

Wrong /out of order telephone number 7 6

Foreign language + 1 1

Duplicates + 17 14

A Total non-valid for sampling 25 20

Starting selection 124 100

Total non-valid - 25 20

Valid Sample 99 80

B Non admissible 0 0

C1 Completed admissible interviews + 40 40

C2 Interviews completed partially + 0 0

Total completed 40 40

Admissible absent + 30 30

Admissible refusal + 10 10

D Total admissible uncompleted 40 40

No answer or answering machine + 19 19

Respondent absent admissibility unknown + 0 0

Refusal admissibility unknown + 0 0

Absent for a long period + 0 0

E Total with valid phone number but admissibility unknown 19 19



APPENDIX C: NON-PARTICIPANT SURVEY STATISTICS{ TC "APPENDIX C: NON-PARTICIPANT SURVEY STATISTICS " \F C \L "1" }

Table C.1: Response Rate Calculation for Non-Participants

Response rate formula %

Response rate at the filter stage (B+D+C1+C2 / B+E+D+C1+C2 ) 68

Response rate to main survey among admissible cases (C / D+C1+C2) 74

Admissibility and incidence rate (AIR) = (D+C1+C2 / B+D+C1+C2) 67

Global response rate to survey (C1+C2 / (D+C1+C2/ (AIR*E) ) 51

Answering ratios

Formula Code Respondents %


Wrong / out of order telephone number 7 4

Duplicates + 23 13




Valid Sample 144 83

B1 Has participated in the program (non admissible) + 10 7

B2 Never heard of the program and wishes program documentation + 20 14

B3

B4

Never heard of the program and does not wish program documentation

Out of business

+

+

1

1

1

1

Total B Non admissible 32 24

C1

C2

Completed admissible interviews

Partially completed +

49

0

31

0












APPENDIX D: PARTICIPANT SURVEY STATISTICS { TC "APPENDIX D: PARTICIPANT SURVEY STATISTICS " \F C \L "1" }

Table D.1: Response Rate Calculation for Participants

Formula calculations %

Response rate at the filter stage:(B+D+C1+C2/B+E+D+C1+C2)* 96

Response rate to main survey among admissible cases (C / D+C1+C2)* 80

Admissibility and incidence rate (AIR) = (D+C1+C2/B+D+C1+C2)* 97

Global response rate to survey (C1+C2/( D+C1+C2+(AIR*E))* 77

Answering ratios

Formula Code Respondents %


Wrong / out of order telephone number 0 0

Duplicates + 12 8




Valid Sample 138 92

B Non admissible 4 3

C1 Completed interviews + 103 75

C2 Interviews partially completed + 0 0












APPENDIX E: EXPERT AUDIT APPRAISAL #1{ TC "APPENDIX E: EXPERT AUDIT APPRAISAL #1" \F C \L "1" }

1 2 3 4 5 6 7 8 9Note 1 Note 2 Note 3 Note 4 Note 5 Note 6 Note 7 Note 8 Note 9

Weight Evaluation criteria Maximum score

Project no. Average

10%5 3 4 4 3 1 1 3 4 3 2,9

20%5 5 5 5 5 1 2 3 4 2 3,65 3 3 4 2 1 2 3 4 3 2,85 3 2 3 2 2 2 3 3 2 2,4

10%5 1 2 4 1 1 1 3 3 2 2,0

10%5 4 5 5 4 4 3 2 4 3 3,85 2 4 4 3 4 1 3 4 2 3,05 3 4 3 2 3 1 1 3 1 2,3

25%5 3 5 4 2 4 2 3 3 2 3,15 3 4 3 3 2 2 2 3 0 2,45 2 5 4 3 3 2 2 3 2 2,9

25%5 3 3 4 2 3 2 2 4 1 2,75 2 1 3 2 2 1 3 4 4 2,4

100% 55% 67% 76% 49% 44% 33% 52% 71% 43% 54%

Recommended improvements are significant on energy costs

TOTAL (%)

Potential savings found justifies the cost of the study

Clear economic evaluations

Reliability of the technical solutions

Comprehensiveness of the study

Degree of innovation

Reliability of the savings calculations

Clear description of the existing situation

Usefulness of the study for the client

Conformity with the guidelines

Presentation of global energy picture has been included

Degree of compliance with the study guidelines

Value i.e. Cost to Benefit ratio

Reliability

Clear description of the improvements

Innovative solutions have been proposed

Reliability of the investment estimates

Clarity of the report

Main energy consuming sectors have been analysed



APPENDIX F: EXPERT AUDIT APPRAISAL #2{ TC "APPENDIX F: EXPERT AUDIT APPRAISAL #2" \F C \L "1" }


10%5 4 4 4 4 1 3 1 4 1 2,9

20%5 5 5 5 5 0 1 0 2 0 2,65 4 4 4 5 2 3 1 3 1 3,05 5 3 5 5 4 3 0 3 3 3,4

10%5 1 1 2 2 1 2 0 1 0 1,1

10%5 5 5 5 5 5 2 0 3 2 3,65 4 5 5 5 5 2 0 3 3 3,65 3 5 3 3 4 4 0 3 0 2,8

25%5 4 4 4 5 5 3 4 5 3,85 3 4 3 1 5 3 3 0 2,45 3 4 3 1 5 4 4 0 2,7

25%5 4 4 4 4 5 4 0 4 2 3,45 3 3 3 3 2 3 0 2 2 2,3

100% 71% 74% 74% 70% 64% 59% 3% 60% 29% 56%

Project no. Average



Reliability


Evaluation criteria Weight Maximum score







Presentation of global energy picture has been includedMain energy consuming sectors have been analysed








TOTAL (%)



APPENDIX G: EXPERT AUDIT APPRAISAL #3{ TC "Appendix G: Expert Audits Appraisal #3" \F C \L "1" }


10%5 4 4 3,5 3 1 1 2 4 3 2,8

20%5 4 4 4 4 1 1 2 4 3 3,05 3 4 3,5 4 1 1 2 4 3 2,85 4 3 1 2 2 1 3 3 2,1

10%5 2 3 2 2 2 2 2 2 2 2,1

10%5 4 4 3,5 3 1 2 2 4 2 2,85 4 4 3 3 3 2 2 4 1 2,95 3 4 3 2 3 2 3 3 1 2,7

25%5 4 4 3 2 3 2 3 4 2 3,05 3 3 4 2 4 2 3 3 0 2,75 3 3 3 2 2 2 3 2 1 2,3

25%5 3 4 4 3 4 1 1 4 1 2,85 2 3 4 2 4 2 2 3 3 2,8

100% 63% 67% 68% 50% 51% 33% 42% 67% 40% 53%



TOTAL (%)






Presentation of global energy picture has been includedMain energy consuming systems have been analysed




Project no.Maximum score




Average



Reliability


Evaluation criteria Weight



APPENDIX H: AUDITOR QUESTIONNAIRE { TC "Appendix H: Auditor Questionnaire" \F C \L "1" }



APPENDIX I: NON-PARTICIPANT QUESTIONNAIRE { TC "Appendix I: Non-Participant Questionnaire" \F C \L "1" }



APPENDIX J: PARTICIPANT QUESTIONNAIRE { TC "Appendix J: Participant Questionnaire" \F C \L "1" }

natural resources canada · 2020. 11. 9. · natural resources canada office of energy efficiency...

Documents