measuring the effectiveness of your energy efficiency efforts carol mulholland american public power...
TRANSCRIPT
Measuring the Effectiveness of Your Energy Efficiency Efforts
Carol Mulholland
American Public Power Association National Conference June 16, 2009
Contents
Why Implement Energy Efficiency Programs? How Do You Know They Work? Evaluation Overview Process Evaluation Impact Evaluation
Why Implement Energy Efficiency Programs?
Rising fuel costs Rising demand for electricity More concern over emissions leading to
difficulty building new power plants More global competition for energy More profitable from utility business
perspective Environmental (global warming) concerns
Policy Changes We Are Seeing Today
Cost Recovery Program spending on the rise (elec/gas) Looking at DSM as alternatives or deferral of
T&D upgrades RPS and EPS on the rise First forward capacity market Local politicians in the GHG arena Potential for carbon tax
Cost of EE vs. Price of Electricity: July 2005
0
0.04
0.08
0.12
0.16
CA CT MA NJ NY VT
Cen
ts/k
Wh
Price Range of Electricity for Residential,Commercial and Industrial Customers (July 2006)
Cost of Energy Efficiency
Source: State of Delaware, Sustainable Energy Utility Task Force Briefing Book, http://www.seu-de.org/documents.html
Energy Efficiency Uncertainties
Motivation - Regulatory, capacity, T&D or energy affordability?
Focus - Internal (system needs) external (customer needs)?
Policy Context- State Energy Efficiency Portfolio Standards, Federal Carbon tax?
Market - Size and make-up of the market for EE by sector?
Financials - Cost and benefits, avoided costs, compared to what?
Contents
Why Implement Energy Efficiency Programs? How Do You Know They Work? Evaluation Overview Process Evaluation Impact Evaluation
How Many of You . . .
. . . Are counting on energy efficiency/DSM programs to offset increased demand?
. .. Are counting on energy efficiency to help offset clip peaks?
. . . Are implementing energy efficiency/DSM programs because someone has told you to?
. . . Have detailed program operations manuals that explain how your utility's energy efficiency programs work?
. . .Maintain a database of all program expenditures and estimated savings?
If you answered “YES” to the questions in BLUE, you need to evaluate your programs to find out whether you’re getting the results you expect.
If you answered “YES” to the questions in GREEN, you’re already part way there.
Contents
Why Implement Energy Efficiency Programs? How Do You Know They Work? Evaluation Overview Process Evaluation Impact Evaluation
The Importance of Program Evaluation
Definition: Evaluation is a systematic process for an organization to obtain information on its activities, its impacts, and the effectiveness of its work, so that it can improve its activities and describe its accomplishments.*
Benefits:An effective evaluation will help you learn about your successes [and shortcomings], share information with key audiences, and improve your services.*
Other reasons: To assign a value to energy efficiency as a potential asset To verify impacts for capacity and carbon trading To compare EE investments against alternative investments
for meeting demand (integrated resources planning)
Why Evaluate?
Evaluation: Measures program impacts Provides valuable information about program operations Offers a positive customer contact opportunity Shows what you’ve accomplished to all interested parties,
including City Councils and Utility Boards Meets regulatory requirements, for those who have them
What are you measuring?
Energy savings
Demand Savings
Customer satisfaction
Technology penetration
The Way it Should be
However, the role of evaluation can go well beyond simply documenting savings to actually improving programs and providing a basis for future savings estimates
If applied concurrently with program implementation, evaluations can provide information in real time to allow for course correction if needed
In summary, evaluation fosters more effective programs and justifies increased levels of energy-efficiency investment. Perhaps this was best put by Economist John Kenneth Galbraith, “Things that are measured tend to improve.”
Interaction
Process EvaluationProcess Evaluation Process EvaluationProcess Evaluation Market Assessment Market Assessment Market Assessment Market Assessment
Impact Evaluation Impact Evaluation Impact Evaluation Impact Evaluation
When To Evaluate?
Program Program Implementation Implementation
Program Program Implementation Implementation Program Design Program Design Program Design Program Design
Program Program Evaluation Evaluation Program Program
Evaluation Evaluation
is what makes
evaluation useful
The feedback
loop…
Evaluate Early and Often
Evaluation should be an ongoing process Feedback loop – measure change over time Use the information – What you discover may
cause you to alter your strategy as you go. Make mid-course corrections as necessary.
Interpretation is the most important step – Be careful how the data you have collected is interpreted (and by whom). You should carefully control the outcomes of evaluation such that raw data does not get misinterpreted by non-professionals.
Contents
Why Implement Energy Efficiency Programs? How Do You Know They Work? Evaluation Overview Process Evaluation Impact Evaluation
Process Evaluation
Assesses the process a program undergoes during implementation
Documents program goals and objectives from a variety of perspectives
Describes program strengths and weaknesses so that success is highlighted and improvements can be made
Designing a process evaluation
Identify target audiences Determine what type of research to undertake with each
(e.g. Interviews? Surveys? Focus groups?) Lay out the evaluation timeline, taking into consideration
both internal and external deadlines Important to seek out independent 3rd party for an effective
evaluation – even more than impact work To encourage candor on the part of delivery
agents and customers alike To provide credibility with outside parties that an
unbiased view will be offered
Market Assessment and Market Transformation
Market Assessment looks at the broader market for EE products and services within which a program operates
It is typically done BEFORE or AS PART OF program design
It can be done as an EVALUATION ACTIVITY, when looking at the broader market effects of an EE program
(e.g, sometimes rebate programs may increase product availability and drive product prices down, resulting in…)
Market Transformation Market transformation is a goal for EE programs that
seek to overcome significant barriers to adoption in the marketplace
Market Assessment Goals
To determine the size of the program market To assess the market’s characteristics To find out what impact, if any, the energy
efficiency programs had
Barriers to Market Transformation
High prices Low availability Low awareness Lack of education as to use of product or its
merits Poor infrastructure for proper installation
and support of product
Content
Why Implement Energy Efficiency Programs? How Do You Know They Work? Evaluation Overview Process Evaluation Impact Evaluation
What do we mean by impacts?
Demand and energy savings, market effects, and the environmental and economic costs and benefits that result from an energy services program.
Savings are typically reported on a gross and net basis. “Net energy and demand savings” adjusts the gross savings for a variety of factors.
Methods of Estimation Program Impacts
• Energy– Engineering
• Simple Engineering• Simulation Models
– Statistical• Simple Pre/Post
(difference of means)
• Regression (Acct for weather)
• Comparison Group (Quasi Experimental Design)
• Detailed Regression
• Demand– Existing Load Factors– Secondary Load Shapes– End-Use Metering– Simulations
• Data Collection– Surveys– Billing Data– Metering– Site Visits
What Do You Measure in Impact Evaluation?
Tracking system gross energy and demand savings
Adjusted gross energy and demand savings
Realization rates Program verified gross
energy and demand savings
4. Program Verified
3. Realized
2. Adjusted Gross
1. Tracking System Gross Energy and Demand Savings
What Do You Measure in Impact Evaluation?
Net-to-Gross or Attribution Factors are typically measured through customer surveys. These factors are then applied to program verified gross savings to estimate the net energy and demand savings attributed to the Program’s activities. Net-to-gross factors include: Free ridership Spillover
BECAUSE these can cancel each other out, some studies do not bother to measure either one!
What Do You Measure in Impact Evaluation?
Economic costs and benefits net energy and demand
savings, program costs, participant costs, societal costs and benefits – all are estimated to calculate benefit-cost ratios for various cost-effectiveness tests
Environmental benefits Sum up electricity savings
(and/or on-site fuel savings, e.g., natural gas)
multiply those savings by emission factors to produce pounds of emissions saved or avoided
emission factors are expressed as pounds of pollutant per MWh of electricity or per therm of natural gas
Approach to Impact Evaluation
A typical impact evaluation requires the following activities or elements: Gather/review program background and savings
information Develop sampling/analysis approach Conduct surveys and interviews Conduct project site data collection Analyze data to develop measure and project level
gross savings Develop gross and net energy and demand savings
by program
Final Comments
Set up the evaluation when you set up the program Get your vendors to do your data tracking, provide
monthly status reports (the program will evaluate itself!)
Make sure your processes are seamless and transparent, so someone else can easily get what they need
Remember this is art, not science!
Carol [email protected]