measuring results from climate change programs: performance indicators for gef

58
Measuring Results from Climate Change Programs Monitoring and Evaluation Working Paper 4 September 2000 Performance Indicators for GEF

Upload: gef-independent-evaluation-office

Post on 24-Feb-2016

216 views

Category:

Documents


0 download

DESCRIPTION

Through its programs in the climate change area, GEF assists developing countries and countries in transition to meet both national development and global climate change objectives by promoting energy efficiency and renewable energy. This promotion takes the form of fostering sound investments and policies, improving national capabilities, and nurturing businesses.

TRANSCRIPT

Page 1: Measuring Results from Climate Change Programs: Performance Indicators for GEF

Measuring Results from Climate Change Programs

Monitoring and Evaluation Working Paper 4September 2000

Performance Indicators for GEF

Page 2: Measuring Results from Climate Change Programs: Performance Indicators for GEF

i

Measuring Results from Climate Change Programs:Performance Indicators for GEF

Monitoring and Evaluation Working Paper 4

September 2000

Page 3: Measuring Results from Climate Change Programs: Performance Indicators for GEF

ii

Published 2000Global Environment Facility

This paper may be reproduced in whole or in part and in any form for educational or nonprofit uses,without special permission, provided acknowledgment of the source is made. The Global EnvironmentFacility secretariat would appreciate receiving a copy of any publication that uses this paper as a source.Copies may be sent to GEF secretariat, 1818 H Street NW, Washington, DC 20433.

No use of this paper may be made for resale or other commercial purpose without prior written consentof the Global Environment Facility secretariat. The designations of geographic entities in this document,and the presentation of materials, do not imply the expression of any opinion whatsoever on the part ofthe GEF concerning the legal status of any country, territory, or area, or its authorities, or concerning thedelimitation of its frontiers or boundaries. The views expressed in this paper are not necessarily those ofthe GEF or its associated agencies.

ISBN 1-884122-86-8ISSN 1020-0894

AuthorsDavid Nichols, Tellus InstituteEric Martinot, Tellus Institute

Study TeamDavid Nichols, Tellus InstituteEric Martinot, Tellus InstituteKeith Kozloff, Haigler-BaillyEdward Vine, Lawrence Berkeley Laboratory

Steering CommitteeDennis Anderson, GEF Scientific and Technical Advisory PanelTom Hamlin, United Nations Environment ProgrammeRichard Hosier, United Nations Development ProgrammeCharles Feinstein, World BankChandrasekhar Sinha, World BankFrank Rittner, GEF Secretariat

Page 4: Measuring Results from Climate Change Programs: Performance Indicators for GEF

iii

Preface

The GEF Monitoring and Evaluation (M&E) team is tasked with analyzing and documenting GEF results.Until now, conclusions of these efforts have been in the form of evaluation and study reports, annual ProjectPerformance Reports, and GEF Lessons Notes. With the introduction of the M&E series of Working Papers,we are publishing reports that are not full-fledged evaluations, but nevertheless deserve attention. Many ofthe issues and early results that these reports identify will be pursued later in broader evaluations to arrive atmore definite conclusions. We expect the M&E working papers to be a valuable catalyst for promoting dia-logue on issues and results of importance within GEF’s operational areas and efforts. We therefore look for-ward to your feedback and suggestions. Please contact us through the coordinates listed below and visit theGEF Web site to find out more about the Monitoring and Evaluation program.

Measuring Results from Climate Change Programs: Performance Indicators for the GEF is the result of anexercise conducted during 1999-2000 to develop a framework for determining the impacts of GEF’s climatechange activities. The exercise was conducted by an external study team, under the guidance of a SteeringCommittee, and in close consultation with GEF management and staff.

We intend to apply performance indicators identified in this report to evaluate the accomplishment of cli-mate change activities supported by the GEF. We also hope that this report will provide guidance to staff inthe GEF and its partner organizations for developing appropriate frameworks for project design, implemen-tation, and performance monitoring.

Jarle HarstadSenior Monitoring and Evaluation Coordinator

GEF Corporate Monitoring and Evaluation Team1818 H Street, NW

Washington, DC 20433, USA

Telephone: (202) 458-2548Fax: (202) 522-3240

E-mail: [email protected]: http://www.gefweb.org

Page 5: Measuring Results from Climate Change Programs: Performance Indicators for GEF

iv

Acknowledgements

The development of climate change performance indicators was carried out by a study team consisting ofDavid Nichols of the Tellus Institute (Boston, Massachusetts), who served as the principal investigator, EricMartinot also of the Tellus Institute, Ed Vine of the Lawrence Berkeley National Laboratory (Berkeley, Cali-fornia), and Keith Kozloff of Hagler-Bailly (Arlington, Virginia). David Nichols and Eric Martinot authoredthis working paper based upon the results and reports produced by the full study team. The study teamworked closely with the Steering Committee and benefited greatly from its advice and guidance.

This exercise was made possible by the excellent cooperation extended by the management and staff at theGEF secretariat and the three implementing agencies, who were willing to spend time with the study teamsharing their perspectives and insights regarding the GEF and is operations. Particular thanks are due toMohamed T. El-Ashry, Kenneth King, Patricia Bliss-Guest, Alan Miller, Dilip Ahuja, Chona Cruz, MichaelSanio, Jarle Harstad, Scott Smith, Lars Vidaeus, Charles Feinstein, Karl Jechoutek, Louis Boorstin, DanaYounger, Rafael Asenjo, Emma Torres, Richard Hosier, Nandita Mongia, Rana Maalouf, Katya Birr, MartinKrause, and Ahmed Djoghlaf.

The study team also benefited from discussions with Claire Parker, Stephen Peake, and Martha Perdomo atthe secretariat for the United Nations Framework Convention on Climate Change. C. Singer from E&Coshared insights regarding performance monitoring of private sector energy projects.

Ramesh RamankuttyTask ManagerDevelopment of Performance Indicators for GEF Climate Change Programs

Page 6: Measuring Results from Climate Change Programs: Performance Indicators for GEF

v

Measuring the Performance of GEF Climate Change Programs......................................................................... 1

Seven Program-Level Indicators ......................................................................................................................... 5

Monitoring and Evaluation Strategies ............................................................................................................... 15

Monitoring and Evaluation Responsibilities and Activities .............................................................................. 25

Annex A. GEF Climate Change Strategies and Programs ................................................................................. 31

Annex B. Indicators Used by Other Agencies ................................................................................................... 33

Annex C. Examples of Performance Indicators ................................................................................................ 39

Bibliography ...................................................................................................................................................... 45

Endnotes ............................................................................................................................................................ 49

List of Tables and Boxes

Table 1. Climate Change Program Objectives in a Logical Framework ............................................................. 3

Table 2. Clustering of GEF Climate Change Projects ......................................................................................... 8

Table 3. Indicators for Solar Home Systems and Rural Energy Services Project Cluster ................................... 9

Table 4. Indicators for Grid-Connected Renewable Energy Project Cluster ..................................................... 10

Table 5. Indicators for Energy-Efficient Product Manufacturing and Markets Project Cluster ........................ 11

Table 6. Indicators for Energy Efficiency Investments in Industry Project Cluster .......................................... 12

Table 7. Project and Program Rating: Illustrative Example .............................................................................. 23

Table 8. Program Indicators and Monitoring Activities in the Logical Framework. ......................................... 28

Table B.1. Terminologies of Different Donor Agencies for Results/Logical Frameworks ............................... 37

Table B.2. DFID Logframe Guide ..................................................................................................................... 38

Table B.3. Performance Indicators Used by a Selection of Other Organizations .............................................. 38

Contents

Page 7: Measuring Results from Climate Change Programs: Performance Indicators for GEF

vi

Box 1. GEF Project-Level and Country-Level Influences on Wind Power Development in India ..................... 6

Box 2. Project-Level and National-Level Indicators for a Solar PV Project in Zimbabwe .............................. 13

Box 3. Cross-Cutting Study: Diffusion of Solar PV Technology ...................................................................... 18

Box 4. Cross-Cutting Study: Development of an ESCO Industry..................................................................... 19

Page 8: Measuring Results from Climate Change Programs: Performance Indicators for GEF

1

Measuring the Performanceof GEF Climate Change Programs

Through its programs in the climate change area,GEF assists developing countries and countries intransition to meet both national development and glo-bal climate change objectives by promoting energyefficiency and renewable energy. This promotiontakes the form of fostering sound investments andpolicies, improving national capabilities, and nurtur-ing businesses. Since 1991, GEF and its implement-ing agencies—the United Nations EnvironmentProgramme, the United Nations DevelopmentProgramme, and the World Bank Group, includingthe Group’s private sector affiliate, the InternationalFinance Corporation—have contributed to the devel-opment and financing of a series of projects thatreflect GEF’s climate change mitigation strategies(see Annex A).

GEF climate change programs are innovative andunique because they reflect integrated strategies toremove barriers and reduce costs for a broad range ofapplications and markets for energy efficiency andrenewable energy technologies. After eight years ofoperation, GEF has amassed a project portfolio of 72such projects in 45 countries, for which it has pro-vided more than US$700 million. So far, perfor-mance indicators have been specified, and a fewevaluations have occurred only for individualprojects. Most projects do a reasonable job of speci-fying project-specific performance indicators duringthe project design and approval process. But theseindicators tend to measure discrete project activitiesor their direct outputs rather than outcomes and theattainment of broader objectives. In addition, the per-formance indicators selected are sometimes not

readily measurable or, if measurable, are lackingdocumentation as to when, how, or indeed whetherthe indicators have been measured.

GEF clearly needs additional performance indicatorsand methods to measure the results of its climatechange programs. GEF must go beyond the resultsfrom individual projects and look at overall programperformance. Good program performance indicatorswould enable GEF to better manage and attain resultsfrom its programs and projects. But what would suchindicators entail and how could they be applied op-erationally?

In 1999, the GEF Monitoring and Evaluation Unit com-missioned a study team to develop program perfor-mance indicators of the results being achieved in theclimate change area. Performance indicators are mea-sures, qualitative or quantitative, used to reflect progresstoward achievement of objectives. Unlike thosesustainability indicators that measure broad physical,economic, energy, and environmental factors at a macrolevel, program performance indicators focus on the de-gree to which a program has achieved its intended re-sults. Program indicators can measure “ends”(achievement of objectives) or “means”(methods toachieve objectives) or a combination at any point alongthe continuum from ends to means.

What are the ends and means in GEF climate changeprojects and programs? The study began with a com-plete review of the GEF project portfolio and strategydocuments. More revealing, however, were inter-views that the team conducted with the GEF chief

Page 9: Measuring Results from Climate Change Programs: Performance Indicators for GEF

2

executive officer (CEO), other senior GEF managers(including the assistant CEO and executive coordina-tors in the three implementing agencies), GEF Cli-mate Change Task Force members, the UnitedNations Framework Convention on Climate Change(UNFCCC) secretariat, and members of the GEF Sci-entific and Technical Advisory Panel. The interviewsaimed to elicit internal stakeholder perspectives onthe kinds of results from GEF’s climate change pro-grams that are appropriate and that are needed formanagement purposes and for reporting to externalstakeholders.

The interviews underscored GEF’s broad aim of em-powering developing countries to incorporate climatechange actions into their development paths morerapidly and effectively. GEF’s programs should en-able governments, the private sector, non-govern-mental organizations (NGOs), and local communitiesto better address climate change. GEF employs multi-faceted strategies and evolving tactics to pursue thiscomprehensive goal. GEF’s direct activities andprojects, however beneficial in themselves, have thebasic objective of catalyzing continuing processesthat help country actors to integrate climate-friendlydynamics into the more immediate and fundamentalstrategies of economic development and poverty alle-viation.

Interview responses emphasized five major types ofresults from GEF climate change strategies:

• Remove market barriers so that the level of mar-ket penetration of sustainable technologies andpractices in given country markets is increasedduring and after GEF-supported market interven-tions. In short, GEF promotes market transforma-tion and technology transfer and diffusion throughbarrier removal

• Build policymakers’ capacity to address challengesranging from meaningful participation in theUNFCCC, to incorporating climate change objec-tives in economic policy, to reformulating specificregulatory, tax, or other economic policies

• Build business infrastructures by triggering addi-tional development aid, public financing, or pri-vate investment, and by demonstrating the busi-ness viability of sustainable energy products andservices

• Add to social reservoirs of both expert and com-munity awareness and knowledge about climatechange issues in general and sustainable energytechnologies in particular, and translate suchawareness into active involvement of non-govern-mental and private sector groups in activities re-lated to climate change

• Demonstrate creative project approaches that pro-mote climate-friendly economic growth, includingimpacts on improved quality of life, by bringing to-gether mixes of government, business, community,and other stakeholders in ways that bridge gaps andcause change. (Demonstration effects may occur atscales from local to global, short term to longer term,and individual to national.)

The interviews suggested that evidence of the re-moval of barriers would, measured over time, consti-tute highly appropriate indicators of GEF’s actualresults. It is seen as critical that GEF projects lead toadditional activities beyond those directly resultingfrom the projects themselves. For example, if in-creases in the market share of renewable energy andenergy efficiency technologies were found to exceedbaseline projections of their market penetration, thiswould mean a more rapid rate of diffusion of sustain-able energy technologies and would be solid evidenceof GEF influence.

Interviews also revealed a diversity of views on theutility of greenhouse gas (GHG) emissions as an indi-cator. Sustainable energy technologies reduce GHGemissions, and GEF requires that GHG emissionsreductions flowing directly from each individualproject be estimated in the project proposal. Manag-ers interviewed generally held that estimating carbonemissions reductions is necessary and useful, yet isnot the sole or major measure of GEF results. It isdifficult to link post-project replication and spillovereffects to rigorous estimates of incremental carbonimpacts attributable to GEF. The study team con-cluded that estimation of carbon impacts may thus bemore important at the level of individual projects thanat the level of measuring climate change programperformance.

During the study, it became clear that there is a strongneed for performance indicators that reflect the mar-ket development objectives of climate change pro-grams and measure changes in markets. There isscant published literature on this subject, and, indeed,

Page 10: Measuring Results from Climate Change Programs: Performance Indicators for GEF

3

there is no single generally accepted academic modelof how markets work. It was clear, therefore, thatGEF would have to be innovative in developing indi-cators to measure market changes.

A logical framework can help in visualizing marketdevelopment (see Table 1; see also Table 8). Projectsconduct market interventions (project outputs),which can lead ultimately, through replication, tosustainable markets in which the full economic po-tential for energy efficiency or renewable energytechnologies is realized. Although performance indi-cators for market interventions are useful for measur-ing project inputs, activities, and direct outputs withinthe boundaries of projects, such indicators do notaddress replication effects. Conversely, indicatorscould also measure the level of untapped economicpotential for climate-friendly technologies or the ag-gregate level of GHG emissions, but it would bedifficult to detect the effects of GEF’s climate changeprogram on such a long-run and macro-level scale.Thus program performance indicators most usefullymeasure the “development objective” level shown inthe logical framework. These indicators go beyonddirect project outputs but still can be plausibly linkedto GEF activities.

The study team also investigated and examined per-formance indicators used by other organizations that

promote sustainable energy development (see AnnexB). Other organizations’ program performance indi-cators vary greatly; moreover, there is a growingnumber of performance indicators used by differentorganizations at different levels. Like GEF project-level indicators, however, most of these performanceindicators measure activities or direct outputs ratherthan outcomes, and some organizations have not evenidentified any outcome indicators. In particular, itwas clear from the survey that institutionalization ofmarket change indicators, and especially market de-velopment indicators, is in its infancy. It was alsosurprising that the development of qualitative capac-ity development and institutional strengthening indi-cators was not as advanced as might be expected.

Drawing from the GEF project portfolio, the inter-views, and indicators from other organizations, thestudy team developed seven core program perfor-mance indicators. These are discussed in the nextsection, along with the monitoring and evaluationactivities needed to assess climate change programimpacts. Three types of monitoring and evaluationactivities are then outlined in the paper’s third sec-tion: (1) cross-cutting program evaluation studies, (2)market studies, and (3) aggregates of project-levelindicators. The final section of the paper providesoperational guidance to GEF on how to apply andmeasure the proposed performance indicators.

Table 1. Climate Change Program Objectives in a Logical Framework

Source: Adapted from Martinot (1998).

Framework Level Objective Indicators and Monitoring Assumptions and Risks

Global objective Reduce CO2 emissions Avoided GHG emissions(Avoided GHG emissions) from energy consumption that result from market

and production development and otherchanges in practices andinfrastructure

Development objective Build markets for energy Market development Changes in products, and(Outcomes as a result of efficiency and renewable indicators measure indirect sales, and investments willproject national market energy technologies project impacts avoid or reduce energyother development production from GHG-impacts) emitting sources

Project outputs (Direct or Technologies installed, Market intervention Project outputs areintended results of enhanced capabilities, new indicators measure project necessary and sufficientprojects) financing services, new outputs and direct impacts to build markets

codes and standards, etc.

Project inputs (Specific Complete specific project Project performance Project activities takenproject activities) activities (e.g., training, monitoring together are sufficient

financing, research, and necessary todemonstration, technical produce project outputsassistance, informationdissemination)

Page 11: Measuring Results from Climate Change Programs: Performance Indicators for GEF

4

Page 12: Measuring Results from Climate Change Programs: Performance Indicators for GEF

5

This section describes seven program-levelindicators, developed through research andconsultation with GEF stakeholders in 1999. Theindicators reflect key GEF climate change strategiesand objectives, particularly the sustainable adoptionand market development of renewable energy, energyefficiency, and other GEF-supported cleantechnologies in all three climate change operationalprograms.

These seven core indicators are:

1. Energy production or savings and installedcapacities

2. Technology cost trajectories3. Business and supporting services development4. Financing availability and mechanisms5. Policy development6. Awareness and understanding of technologies7. Energy consumption, fuel-use patterns, and

impacts on end users.

These indicators are static. GEF program evaluatorswill need to monitor them periodically in order toassess changes over time. The indicators reflect bothbroader trends as well as specific results of GEFprojects; this means that plausible linkages betweenGEF activities and changes in the indicators need tobe established through supporting evaluationactivities (this subject is covered in more detail in thenext section, “Monitoring and EvaluationStrategies”).

Seven Program-Level Indicators

The seven core indicators can be measured at threelevels:

• At the project level, indicators measure a project’sdirect activities and outputs—the project-levelresults for which implementing agencies are di-rectly responsible. These are the types of indica-tors generally put forth in project evaluation andsupervision reports by implementing agencies.1

• At the country level, indicators become “nationalprofiles” showing national technology, market, andpolicy trends for energy efficiency and/or renew-able energy in a specific country. Linkages can beinferred between direct GEF project results andnational trends to show areas of relevance and in-fluence. Usually GEF projects are designed to in-fluence national trends directly; in these cases,country-level indicators are an inherent part of themonitoring and evaluation activities for individualprojects performed by implementing agencies.

• At the international level, the indicators show in-ternational trends and linkages. Linkages can beinferred between direct GEF interventions andinternational trends to show areas of relevance andinfluence (for example, international costs of so-lar thermal power plants or the number of coun-tries with regulatory frameworks that support util-ity power purchases of wind, biomass, or mini-hydro generation), or to show how successfulGEF-supported experiences in one country havecontributed to market or policy development inother countries.

Page 13: Measuring Results from Climate Change Programs: Performance Indicators for GEF

6

All three levels are necessary to establish GEF pro-grammatic results (see also Table 8). Linkages be-tween levels are what make GEF a programmaticentity and not just a funder of a collection of projects.Plausible linkages means analyzing whether the re-sults being achieved in GEF projects are relevant tobroader trends, whether GEF project results are influ-encing or contributing to broader trends, and whetherexperience from GEF projects is being used outsideimmediate project contexts.

For example, evaluators could look at the total in-stalled capacity of wind turbines directly resultingfrom GEF projects, the total capacity installed incountries where GEF has wind projects, and the totalin all GEF client countries and ask: How did thedirectly financed GEF capacity (Indicator #1 at theproject level) influence national capacity installations(Indicator #1 at the country level) and capacity instal-lations in all GEF client countries (Indicator #1 at theinternational level)? Or how did financing modelsdeveloped under the project (Indicator #4 at theproject level) influence the availability of financingfor wind turbines within a given country (Indicator #4at the country level)? See Box 1 for an illustration ofthese questions for the case of India.

Capacity strengthening is employed in virtually allGEF climate change activities. But rather than beinga separate indicator, capacity strengthening is an inte-gral and inherent feature of most of the seven coreindicators. For example, policy development will re-flect increased regulatory capabilities and under-standing of technologies on the part of policymakers.Business and supporting services development willreflect increased capabilities among businesses. Weview capacity (or capability) strengthening as themeans to other objectives in GEF projects and think itis better to use indicators that measure the accom-

plishments resulting from strengthened capacities.Nevertheless, at the project and national levels it maystill be fruitful to employ additional indicators relat-ing to capacity strengthening, depending on the na-ture of the project.

GEF climate change projects have impacts on endusers. Projects may contribute to changes in liveli-hood and lifestyle as a result of a shift in energy usepatterns, or as power is introduced, for example, intoremote and rural villages. Such impacts may be re-flected in changes in social conditions, gender-differ-entiated activities, and (in some cases) in income andliving standards. For example, for some 300 to 400million households worldwide without electricity thatare unlikely to receive grid power in the near future,indicators can measure poverty reduction and othersocial benefits of off-grid decentralized energy fromsolar photovoltaic (PV), small hydropower, and windsystems.

End user impacts are captured in Indicator #7 inparticular, although they may also be reflected in theother indicators. At the project level, indicators mea-sure the direct effects on relevant end users and in-tended project beneficiaries. At the country level,indicators measure broader socioeconomic impacts(for example, increased percentage of householdselectrified). At the international level, internationaltrends in poverty reduction strategies and programsusing renewable energy that target the poorest of thepoor can be tracked.

There is also a potential role for estimated reductionsin GHG emissions as a type of program performanceindicator. The reduction of carbon emissions is ofcourse a GEF goal and a fundamental objective of itsclimate change programs. One reflection of its impor-tance is the fact that, in order for a project to be

Box 1. GEF Project-Level and Country-Level Influences on Wind Power Development in India

In India, GEF support for wind power occurred in parallel with the explosive market growth of the mid-1990sfueled by favorable investment tax policies. By 1998, almost 1,000 MW of wind capacity had been installed inIndia, and dozens of wind turbine manufacturers had emerged. During the 1990s, the World Bank-GEF projectdirectly financed 41 MW of wind turbine installations in India. The project strengthened the capabilities of the IndiaRenewable Energy Development Agency (IREDA) to promote and finance private sector investments, and morethan 270 MW of wind projects were financed through IREDA. The project also promoted the acceptability of windpower among investors and banking institutions. As a result, along with favorable market conditions, manydomestic sources of finance became available for wind power that helped fuel market growth.

Source: Martinot (2000).

Page 14: Measuring Results from Climate Change Programs: Performance Indicators for GEF

7

accepted by GEF, its proponents must make the casethat it will lead to incremental carbon reductions, andindeed must estimate the magnitude of the reductions.A rigorous approach to estimating and verifying car-bon reductions is feasible at the level of project out-puts, but the climate change programs aim forpost-project replication and spillover effects. Carbonreductions from such indirect effects (at national andinternational levels) are much harder to measure andrigorously link to GEF activities. For this reason, wehave not included estimation of incremental carbonimpacts as a core program performance indicator.

It is difficult to imagine measuring these seven indi-cators in the same way across the entire diversity of

projects in the GEF climate change project portfolio.Rather, these seven generic program indicators aremost usefully discussed in the context of specificclusters of projects. For program performance evalu-ation purposes, we find it useful to group GEFprojects into nine clusters of similar projects (seeTable 2).2 Some projects will contain componentsfrom different clusters and so must be evaluated us-ing different sets of indicators.

Tables 3 through 6 show the seven indicators appliedto four different project clusters. Box 2 illustrates theseven indicators at the country level for a solar PVproject in Zimbabwe supported by the United Na-tions Development Programme (UNDP) and GEF.

Page 15: Measuring Results from Climate Change Programs: Performance Indicators for GEF

8

Table 2. Clustering of GEF Climate Change Projects

Cluster Types of Projects

Solar home systems and ruralenergy services

Projects to promote the adoption of off-grid rural energy services using solarhome systems or other village power configurations

Grid-connected renewableenergy

Projects to promote the adoption of grid-connected electricity generation fromwind, small-scale hydropower, biomass, bagasse, or geothermal resources

Solar hot water supply Projects to promote the adoption of solar thermal hot water supply indomestic, public, and commercial applications

Pre-commercial renewableenergy technologies

Projects to reduce long-term costs of selected electric[ity?] generationtechnologies distributed and central station PV power, hybrid solar-thermal orgas power plants, and biomass integrated gasification/gas turbine technology(OP7 projects)

Energy-efficient productmanufacturing and markets

Projects to build markets for efficient lighting, refrigeration, motors, industrialboilers, and other manufactured products by simultaneously helpingmanufacturers adopt, produce, and sell more efficient models and increasingconsumer awareness and demand for more efficient models

Energy efficiency investments inindustry

Projects to reduce energy consumption in industrial processes, either directlyby supporting industrial enterprises to implement a variety of measures suchas efficient motors, efficient boilers, and energy auditing and management; orindirectly through support and promotion of energy service companies

Energy-efficient building codesand construction

Projects to promote energy efficiency in buildings through new constructioncodes and practices, retrofits of existing buildings, energy managementtechniques, and labeling or installation of efficient equipment

District heating energy efficiencyimprovements

Projects to develop the institutional, technical, financial, and skill resources forongoing improvement of centralized and decentralized space and waterheating systems in countries in transition

Fuel switching andproduction/recovery

Short-term response measures promoting switching from high-carbon to lower-carbon fuels (such as from coal to gas), or producing methane from coal bedsor municipal waste

Page 16: Measuring Results from Climate Change Programs: Performance Indicators for GEF

9

Table 3. Indicators for Solar Home Systems and Rural Energy Services Project Cluster

Indicator Examples

1. Energy production or savingsand installed capacities

• Number of individual solar home systems installed

• Capacity of off-grid village power supplies from mini-hydro, biomass, wind,and solar PV(MW)

2. Technology cost trajectories • Installed costs or life cycle system costs of solar home systems

• Unit electricity costs of renewable-energy-produced power relative toconventional power costs (e.g.,from diesel generators)

3. Business and supportingservices development

• Number of solar home system manufacturers, system assemblers, dealers,installers, and service firms (including firms for which solar home systems arenot the primary business line)

• Existence and appropriateness (to local needs) of equipment qualitystandards and certification procedures/institutions for equipment andinstallation

4. Financing availability andmechanisms

• Availability of consumer credit for purchase of solar home systems, includingdealer-supplied credit, microfinance, and credit from development banks

• Number of financial institutions and volume of lending for off-grid villagepower

5. Policy development • Existence of policies and/or plans that explicitly recognize and account forthe role of renewable energy technologies in rural electrification

• Existence of working regulatory/social models for village power schemes,including tariffs, responsibilities for ownership and maintenance, and equity

6. Awareness and understandingof technologies

• Awareness among rural households of benefits and costs of solar homesystems

• Abilities of village leaders or project developers to implement and managevillage power schemes

7. Energy consumption, fuel-usepatterns, and impacts on endusers

• Percentage of off-grid households receiving energy services from renewableenergy sources relative to conventional sources (by income group or othersocial parameters)

• Consumer satisfaction (by income group or other social parameters)

Page 17: Measuring Results from Climate Change Programs: Performance Indicators for GEF

10

Table 4. Indicators for Grid-Connected Renewable Energy Project Cluster

Indicator Examples

1. Energy production or savingsand installed capacities

• Installed capacity (MW) of wind, biomass, geothermal, small-hydro

• Annual or cumulative production (MWh)

2. Technology cost trajectories • Installed costs per kW in selected GEF client countries for each technology

• Levelized production costs (per kWh) in selected GEF client countries foreach technology

3. Business and supportingservices development

• Number of domestic commercial businesses that canmanufacture/assemble, sell/install, and service/maintain technologies

• Existence of quality standards and certification procedures for equipmentand installation

4. Financing availability andmechanisms

• Amounts of government, commercial, and bilateral/multilateral financing forgrid-connected renewable energy generation facilities

• Characteristics of financing programs dedicated to renewable energytechnologies

5. Policy development • Existence and characteristics of electric power regulation and policies forindependent power producers, power purchase agreements, transmissionwheeling of generated power, and power dispatch requirements

• Fairness/equivalence of wholesale tariff structures for generation fromrenewable energy sources relative to those for conventional fuels

6. Awareness and understandingof technologies

• Awareness among financiers, project developers, and utilities

• Knowledge base for technology application (e.g.,wind resource maps, windturbine siting experience, wind farm operations, and maintenance experience)

7. Energy consumption, fuel-usepatterns, and impacts on endusers

• Share of renewable energy generation relative to total generation (% kWh)

• Number of equivalent beneficiaries (households) supplied by renewableenergy generation (same as kWh share but translated into householdequivalents)

Page 18: Measuring Results from Climate Change Programs: Performance Indicators for GEF

11

Table 5. Indicators for Energy-Efficient Product Manufacturing and Markets Project Cluster

Indicator Examples

1. Energy production or savingsand installed capacities

• Cumulative or annual energy savings (MWh) from stream of energy-efficientmanufactured products

• Electric power demand reductions (MW) from stream of energy-efficientmanufactured products

2. Technology cost trajectories • Market price of manufactured product

• Life cycle cost of manufactured product relative to that of conventionalproduct (e.g., compact fluorescent lamps relative to conventional lighting)

3. Business and supportingservices development

• Number of in-country commercial businesses that canmanufacture/assemble, sell/install, and service manufactured product

• Existence of equipment quality standards and certification procedures forservice personnel

4. Financing availability andmechanisms

• Availability of financing to manufacturers that serve target market to upgradeproduct designs and production processes for energy-efficient products

• Amount of relevant investment, if obtainable, by these manufacturers

5. Policy development • Existence of policies or regulations that have a significant influence (+/-) ondemand/supply of manufactured product (efficiency standards, equipmentlabels, environmental emissions limits, import quotas, etc.)

6. Awareness and understandingof technologies

• Consumer or industry awareness of characteristics, costs, and benefits ofmanufactured product

• Mentions of targeted technologies in media

7. Energy consumption, fuel-usepatterns, and impacts on endusers

• Market share of manufactured product (e.g., sales of efficient vs. normalboilers)

• Penetration of manufactured product (e.g., share of households or industrialenterprises with efficient product)

Page 19: Measuring Results from Climate Change Programs: Performance Indicators for GEF

12

Table 6. Indicators for Energy Efficiency Investments in Industry Project Cluster

Indicator Examples

1. Energy production or savingsand installed capacities

• Annual or cumulative energy savings (MWh) from energy efficiencyinvestments in industry, either by industrial firms or by energy servicecompanies

• Electric power capacity (MW) reduced through energy efficiency investments

2. Technology cost trajectories • Rates of return achieved from energy efficiency investments in industry

• Costs of conserved energy (e.g., cents/kWh)

3. Business and supportingservices development

• Number of operating energy service companies or other companies offeringefficiency improvement equipment or services in target market

• Existence of energy service company trade association that establishesstandards of professional practice and measurement

4. Financing availability andmechanisms

• Availability of business financing for energy service companies

• Availability of lease financing and performance contract financing for energy-related in-plant projects

• Volume of financing in energy efficiency investments from energy servicecompanies or by industry

5. Policy development • Existence of regulatory/contracting frameworks that support energy servicecompanies (i.e., policies supporting performance contracting)

• Existence of policies creating incentives for industry to improve energyefficiency

6. Awareness and understandingof technologies

• Awareness within industry of the benefits of energy efficiency investmentsand of potential contracting approaches with energy service companies (i.e.,performance contracting)

• Awareness and capability of energy service companies to make profitableinvestments in industry and sustain a profitable business

7. Energy consumption, fuel-usepatterns, and impacts on endusers

• Energy intensities of particular industrial subsectors, compared with pastyears and baseline projections

Page 20: Measuring Results from Climate Change Programs: Performance Indicators for GEF

13

Box 2. Project-Level and National-Level Indicators for a Solar PV Project in Zimbabwe

A UNDP/GEF project in Zimbabwe from 1995 to 1998 was designed to enhance and upgrade indigenous solarmanufacturing and delivery infrastructure, develop an expanded commercial market in rural areas for affordabledomestic solar electric lighting by providing low-interest financing through existing institutions, and establish newcredit mechanisms at the grassroots level to benefit lower income groups in rural areas. The project has had anumber of impacts on the market for PV systems in Zimbabwe:

1. Energy production or savings and installed capacities. Roughly 10,000 solar home systems were installed byproject completion, 300 of which were provided by NGOs and the rest of which were provided by privatesector dealers. An estimated 3,000 PV systems had been installed prior to the project, mostly in rural andsemi-urban centers such as health clinics, schools, community centers, and commercial farms, so the projecthas greatly increased the installed base. A market baseline, estimated before the project started, projectedsales of 320 home systems per year, which would have equaled roughly 1,600 systems over the five-yearproject.

2. Technology cost trajectories. Import duties of 40 percent on imported components were waived during theproject, which resulted in substantial cost reductions for installed PV systems. Installed costs also declinedduring the project based upon increased dealer experience, competition, and economies of scale.

3. Business and supporting services development. Prior to the project there was one PV module and systemsmanufacturer, and three smaller firms performing installation and system integration. At the completion of theproject, 60 firms were registered with the Solar Energy Industries Association(although only 30 had renewedby 1999, and only 6 accounted for 80 percent of the market share for the project. A code of conduct for theSolar Energy Industries Association was added to the organization’s constitution. Technicians from five in-staller companies were trained during the project. No new module assembly companies have emerged as aresult of the project; this is attributable to lack of funds for plants and machinery. Some project observers haveworried about industry shake-outs after the project if demand slackens.

4. Financing availability and mechanisms. The Zimbabwe Agricultural Finance Corporation (AFC) successfullyprovided low-interest credit to 4,200 PV consumers through a revolving fund mechanism. No data are avail-able on the repayment rates and turnover for the revolving fund, and thus what additional resources will berequired to sustain it. The AFC has been unable to replenish the fund, which will deplete without replenish-ment.

5. Policy development. The project did not target policy development explicitly, although original plans for thenational electric utility to participate in the project could have potentially influenced rural electrification policy.Standards are another potential area of policy impact. The Standards Association of Zimbabwe and the SolarEnergy Industries Association created PV module standards that were used to certify and warranty installedsystems. Standards were being drafted for batteries, lamps, and charge controllers.

6. Awareness and understanding of technologies. Two colleges and one polytechnic institute will begin to pro-vide courses on PV technology initially in the form of adult education courses. The University of Zimbabwebegan to offer a M.Sc degree in renewable energy systems in 1996. The project management unit and theSolar Energy Industries Association have produced a solar magazine. According to one evaluation report,“There is now much greater awareness by government, NGOs, and the public about home PV systems thanbefore the project.”

7. Energy consumption, fuel-use patterns, and impacts on end users. Changes in household energy use pat-terns have not yet been measured. The social impacts of the project, in terms of livelihood support and socialequity, are not well understood. While the project supported the country’s rural poverty alleviation program byproviding the basic energy requirements for investments in social infrastructure (e.g., health centers andschools), further survey work is needed to investigate these impacts.

Source: Martinot and McDoom (2000).

Page 21: Measuring Results from Climate Change Programs: Performance Indicators for GEF

14

Page 22: Measuring Results from Climate Change Programs: Performance Indicators for GEF

15

Monitoring and Evaluation Strategies

Monitoring and evaluation strategies are needed toestablish the values of the indicators, aggregate themin appropriate ways, and measure trends over time.Considered in isolation, indicators alone do not mea-sure performance. Monitoring and evaluation effortsmust also be initiated to collect evidence linking thestate of the indicators to GEF activities. In general,this evidence can be strongest and most direct at thelevel of specific projects. As the focus of assessmentmoves from immediate activities and project outputstoward achievement of project- and program-levelobjectives in national and international markets, perfor-mance indicators become more aggregate in nature.

At a program level it also becomes more difficult todraw causal linkages of national- and international-level changes back to GEF activities. Because of thecomplex and evolving nature of national and interna-tional markets, the number of actors involved, and themultiplicity of influences that shape technology andmarket changes over time, it can be analytically diffi-cult, time-consuming, and expensive to isolate GEF’sinfluence on market development at these levels witha high degree of technical rigor. We have soughtcheaper and more practical solutions. At the least,evaluation research can create documented historicalnarratives that allow readers to make plausible judg-ments about the meaning of performance indicatorsas measures of GEF’s role and influence in marketdevelopment, energy policy, and improvements in thelives of affected populations.

Strategies for measuring the capacity-strengtheningelements of program indicators are perhaps the most

problematic. Capacity strengthening is inherentlyspecific to certain groups or institutions at the projectlevel, and the linkages between groups and the trans-mission of knowledge and capabilities across groupsare difficult to assess. Indeed, capacity strengtheningand social impact objectives are part of many energyprojects conducted by development and environmen-tal agencies around the world, yet our review foundthat few employ output indicators to measure theresults of these activities (as opposed, for example, tosimply documenting how many people or organiza-tions underwent training). In general, efforts to mea-sure capacity strengthening will tend to be relativelycostly in terms of specialized resources and time.

Strategies for measuring social impacts among af-fected populations and direct beneficiaries may re-quire social surveys, ethnographic methods,information-gathering techniques such as focusgroups and expert interviews, consensus building,and targeted stakeholder evaluations such as genderanalysis. If project objectives are focused largely onmarket transformations, as is often the case withGEF-supported climate change projects, social im-pacts are indirect and thus harder to measure thanmarket impacts. At the same time, it is recognizedthat in the case of off-grid renewable energy projects,the beneficiaries are largely poor rural communitieswhere the social impacts are important. Social sur-veys may also measure the degree to which stake-holders were consulted or expressed satisfaction withthe project.

Page 23: Measuring Results from Climate Change Programs: Performance Indicators for GEF

16

Availability of Data and Costs of Measurement

The availability of data and the cost of gathering dataare key issues in measuring performance. By empha-sizing the use of available information and by for-mally incorporating indicators and measurement intothe design of individual projects as described in thenext section (“Monitoring and Evaluation Responsi-bilities and Activities”), the costs of measurement canbe kept low(at least relative to the costs of collectingdata through original research such as field surveys.Nevertheless, the costs of measurement will vary.Sources of information for measurement are indi-cated in general terms and estimates of how measure-ment costs will vary—low, moderate, or high cost, on arelative basis across indicators—are discussed below.

1. Energy production or savings and installed ca-pacities. These data are usually available fromGEF project reports, published government data,or published industry or market reports (low cost).For some project clusters, data may need to beobtained from local unpublished sources or indus-try observers (medium cost). Energy savings fig-ures will be readily available for direct projectoutputs (low cost) but may not be available at allon a national or industrywide basis (high cost), ormay require limited sampling of installations inthe field (medium cost). Energy savings from util-ity demand-side management (DSM) projects us-ing national electric power utilities should bereadily available (low cost). Energy savings fig-ures will generally be unavailable at the interna-tional level.

2. Costs per technology unit or measure installed.For renewable energy projects and programs, thesedata are usually available from GEF project re-ports, published government data, utility data, orpublished industry or market reports (low cost).3

For energy efficiency projects and programs, ratesof return and costs of conserved energy are gen-erally available at the project level only. Energyefficiency project and program data at national andinternational levels are either unavailable or wouldrequire sampled industry surveys—which wouldnot be expected to exist in all cases (medium tohigh cost).

3. Business and supporting services development.The number of businesses or services in a givennational or regional market is usually available

from GEF project reports, local government data,or private sector market surveys (low cost); orthrough limited field surveys (medium cost). In-ternational compilations of these indicators aremore difficult but probably less useful anyway.The condition, capabilities, and profitability ofthose businesses are much more difficult to ascer-tain without individual business surveys (highcost). The exception is where a GEF project tar-gets specific firms (for example, a small numberof model energy service businesses), in which casethe project must collect data that demonstrate theviability of these specific firms (low to mediumcost). The existence of quality standards and cer-tification procedures and institutions is generallyeasy to learn (low cost), but the effectiveness andappropriateness of these standards and proceduresrequires in-depth industry and market analysis(high cost).

4. Financing availability and mechanisms. The num-ber of financing programs and mechanisms dedi-cated to target measures should be available fromgovernment and donor agencies (low cost). Sur-veys of commercial banks as to their lending pat-terns, interest rates, and views of specific tech-nologies, as well as industry views on the avail-ability of financing would require much greaterresources (medium to high cost).

5. Policy development. The existence of policies thathave a significant effect on the market penetra-tion of targeted technologies or measures must bea subjective judgment, based on information inproject reports and reports of local governmentsand development agencies (low cost). The effec-tiveness of these policies (whether they supportor retard market penetration, for example) is muchmore difficult to judge, requiring expert policyanalysis and understanding of policy context (highcost). In some cases, very visible policies (suchas India’s policy experience with investment taxcredits for wind power) have been the subject ofmany published analyses (low cost); the challengein this case is to portray often conflicting conclu-sions in a fair manner.

6. Awareness and understanding of technologies.Awareness of sustainable energy technologiesamong households and firms that might potentiallyacquire or install them can be measured directlythrough field surveys (high cost). Cheaper, proxy

Page 24: Measuring Results from Climate Change Programs: Performance Indicators for GEF

17

approaches can be used to infer awareness, suchas increases in mentions of the target technolo-gies in the media, industry meetings and publica-tions, and other forms of communication (mediumcost). For some clusters of projects, the sets ofpotential users are small, so the costs of measur-ing awareness through targeted interviews shouldbe lower (low to medium cost). Projects with goodmonitoring and evaluation plans will conduct pre-project market surveys that would provide abaseline of awareness, followed by later marketsurveys that would measure awareness.

7. Energy consumption, fuel-use patterns, and enduser impacts. For some project clusters, such asproduct manufacturing (e.g., efficient boilers orcompact fluorescent lamps [CFLs]), market pen-etration is relatively easy to determine throughexisting market or project data (low cost). Utilitydata, usually compiled statistically, make it simi-larly easy to analyze grid-connected renewableenergy generation relative to conventional alter-natives (low cost). But for most project clustersand technologies, these indicators will be relativelyexpensive to determine because extensive analyti-cal work would be needed with large amounts ofdata to collect (medium to high cost). Examplesare changes in energy intensities of particular in-dustries, which need to be measured with respectto those industry sectors that the cluster of projectsin this area have endeavored to affect; or changesin building (or centralized heating system) energyefficiency with the penetration of practices for de-signing and constructing more efficient buildings(or centralized heating systems); or, for fuelswitching and production/recovery, the percent-age of potential fuel switching or methane recov-ery that has been tapped. Impacts on end usersgenerally require new information, e.g., from so-cial surveys.

Cross-Cutting Program Studies

Special cross-cutting program-level evaluations canassist in measuring program performance indicatorsin the near term. In-depth studies can encompassclusters of activities or emerging results whose per-formance can be assessed together. Each evaluationmay need to include substantial data collection com-ponents, because of the lack of readily available dataand the relatively slow growth in the available evalu-ations of individual GEF projects to date. Cross-cut-ting studies must include multiple methodologiesdesigned to capture spillover or learning effects(i.e.,market effects that reflect learning about sustainableenergy technologies in regions outside projects’ im-mediate geographic boundaries(or market transfor-mation effects that persist after projects arecompleted. To allow time for learning effects to dif-fuse, evaluations may need follow-up phases aimed atcapturing them.

Cross-cutting multicountry impact evaluations can becostly in nature if field research and surveys are re-quired. A process of prioritization is needed to decidewhat cross-cutting evaluation studies would be mostuseful. GEF could then scope out any studies electedfor a first round of cross-cutting evaluations. Usefulresults from them could be expected in a two- tothree-year time frame.

The nine clusters set forth above (Table 2) can pro-vide a basis for evaluation studies. For example, thesolar home systems and rural energy services clusterrepresents 20 individual projects (past, existing, andnew). GEF appears to be having a discernable impacton diffusion of this technology in developing coun-tries. Box 3 offers a brief outline for an evaluation ofGEF’s role in PV technology diffusion.

Page 25: Measuring Results from Climate Change Programs: Performance Indicators for GEF

18

There is no single method of clustering to identifysubsets of climate change projects upon which it ismay be useful to focus cross-cutting in-depth studies,and so consideration of potential cross-cutting evalu-ation studies need not be restricted to the nine clustersset out in Table 2. Alternative clusterings could bebased on the type of technology, the primary modal-ity (type of activity/intervention strategy), countriesin which projects occur, the implementing agencyinvolved, the level of GEF financial commitment, theimportance accorded a set of projects, or other crite-ria. Some topics of potential study interest based onalternative clusterings follow.

• Enabling activities related to the UNFCCC. GEF’senabling activities support UNFCCC-related com-munications and studies in many countries throughnumerous relatively small grants and are bestevaluated as a whole.

• Efficient lighting technology diffusion. A varietyof projects in different locations have promotedthese technologies, and a new global initiative—the International Finance Corporation (IFC)-GEFEfficient Lighting Initiative4 —has been launched.

Box 3. Cross-Cutting Study: Diffusion of Solar PV Technology

A cross-cutting study of the market diffusion of off-grid PV systems would focus on the countries in which GEFprojects have occurred or are occurring. It would also include components to investigate broader replication fromGEF projects. Information on the seven core program-level indicators would need to be collected, for example:

• Capacity installed to date and annual electricity supplied• Level and trends in installed total costs of PVs• Number and size of PV vendors• Level of PV activity (value of sales)• Trends in local manufacturing and assembly capacity• Capacity of businesses and individuals to service PVs over time• Availability and terms of financing for PVs• Trends in key policies affecting PV market penetration• Consumer acceptance and understanding of PVs• Impacts on affected populations• Portions of end use demands for electricity and other fuels being met through PVs• Technology mainstreamed in other multilateral programs. In addition, the study would investigate what market

barriers are influencing the further development of the PV industry.

The study could recommend additional projects, if warranted, that would seek to reduce these barriers. The studycould be structured along two different, but complementary, paths: (1) What are the opportunities for additionalPV industry development?, and (2) How could projects focus on innovators for promoting PV industry develop-ment? Data collection would begin with a review of existing documents but would primarily involve interviews withkey stakeholders in countries where PV industries are being developed pursuant to GEF projects.

• Impacts of GEF-supported projects on China’sadoption of sustainable energy techniques. InChina, some of the four GEF-supported projectsare reputed to have had a substantial policy devel-opment and technology transfer impact.

• Creation of financing mechanisms. This refers tothe establishment of funds or mechanisms to fi-nance energy efficiency and/or renewable energyprojects. Some GEF projects have this objectiveas their major thrust—e.g., the Renewable Energyand Energy Efficiency Fund and the earlier SolarDevelopment Corporation—while in others, it isan important project component.

• The ESCO industry. Development of an “ESCO”industry—energy service companies that offerleasing, performance contracting, or other busi-ness arrangements for “win-win” efficiency invest-ments at host facilities—is an objective of at least10 GEF projects.

Box 4 provides an illustrative outline of an evaluationin the last of the additional areas identified, that ofESCO development.

Page 26: Measuring Results from Climate Change Programs: Performance Indicators for GEF

19

Four Key Elements of Project-LevelMonitoring and Evaluation

The selection of measurable performance indicatorsand associated performance targets as part of theproject design process promotes a focus on achiev-able impacts and post-project sustainability. Projectindicator measurement provides a source of feedbackon project effectiveness. The development of goodproject-level indicators and measurement is also inte-grally connected to the task of operationalizing thecore program-level performance indicators (see nextsection, “Monitoring and Evaluation Responsibilitiesand Activities”). Project-level indicator measurementcan provide basic data that can be added to informa-tion from other projects to develop program-levelperformance information. This subsection discusseshow GEF can incorporate more systematic use ofperformance indicators, targets, and impact evalua-tion studies in all of its new projects. Improved moni-toring and evaluation at the project level canstrengthen measurement of performance indicators atall levels.

Below we consider four elements of good project-level indicators: (1) generate production reports ofproject activities and results, (2) select appropriate

Box 4. Cross-Cutting Study: Development of an ESCO Industry

In a cross-cutting study of the ESCO industry, information on key project-level indicators would need to becollected. Types of information such as the following are appropriate:

• Annual energy saved through ESCO contracts• Co-benefits in ESCO projects (waste reduction, pollution prevention, water conservation)• Estimated cost of saved energy in ESCO projects• Number of ESCOs formed• Level of ESCO activity (number and value of contracts with customers)• Business advisory services for facilitating the development of an ESCO industry• Training of ESCOs and NGOs on providing energy efficiency services• Number of business plans developed for ESCOs• Pilot energy performance contracting program (loan guarantees to support performance contracts)• Policies implemented to facilitate performance contracting• Host facility managers awareness of performance contractings structure and benefits.

In addition, the study would focus on what market barriers are confronting the development of an ESCO industryand how these barriers are being overcome (if they are) by GEF-funded projects. The study could recommendadditional projects that would seek to reduce these barriers. The study could be structured along two different, butcomplementary, paths: (1) What are the opportunities for ESCO development?, and (2) How could projects focuson innovators for promoting ESCO development? Data collection would begin with a review of existing docu-ments, but would primarily involve interviews with key stakeholders in countries where ESCOs are beingdeveloped pursuant to GEF projects.

project performance indicators, (3) developbaselines and targets, and (4) conduct project impactevaluations.

Generate Production Reports on Direct ProjectResults

An assembly of data on key project activities anddirect results—what we call a production report—can increase understanding of what GEF is doing andallow observers to make informed subjective assess-ments of the value of, and likely outcomes from, theactivities reported.5 When a project is under way, itsmanagers should track all basic project activities andresults in a straightforward way and do so in quantita-tive terms to the extent possible. Continuous activitytracking provides a tool for monitoring implementa-tion as well as for reporting to project sponsors. Acommon tracking and reporting format should beused across all GEF projects. Data should be trackedon an annual basis. A production report can be orga-nized by the seven indicators discussed in the previ-ous section:

1. Energy production or savings and installed capaci-ties (i.e., the electric capacity or capacity savingsof the measures installed; the energy production

Page 27: Measuring Results from Climate Change Programs: Performance Indicators for GEF

20

or energy savings of measures installed; and/orthe number of technologies/measures sold, fi-nanced, or directly installed through the project)

2. Technology cost trajectories (the costs of measuresdirectly installed through the project)

3. Business and supporting services development(number of businesses supported and number ofpersonnel receiving training)

4. Financing availability and mechanisms (subprojectfinancing committed or dispersed)

5. Policy development (agencies created or policiesdeveloped as a direct result of project activity)

6. Awareness and understanding of technologies(number of participants with increased awarenessand understanding, by type of participant, such asenergy end users, energy-related businesses, andNGOs)

7. Energy consumption and fuel-use patterns and shares(for directly supported project beneficiaries).

Other direct project activities can be tracked that donot necessarily correspond to the seven performanceindicators. For example, the flow of dollars into theproject by source, or the use of dollars by the projectby application, may be useful to track. The number ofparticipants in different project activities may be an-other useful indicator to track that does not necessar-ily correspond to one of the seven indicators.

Such activity tracking can be useful for assessingGEF climate change performance at the programlevel. First, it institutionalizes the collection of datathat can be used in the project impact evaluationprocess, which in turn feeds into program perfor-mance indicator measurement. Second, it providesthe basis for a simple aggregation of all the activitythat GEF catalyzes through its climate change pro-grams. Despite the emphasis in this report on perfor-mance indicators related to outcomes, simplysummarizing all climate change activities can itselfbe a useful supplementary form of performance re-porting. Moreover, it could be accomplished in arelatively short time.

Select Appropriate Project PerformanceIndicators

Each new GEF project should include a set of perfor-mance indicators that can be measured over time toreflect the project’s outputs and its progress in attain-ing objectives. At the current time, in providing infor-mation for the annual Project Implementation Review(PIR), projects report their degree of success inimplementation and in achieving impacts withoutsystematically linking these assessments of successto performance indicators. Each project should selectmeasurable indicators that reflect the seven core indi-cators at the project level. Insofar as possible, theunits of measurement employed should be those setforth above. Additional or supplementary units ofmeasurement may be appropriate for a particularproject.

In order to have a coherent basis for improved pro-gram performance measurement over time, eachproject should consider the applicability of the sevencore indicators and use them at the project level to theextent possible. These project-level performance in-dicators must reflect each project’s own objectives.The stakeholders in a project must agree at the outsetas to what measurable performance indicators arelogically related to the objectives they have agreed topursue through project activity.

Indicators might include a mix of output and outcomemeasures. Output measures can also provide leadingindicators of project outcomes (i.e., results relating tothe broader objectives of the project); such measurescan describe early effects that are measurable beforethe project begins to achieve its outcomes.

Indicators must be specified with reference to a par-ticular market. The reference market definition hastwo components: (1) geographic scope, defined as thecountry (or other region) within which the project isexpected to promote market development; and (2)market segment, defined as the energy end use orsupply sector within which market development isexpected to occur. The project clusters described inthe previous section group climate change projectsinto nine market segments.

Page 28: Measuring Results from Climate Change Programs: Performance Indicators for GEF

21

For each individual indicator, the following arerequired:

• An operational definition of the indicator, i.e., howit is measured

• Target values of the indicator for each year, fromthe first (or next) full year of project implementa-tion through the end of the time period for mea-surement

• Identification of the entity and staff person respon-sible for measuring and recording actual valuesof the indicator for each year throughout the mea-surement period

• Identification of the means to be used to measurethe indicator.

Key assumptions made in defining the indicators,developing target values, assigning measurement re-sponsibility, and identifying means of measurementshould be set forth explicitly. Assumptions are keyconditions under which indicators can be used asplanned. Should these conditions change signifi-cantly after project indicators and indicator measure-ments are implemented, one or more aspects ofindicator use and measurement may need to bechanged.6

Indicator selection is driven by the objectives of theproject. Each project has a unique mix of immediateobjectives, reflected in the market interventions andplanned outputs included in the project design. Theseneed to be reflected in measurable, project-specificindicators that can help project stakeholders recog-nize the degree to which these objectives are beingachieved. Each project also has what might be termedintermediate objectives, i.e., planned outcomes in-volving replication and market development. Objec-tives at this level will have elements of uniqueness aswell as elements in common with many other climatechange projects. One of the purposes of the coreindicators developed in this paper is to encourageproject developers to reflect market development ob-jectives that are shared with other non-GEF projectsand programs through common indicators.

Annex C presents a range of possible projectindicators.

Establish Baselines and Targets

Performance indicators can and probably shouldfunction as targets. Each new GEF project should set,ex ante, the schedule of expected values for its se-lected output and outcome indicators. Target valuesfor output indicators should be set out by year for theexpected duration of the project. Consistent withGEF’s strategic thrust toward long-run priorities, tar-get values for outcome indicators could extend be-yond the project time period, covering a mid-termtime frame (for example, the period through the year2030. Performance targets might also be establishedby existing projects that still have several years ofactivity ahead of them.

For projects whose objectives include market trans-formation (barrier removal), two kinds of baselineassessments are needed to select indicators and tar-gets. These are:

• Baseline market characterization

• Baseline market development forecast.

The baseline market characterization applies to thereference market to be addressed by anticipatedproject outputs and outcomes. The reference marketin turn refers to the energy end use or supply sectorwithin the country (or other region) in which theproject is expected to have a market transformativeinfluence. Baselines establish the existing pattern ofenergy technology choices and the political, eco-nomic, and social factors that appear to be linked tothis pattern. Market barriers that the project wouldattempt to overcome would be identified. Thisbaseline information is an important input to the de-sign of a sustainable energy project.

Also needed is a projection of how the referencemarket is expected to evolve absent programmaticintervention. In particular, the expected market pen-etration of the sustainable energy technologies of in-terest should be forecast under the assumption thatthere will be no project. Several methods could beapplied in making such a baseline forecast. For ex-ample, expert judgments could be collected aboutlikely trends from market observers, historical datacould be examined and a simple time trend devel-oped, or an energy forecasting (supply/demand)model could be applied to develop expected trends.

Page 29: Measuring Results from Climate Change Programs: Performance Indicators for GEF

22

Baseline forecasts can inform the setting of targetvalues for performance indicators—most impor-tantly, outcome indicators. Thus, the forecast periodshould substantially exceed the project time frame.Performance targets would then be set to exceed cur-rent expectations as reflected in baseline forecasts.Baseline forecasts can also simplify later evaluationstudies, providing a supplement to, or even an alter-native to, data-intensive ex post methods of inferringincremental impacts from GEF’s activities.7

Conduct Project Impact Evaluations

A key function of monitoring and evaluation is todetermine whether and to what extent anticipatedproject results and effects do in fact flow. Systematicuse of performance indicators, targets, and projectimpact evaluation studies in full GEF projects cansupport objectively based assessment and aggrega-tion of the results from projects. By evaluation stud-ies, we mean formal research designed to collect andanalyze data bearing on the project’s results at theoutput level and, especially, at the outcome level( i.e.,results related to the achievement of the project’sobjectives. Such studies are needed to estimate theactual values of those performance indicators thatcannot simply be tracked by project sponsors andpartners. Formal evaluation should be scheduled tooccur at least (1) by the completion of a project and(2) several years post-project.

Evaluation activities may or may not endeavor toestablish strict ex post facto causality. Some evalua-tion activities attempt to establish causality, for ex-ample through quantitative analysis of data to verifyhypotheses about caused effects within acceptablestatistical parameters. Other monitoring and evalua-tion activities use information in a less rigorousframework, assessing qualitative information to de-termine subjectively the degree to which it is consis-tent with the causal hypotheses inherent in the logicalframework (logframe). The latter approach will fre-quently be appropriate to GEF’s purposes. Multiplequantitative and qualitative methods may be used inevaluation. The intent here is not to urge specifictypes of evaluation methods, but rather to recommendthe consideration and incorporation of an appropriatemix of formal evaluation activities into every fullproject’s design and budget, and to recommend thatevaluations (among other tools) be used to determinethe values of performance target indicators identifiedat project outset.

Project evaluation plans are included in newer GEFproject proposals with increasing frequency. An ex-plicitly broken out evaluation budget should be partof every full project proposal (as well as of proposalsfor other activities that are costly or that are expectedto have large impacts).

Scoring or Rating Performance

Finally we address GEF’s need to assess how wellindividual projects are performing relative to one an-other. To assist in this assessment, project achieve-ments relative to performance indicator targets maybe scored using a simple system. At the outset, eachproject would identify the performance indicators onwhich it would propose its near-term and longer termsuccess to be judged. Target values for those indica-tors would be set as discussed above. Attaining orexceeding the target value of a chosen indicatorwould warrant a “1,” while failing to attain the tar-geted value would be scored “0.” The average valuefor all indicators committed to at the point of projectapproval would be calculated, then multiplied by 100percent to create a project impact score. This wouldmap onto the project ratings GEF currently requestsfrom implementing agencies—Highly Satisfactory(75%+), Satisfactory (50-75%), Unsatisfactory (25-50%), and Highly Unsatisfactory (<25%).

Each project would then have a score based on veri-fied results as soon as performance indicator valuesbegan to be measured (see Table 7). Measurementwould begin at once for some indicators, while forothers it would be delayed, in accordance with themonitoring and evaluation plan of the project. Anillustration is provided on the next page; this is ahypothetical climate change program consisting ofthree projects. The example assumes the scoring sys-tem is used for the impact ratings GEF collects.8 Inthe example, program impacts are “satisfactory” atboth Year 4 (53 percent) and Year 8 (67 percent). (Inactual implementation, measurement of at least someindicators would occur in each year, not just in Years4 and 8.) Projects would likely use more than thenumber of indicators assumed in the example. Usingsuch a scoring system, the average score of anygrouping of projects that were of interest could bedetermined. Similarly, the portfolio of efficiency, re-newable, or all climate change programs could bescored.

Page 30: Measuring Results from Climate Change Programs: Performance Indicators for GEF

23

Table 7. Project and Program Rating: Illustrative Example

Ratings: 0 = Below target; 1 = Greater than or equal to target

Note: For installed price indicator, ratings are switched: less than or equal to target = 1; above target = 0.

Project and Indicators Unit of Measurement Target Measured Rating

@4 yrs. @8 yrs. @4 yrs. @8 yrs. @4 yrs. @8 yrs.

Rooftop PVs in Islandia

Financing availability Number of lenders 1 2 1 3 1 1

Businesses in market Number of vendors 5 10 6 12 1 1

Installed capacity Total kW 5,000 10,000 5,050 11,000 1 1

Installed price Real price in x 115 100 120 95 0 1

Consumer awareness Unit to be specified 30 40 30 45 1 1

Total project rating Score 0.8 1

Percent 80% 100%

Efficient Lightingin Centralia

Financing availability Number of lenders 5 7 4 7 0 1

Businesses in market Number of vendors 20 40 20 35 1 0

Market share Percent of total sales 30 60 60 85 1 1

Installed price Real price in x 20 10 25 15 0 0

Institutional DSM dept. established 1 1 1 0 1 0development

Total project rating Score 0.6 0.4

Percent 60% 40%

Wind Power inMountainia

Financing availability Number of lenders 2 4 0 1 0 0

Businesses in market Number of vendors 4 8 2 3 0 0

Market share Number of systems 500 1,500 200 1,500 0 1

Installed price Real price in x 800 500 1,100 500 0 1

Regulatory reform Tariffs restructured 1 1 1 1 1 1

Total project rating Score 0.2 0.6

Percent 20% 60%

Total program rating Score 0.5 0.7

Percent 53% 67%

Page 31: Measuring Results from Climate Change Programs: Performance Indicators for GEF

24

Page 32: Measuring Results from Climate Change Programs: Performance Indicators for GEF

25

Monitoring and Evaluation Responsibilitiesand Activities

In the previous section, we addressed monitoring andevaluation from a functional viewpoint, describingways to implement a range of monitoring and evalua-tion activities, including use of performance indica-tors. In this section, we address the same issues froma structural viewpoint, suggesting steps that differentagencies associated with GEF might take to imple-ment and integrate these activities. Measurement andsynthesis of program performance indicators impliesjoint efforts by GEF’s Monitoring and EvaluationUnit, implementing agencies, executing agencies,government counterparts, consultants, and other par-ties. Some of the information needed to compile andaggregate indicator measurements can be provided byongoing project monitoring and evaluation activitiesbuilt into individual projects and by the annualProject Performance Review (PPR) process. Otherinformation can be gleaned from special evaluationstudies conducted for individual projects or clustersof projects.

It is useful to place the proposed performance indica-tors and performance assessment process in an ex-plicit framework describing the logic of the climatechange programs. Such a logical framework can linkobjectives, activities, immediate results, and longerterm or more fundamental results in a hierarchy de-scriptive of expected causal chains and effects. Table8 puts the seven indicators used at the project, na-tional, and international levels in a logical frame-work. It also addresses the means of verification andpotential responsibilities and activities by GEF,implementing agencies, and governments.

GEF Secretariat

The GEF secretariat should develop additional re-quirements regarding the monitoring and evaluationelements to be included in GEF projects. These re-quirements should be simple and explicit, building onexisting elements of the GEF project cycle. For thedesign and approval stage, the following require-ments should be specified by the GEF secretariat:

1. Identify the categories of data that should be in-cluded in a common program-level tracking andreporting format across all projects in the portfo-lio. (See “Generate Production Reports of DirectProject Results” subsection.)

2. Help implementing agencies identify project-levelperformance indicators that will be used by theproject. This would include measures of each of theseven core indicators as well as unique indicatorsspecific to the project. (See “Select AppropriateProject Performance Indicators” subsection.)

3. Elaborate the methods to be used to measure in-dicators. (See “Select Appropriate Project Perfor-mance Indicators” subsection.)

4. Establish multiyear target values of those indica-tors whose measurement will be used to assessprogram performance. (See “Establish Baselinesand Targets” subsection.)

5. Conduct programmatic impact evaluation studies,drawing on individual project evaluations, that

Page 33: Measuring Results from Climate Change Programs: Performance Indicators for GEF

26

inform project design and strategy. (See “ConductProject Impact Evaluations” subsection.)

During and after project implementation, the resultsof ongoing monitoring and evaluation are used by theimplementing agencies to assess project progress,and are also reported to GEF. A logical frameworkfor assembling and integrating performance informa-tion is the annual PPR. The GEF secretariat should, asnecessary, extend the guidelines for information forthe PPR so that the results from the procedures listedabove are assembled by each implementing agencyand provided to GEF.

To implement enhanced monitoring and evaluationincluding performance indicator measurement, theGEF secretariat has direct responsibility for integrat-ing, interpreting, and reporting information on theoverall performance of the climate change programs.Relevant activities may include:

• Synthesizing performance data from individualprojects into overall performance studies

• Conducting periodic thematic reviews by projectcluster

• Evaluating relevant technology, market, and policytrends in GEF client countries

• Evaluating program performance against nationaland international trends

• Drawing plausible linkages between direct GEFresults and national and international trends

• Aggregating project results into production reportsas one measure of program performance

• Providing guidance to implementing agencieson monitoring and evaluation practices and re-quirements.

Implementing and Executing Agencies

Project performance indicators are primarily mea-sured in the context of overall project monitoring andevaluation. Process evaluations during projects, andimpact evaluation studies conducted once a criticalmass of project activity has occurred, should be stan-dard practice. The selection of measurable perfor-mance indicators and associated performance targets

as part of the project design process promotes goodproject design practices. It fosters a focus on achiev-able impacts and on post-project sustainability. Onceimplemented, project-level indicator measurementprovides a source of feedback on project effective-ness. Finally, good project-level indicators and mea-surement provide basic data that can be added to andcombined with information from similar projects todevelop program-level performance information.

The following types of information should be col-lected and reported by implementing and project ex-ecuting agencies to the GEF secretariat:

• Baseline information—relevant technology, mar-ket, and policy trends

• Measurement of the seven indicators at the projectlevel

• Measurement of the seven indicators at the coun-try level (i.e., for projects that target country-levelimpacts)9

• Evidence of replication or linkages betweenproject results and broader trends

• Other country-specific market trends not capturedin information above, but relevant to projectsustainability.

There are five primary vehicles through which agen-cies can make their reports:

• In the text of project briefs and project documentsduring project development

• As supplementary reports during project imple-mentation (usually mid-term reports)

• As part of the annual PPR

• As a final report upon project completion

• As a follow-up report one or two years after projectcompletion.

Project briefs in particular should include baselinecharacterizations of existing conditions and relevanttechnology, market, and policy trends. Project de-signs should identify the linkages between thisbaseline information and the project’s elements, as

Page 34: Measuring Results from Climate Change Programs: Performance Indicators for GEF

27

well as the selection of target values for the keyperformance indicators. (Again, see “EstablishBaselines and Targets” subsection.)

Each implementing agency’s responsibility forproject evaluations should be the same as it is forother aspects of its projects. Given the emphasis onmarket transformation and other sustainable and rep-licable results, an evaluation a few years after eachproject’s conclusion should be budgeted for, with thefunds escrowed in some fashion until the evaluationis performed. Post-completion evaluation should be

designed to answer questions about the degree towhich market and institutional development has oc-curred, as evidenced by the degree to which markettransformation has occurred in the project’s targetmarket. Ideally, project evaluation studies shouldusually be done by parties other than those directlyimplementing the project. Project managers maytherefore elect to use local consultants to help withthese evaluations. In addition, executing and imple-menting agencies should provide the GEF secretariatwith any available market studies or other data thatprovide measurements of the seven indicators.

Page 35: Measuring Results from Climate Change Programs: Performance Indicators for GEF

28

Table 8. Program Indicators and Monitoring Activities in the Logical Framework

Level Indicators Means of Verification/Measurement Risks and Assumptions

ProgramObjective

OP5

OP6

OP7

A. International-level indicators

1. Total capacity installed

2. International technology costs

3. Involvement and activities of keymultinational corporations (MNCbusiness mainstreaming)

4. Overall portfolios/activities ofinternational private financiers, MDBsand bilateral aid agencies

5. International policies (e.g., of WorldTrade Organization, Kyoto CleanDevelopment Mechanism, exportcredits)

6. Awareness and understandingamong international agencies andNGOs

7. Energy consumption and fuel-usepatterns (e.g., industrial energy intensity,household fuel use)

B. Aggregate of country-levelindicators for specific groups ofcountries

1. Capacity installed

2. Costs per measure

3. Business development andsupporting services

4. Financing services for end users

5. Country policies (e.g., energy tariffs,import duties, FDI [spell out]policies,taxation, research and development)

6. Awareness and understandingamong users, policymakers, and otherstakeholders

7. Energy consumption and fuel-usepatterns

By GEF secretariat, implementing agencies,government counterparts, or other monitoringand evaluation agencies, generally throughspecial studies (i.e., market surveys) beyondproject reports:

• Define the market scope for each specifictechnology/cluster and therelevance/meaning of the internationalmarket compared to national markets (OP7likely to be different from OP5 and OP6)

• Establish baselines for international marketand measure trends over time

• Seek partners to create a multipartyprogram of studies to measure internationalmarkets across all countries in an ongoingfashion

• Aggregate and synthesize national marketindicators (from next lower level) to obtainprogram-level performance

• Collect evidence that changes in nationalmarkets (and even direct projectinterventions) are influencing broader trendsand activities in the international market(replication)

• Relate market trends to expectedreplication from program objectives (e.g.,mainstreaming)

• Collect evidence that project outputs andoutcomes are influencing marketdevelopment trends and related activities inother countries

• Conduct cross-cutting studies for keytechnologies/clusters over the next 2-3 yearsand thereafter (e.g., solar home systems,ESCOs, investment funds)

Replication. Replication occursacross countries and fromnational markets to internationalmarkets

Leverage. GEF activities haveleveraged financing from othersources Relevance. The mostappropriate technologies,markets, and countries havebeen included in the portfolioGHG emissions. Changes inmarket indicators correlate withreductions in GHG emissionsover time

Causality. GEFs influence canbe discerned among overalltrends and the influence of otheragencies

Complementarity. GEF activitiescomplement those of otherdonor agencies

Page 36: Measuring Results from Climate Change Programs: Performance Indicators for GEF

29

Level Indicators Means of Verification/Measurement Risks and Assumptions

ProjectOutcomes(ProgramOutputs)

Country-level indicators for a specificcountry (same as seven indicatorsunder part B above)

By implementing agency as part of projectpreparation, implementation, and evaluation;project reports are usual source of data, butmay need to be supplemented with fieldstudies:

• Define the national market scope Establishnational market baselines and relate trendsto expected or targeted replication in projectdesigns

• Collect evidence that project interventionsare influencing market development trendsand related activities at the national level

• Measure trends and changes over time(such as annually, post-project, mediumterm, long term; may require special marketsurveys)

Causality. Trends and changesover time in national marketscan be linked to GEF projectinterventions

Relevance. National marketchanges support GEF programobjectives

Timing and resources. Nationalmarket changes can beobserved in a meaningful timeframe, and monitoring andevaluation resources andagencies exist to measurechanges after projectcompletion

ProjectOutputs

Project-level indicators of installedcapacity, costs, skills, etc., directlyinstalled, financed, strengthened, orrelated to GEF project interventions(these are generally restricted to aspecific group of organizations andregions within a country)

By implementing agency as part of projectpreparation, implementation, and evaluation:

• Conduct project-level evaluations of directproject outputs

• Conduct project performance scoring (iftargets are set for new or ongoing projects)

• Conduct selective field inquiries to fill gaps

Project outputs influencenational markets throughexpected replicationmechanisms

Project outputs are necessaryand sufficient to achieve projectoutcomes

Table 8. Program Indicators and Monitoring Activities in the Logical Framework (continued)

Page 37: Measuring Results from Climate Change Programs: Performance Indicators for GEF

30

Page 38: Measuring Results from Climate Change Programs: Performance Indicators for GEF

31

Annex A. GEF Climate Change Strategiesand Programs

The activities that GEF finances conform to guidancefrom the Conference of the Parties (COP) to theUNFCCC. In 1995, the COP endorsed orienting GEFactivities to a mix of short- and long-term programpriorities, and agreed with GEF that, over the longerrun, projects reflecting long-term priorities wouldhave the greatest impact as these “would drive downcosts, build capacity, and start to put in place thetechnologies that can ultimately avoid (rather thanmerely reduce) greenhouse gas emissions—such asfossil-fuel-free technologies in the energy sector”(GEF 1995, p. 5).

GEF priorities are set forth in its Operational Strat-egy, which states that:

Long-term measures will constitute the largestshare of the GEF climate change portfolio, withenabling activities in support of national com-munications a relatively small and decliningshare. Short-term mitigation projects will con-stitute only a small share of the portfolio, inorder to maintain the operational emphasis onlong-term measures (GEF 1996a, p. 32).

When GEF adopted its operational strategy it alsodesigned three long-term operational programs de-signed to promote energy efficiency and renewableenergy by removing barriers, reducing implementa-tion costs, and reducing long-term technology costs(GEF 1997a). A significant goal of these programs isto catalyze sustainable markets in the long term andenable the private sector to finance and diffuse tech-nologies. The operational programs for climatechange are as follows.

Operational Program 5: Removal of Barriers toEnergy Conservation and Energy Efficiency. Thisprogram is designed to remove market barriers to thelarge-scale application and dissemination of least-economic-cost, commercially established, or newlydeveloped energy-efficient technologies and to pro-mote more efficient energy use where a reduction inGHG emissions will result.

Operational Program 6: Promoting the Adoptionof Renewable Energy by Removing Barriers andReducing Implementation Costs. This program isdesigned to remove market barriers to the use ofcommercial and near-commercial renewable energytechnologies, as well as reduce additional implemen-tation costs for renewable energy technologies thatresult from lack of practical experience, initiallow-volume markets, or the dispersed nature ofapplications.

Operational Program 7: Reduction of the Long-Term Costs of Low-Greenhouse-Gas-EmittingEnergy Technologies. This program is designed toaccelerate technological development and increasethe market share of low-GHG-emitting technologiesthat have not yet become commercial lower cost al-ternatives to fossil fuels but that show the potential tobecome so.

The programs are referred to in this paper as OP5,OP6, and OP7, respectively. The goals of the first twoprograms are similar. Both aim to reduce long-termcarbon dioxide or equivalent emissions associatedwith energy consumption and production by promot-

Page 39: Measuring Results from Climate Change Programs: Performance Indicators for GEF

32

ing increased use of commercial or near-commercialtechnologies, removing barriers to market-orientedtransactions and policies, and catalyzing public andprivate sector investments in profitable mitigationprojects. Although the two programs address differ-ent technologies that often face different specific bar-riers, both have long-term market transformationobjectives.

The goal of OP7 is cost reduction. GEF expects thattechnological learning and economies of scale (alsocalled cost buy down), achieved at least in partthrough GEF projects, will reduce long-term costs tocommercially competitive levels. For many technolo-gies in this program, the buy down process will takeyears or even decades; GEF’s goal is to acceleratethis process. Some technologies in this program arerenewable energy technologies, while others are tran-sitional technologies that reduce GHG emissions.The technologies promoted by the three operationalprograms may be referred to collectively as climate-friendly energy technologies.

Additional programs for sustainable transport and in-tegrated ecosystem management related to climatechange are also being developed.

GEF’s three long-term operational programs in theclimate change area are designed to promote energyefficiency and renewable energy by removing barri-ers, reducing implementation costs, and reducinglong-term technology costs. Programs are designed tobuild sustainable commercial markets, leverage fi-nancing from public and private sources, and facili-tate technology diffusion. With its limited resources,GEF cannot significantly affect GHG emissions inthe short term; rather, GEF promotes the develop-ment and use of technologies that are critical foraddressing the climate change problem in the longterm.

GEF climate change projects represent an emergingbody of experimental, case-oriented information oninnovative approaches to promoting energy effi-ciency and renewable energy technologies in devel-oping countries and countries in transition. The firstGEF Assembly in 1998 agreed that “GEF should

remain a facility at the cutting edge, innovative, flex-ible and responsive to the needs of its recipient coun-tries, as well as a catalyst for other institutions andefforts” (GEF 1998d, p. 9). Indeed, GEF projects areoften the first of their kind in the countries where theyoccur (Martinot and McDoom 2000).

Individual project proposals are developed by coun-try governments and experts. Proposed projects arethen reviewed by the GEF Council, GEF’s Scientificand Technical Advisory Panel, GEF secretariat staff,all three implementing agencies, and the UNFCCCsecretariat. The many types of organizations imple-menting these projects in the field include nationaland local governments, private sector entities, NGOs,and community organizations. The range of projectbeneficiaries, cofinancing sources, and other stake-holders is equally diverse.

Note that GEF assists countries with some climatechange activities that are not core projects of theoperational programs—activities such as developingGHG inventories, building national planning capaci-ties, communicating with the UNFCCC, and imple-menting small projects that realize immediateenvironmental benefits as opposed to longer termones. This paper does not address such activities, forit focuses on the cumulative impacts of programs andprojects aimed at achieving longer term resultsthrough market transformation.

Replication of direct impacts to produce indirect im-pacts is integral to GEF climate change strategies.Replication occurs when market actors learn thingsfrom projects that lead to market changes such asincreased consumer acceptance and demand for atechnology or more domestic firms or joint venturessupplying a technology. Replication may occur fromlocal to national markets, from one private sector firmto others, from one local government to another, andfrom one country to another. Although mechanismsto promote dissemination of accomplishments can beincorporated into projects, replication ultimately de-pends on the actions of governments, consumers,NGOs, and/or the private sector after a project iscompleted.

Page 40: Measuring Results from Climate Change Programs: Performance Indicators for GEF

33

Annex B. Indicators Usedby Other Organizations

The performance indicator study team examined in-formation on indicators from a variety of develop-ment agencies, other governmental energy/environment agencies, and NGOs, as well as the in-formation on indicators compiled by the consultantteam investigating performance indicators for GEF’sBiodiversity Programs in 1999. The indicators forselected projects of these organizations were summa-rized in an internal report to GEF.

Monitoring and evaluation activities are conducted atsome level by most organizations engaged in pro-grammatic activities designed to influence levels ofcarbon accumulation in the atmosphere, whether di-rectly through energy market interventions, or moreindirectly, such as through capacity building and in-stitutional strengthening related to energy and cli-mate activities. Some of these organizations employexplicit logical frameworks (logframes). Logframeslink objectives, activities, immediate results, andlonger term or more fundamental results in a hierar-chy that describes logically expected causal chainsand effects.

Most of the logframes that have been developed pre-sume an already developed project—specifically,project inputs and the activities they support—thenposit chains of results from activities to outputs, fromoutputs to project objectives, and from project objec-tives to ultimate goal. Rugh (1998) has prepared atable comparing several logframes, reproduced inTable B.1. Table B.2 gives another example, one usedby the former UK Overseas Development Adminis-tration (now DFID).

Not all agencies use explicit logframes. For example,in North American domestic energy projects con-ducted by governments and utilities, there is exten-sive and varied experience with performanceindicators in the context of reporting, monitoring, andevaluation. In this venue, however, explicitoverarching strategic logical frameworks have beenthe exception rather the rule. Project logic and thelogic of performance measurement have, up to now,been mostly worked out on a case-by-case basis. Im-plicit logframes can be inferred from the total contextof policy and program design. On the other hand,large organizations with several goals and programareas may use several strategic frameworks and setsof indicators. Such is the case, for example, at theU.S. Agency for International Development(USAID).

The organizations and types of indicators examinedin most depth by the study team are summarized inTable B.3. Points from the survey of indicatorsemerge at four levels:

1. Inputs and activities. These are indicators that aremeasures of the project’s inputs and the directactivities involved in its implementation.

2. Outputs. These are indicators that move beyondproject boundaries to measure the immediate re-sults of or outputs from the project’s activities.They are intervention indicators.

3. Intermediate outcomes. These are indicators thatseek to measure the extent to which the project’sobjectives or purposes have been attained.

Page 41: Measuring Results from Climate Change Programs: Performance Indicators for GEF

34

4. End outcomes. These are indicators that measurethe project’s contribution to attainment of theorganization’s strategic objectives or overall goal.

Activity Indicators

Activity indicators measure performance within theboundaries of project activity and are the moststraightforward kinds of indicators to use. Indeed,many of the indicators summarized in the this annexare simply project activities that are designated asperformance indicators. Worth noting is the fact thatactivity indicators presuppose that the organizationconducting a project is in fact tracking ongoingimplementation of the project. Often, inputs to data-bases track the business activitiesof theproject(expenditures, staff hours, services procured,and “sales” (e.g., devices sold, loans made, workshopparticipants, researchers funded, etc.). Scoping outthe range of data to be collected and ensuring dataquality are challenges that must be met in establishingan activity/participation tracking system. Adequatetracking is fundamental for subsequent monitoringand evaluation exercises.

Output Indicators

Technology, Energy, and Emissions

Physical output indicators are widely employed andextensively verified. Some organizations use outputindicators as targets—i.e., the quantitative level to beattained is set out in advance—against which subse-quent results are measured. We discuss energy effi-ciency and renewable energy technology separately.

Energy efficiency. Organizations supporting energyprojects typically track the physical outputs fromtheir market interventions. Much experience in mea-suring these outputs was gained by the utilities thatconducted DSM projects in North America and else-where in the 1990s. In DSM programs and projects,the number of measures (technologies or practices)implemented by participants is almost alwaystracked; usually, it is combined with estimates ofenergy savings per measure to produce aggregatedenergy savings estimates for projects. When a projectcauses multiple implementations of similar measures,savings are usually calculated on a per-measure basisand aggregated over total measures. When a project isa one-of-a-kind installation, its unique energy savingswill be calculated making use of one or more of avariety of measurement and estimation techniques.

For projects affecting electric demand, savings aretypically calculated over various time periods rangingfrom the point of system peak demand to the entireyear. Methodologies for establishing measure savingsvary and can be quite detailed. For a thorough discus-sion of these issues, see Vine and Sathaye (1999).

Emerging national and international measurementand verification protocols provide methodologies andoptions for calculating savings from efficiencyprojects (see Vine and Sathaye 1997). In energy effi-ciency projects operated by or through energy distribu-tion utilities, pains are typically taken to net out “freeriders” in order to arrive at a net estimate reflectingenergy savings that are additional to the non-project,counterfactual scenario. The utilities typically aggregatethe net savings from their particular projects (individualprograms) to calculate savings from the entire DSMprogram, which are often reported on an incremental,annual, and cumulative basis.

In energy efficiency projects undertaken by entitiesother than utilities, such as governmental agencies,the attention given to netting out free riders has beenmore sporadic. When the managers or evaluators ofan energy efficiency project undertake to estimatesavings without taking into account an explicitcounterfactual scenario, they do not avoid the intel-lectual and technical difficulties involved in trying todetermine what would have happened without theproject. Rather, they explicitly or implicitly make theassumption that in the counterfactual scenario, theproject’s efficiency measures would not have beenimplemented.

Reductions in GHG impacts from efficiency projectsare calculated with increasing frequency, given theglobal interest in climate change issues. This is re-flected in some of the indicators in the Annex. Emis-sions factors based on the characteristics of end-useequipment and of the energy production and distribu-tion system are used in making such calculations.Calculations of the quantity of GHGs reduced fromefficiency projects, when made, are always based onprior calculations of energy saved through projectmeasures implemented. These issues are, of course,highlighted by the discussion of implementing aClean Development Mechanism in connection withthe Kyoto Protocol.

Renewable energy. Relatively few of the indicatorsreviewed related to renewable energy projects per se.In our experience with utility DSM and renewable

Page 42: Measuring Results from Climate Change Programs: Performance Indicators for GEF

35

energy projects of government agencies, we havefound that the most obvious performance indicator(the sum of the generating capacity ratings of renew-able generation technologies installed) is virtuallyalways employed. Other common indicators are an-nual energy produced by renewable energy technolo-gies installed, and environmental impacts, includingGHG emissions, avoided as a result. Measurement ofcapacity added and generation produced is simpleand straightforward. Calculation of emission impactsrequires a counterfactual scenario describing electric-ity production with and without the project. Curi-ously, there has been little effort among organizationsoperating renewable energy projects to identify thenet additions of renewable capacity, i.e., incrementsabove and beyond the capacity that would have beenadded by the unaided market.

Socio-Institutional Changes

Market changes. Energy efficiency and renewableenergy projects usually have as objectives the instal-lation of additional technologies over and above thelevels that would be expected based on market forcesand structures alone. For many years, this physicalobjective (procuring additional renewable or energyefficiency resources) was seen as primary, and mar-ket transformation objectives were commonly sec-ondary. In recent years, however, there has been anincreased emphasis among domestic governmentagencies and development agencies, banks, and orga-nizations on transforming market structures and func-tions to facilitate a greater continuing level of marketpenetration of sustainable energy technologies. Thisemphasis is reflected not only in the market transfor-mation objectives of GEF’s climate change programs(OPs 5, 6, 7), but in the evolving paradigms of otherorganizations as well.

Market changes from projects can be relatively proxi-mate and immediate, or they can appear further outfrom a project in space and time, promising morelong-lasting and sustainable changes. Labels for theseimmediate and longer term effects vary. Martinot(1998) has called the former market intervention andthe latter market development. Energy efficiencypolicymakers in California call the former marketchanges and the latter market effects. Feldman (1995)proposed that market changes first reveal themselvesin leading indicators of change, observable early on,and can later be confirmed through lagging indica-tors of market transformation. We turn to the former

of these variously named aspects of market change,those detectable through project output indicators.

Many types of output indicators for market changeshave been proposed. On the buyers’ side of the mar-ket, general and technical information can be re-ceived, the understanding of and satisfaction withsustainable technologies by consumers and other en-ergy users can increase, and their purchasing plansand preferences can shift toward these technologies.On the sellers’ side of the market, the number oftechnology providers and their capability to make,sell, and service sustainable technologies can grow;and contracting protocols, financing mechanisms,and credit availability for sustainable technologiescan develop. Governmental and policy changes thatfacilitate market transactions in sustainable technolo-gies can occur—regulatory reform, enhanced capa-bility in relevant agencies, new plans and programs.

Development of market intervention indicators haslagged behind the growth of the market transforma-tion paradigm. Some market intervention indicatorsare included within specific GEF projects, particu-larly those of more recent vintage. A slowly growingnumber of other organizations now set out specificmarket intervention indicators and/or targets that arepotentially capable of measurement. Examples arethe project indicators used by the Northwest EnergyEfficiency Alliance in the United States and by NewSouth Wales in Australia.

Capacity building and institutional strengthening.Capacity building and institutional strengthening areobjectives of many energy projects operated by orga-nizations around the world. There appear to be fewtrue output indicators in use relating to these projectobjectives. The indicators organizations generallyuse are really activity or implementation measures, orvery close thereto. An example of this may be seen inthe performance monitoring package of the USAID’sClimate Change Initiative. Capacity-building indica-tors are numbers of NGOs provided with strengthen-ing services (by type of NGO) and numbers ofindividuals trained.

Policy strengthening more readily lends itself to out-put indicators. Policy actions can be specified in ad-vance, and (regardless of whether they are achieved)can later be observed and reported, for the most parton a yes-or-no basis.

Page 43: Measuring Results from Climate Change Programs: Performance Indicators for GEF

36

Intermediate Outcomes/Market Development

Technology, Energy, Emissions

The same kinds of indicators that were described asoutput indicators above are also employed as out-come indicators. Primarily these are estimated effi-ciency savings, installed renewable generatingcapacity, and the associated changes in carbon emis-sions. The difference is in the level at which theseindicators are measured. Where a broader marketthan the immediate market segment or area directlyaffected by project activities is considered(for ex-ample, renewable generating capacity installed at thecountry level—the indicator is at a higher level thanan output measure.

In contrast with output indicators, which are oftenverified, there is very little actual verification (mea-surement) of variables at what we would consider tobe the outcome indicator level. We posit that this lackof measurement is due to the difficulties in establish-ing causality or even plausible influence. Supposethat there is a set of energy efficiency projects withina country or state which achieve a certain degree ofmarket penetration and for which energy savings out-puts are calculated. There are two distinctly opposingpossibilities for considering energy efficiency trendsfor similar end uses outside the project areas:

• These trends, uninfluenced by the projects, are abenchmark for measuring what would have hap-pened in the absence of the projects.

• By contrast, the efficiency projects cause spilloverbenefits that are reflected in efficiency gains out-side the project areas too.

Issues such as these can be addressed through care-fully designed evaluation studies, but such studiestake time and money. In the future, we expect that theissues around establishing and verifying plausibleoutcome indicators through appropriate evaluationstudies may be addressed more frequently.

Socio-Institutional ChangeAs indicated earlier, there has been some develop-ment of market impact indicators, many of which fallinto the market development level insofar as they arelogically linked to project objectives and augur shifts

in the market that may be self-sustaining. Other thanmarket development indicators, few outcome-levelindicators of capacity building or institutionalstrengthening have been identified in the survey ofother organizations at this writing. Apart from themarket development area, indicators of socio-institu-tional development outcomes from climate changeprojects appear to be largely undeveloped at this time.

End Outcomes/Goal Attainment

In principle, performance indicators may be devel-oped to measure (quantitatively) or judge (qualita-tively) the progress of any project, program, ororganization toward achieving its stated end goals.Some organizations use macro-level indicators (car-bon intensity is an obvious example), not so much asperformance indicators, which poses large issues ofcausality or at least plausible influence, but as mea-sures of how much more remains to be accomplishedto attain end goals. In practice, most of the explicitlyarticulated performance indicators that we havefound address “lower” levels of performance.

Conclusions and Implications

The text of this paper makes recommendations re-garding project-level monitoring and evaluation, in-cluding the systematic use of performance indicatorsfor individual projects. If GEF decides to implementthese recommendations, a particularly good frame-work to consult while drawing up the indicated proce-dures is that developed by the German TechnicalCooperation (GTZ). Its objectives-oriented projectplanning and implementation framework (ZOPP)provides a practical approach to explicit linkages be-tween project objectives, selection and measurementof indicators, and monitoring and evaluation, includ-ing identification of unintended as well as intendedimpacts (Deutsche Gesellschaft für TechnischeZusammenarbeit 1997).

The overall conclusion is that other organizations’performance indicators vary greatly. There is a grow-ing number of performance indicators used by differ-ent organizations at different levels. However,institutionalization of market change indicators, andespecially market development indicators, is in itsinfancy, as is development of qualitative capacitydevelopment/institutional strengthening indicators.

Page 44: Measuring Results from Climate Change Programs: Performance Indicators for GEF

37

Table B.1. Terminologies of Different Donor Agencies for Results/Logical Frameworks

Source: Compiled by Jim Rugh for CARE and InterAction’s Evaluation Interest Group (Rugh 1998).

Framework End Outcomes Intermediate Outcomes Outputs Interventions

Process

CARE Impact Effects Outputs Activities and Inputsterminology

CARE logframe Final goal Intermediate goals Outputs Activities and Inputs

PC/LogFrame Goal Purpose Outputs Activities

USAID Strategic objective Intermediate results Outputs Activities and Inputs

USAID Logframe Final goal Strategic goal/ objective Intermediate Activitiesresults

DFID (ODA) Wider objectives Immediate objectives Outputs Inputs

DANIDA + DFID Goal Purpose Outputs Activities

CIDA Overall goal Project purpose Results/outputs Activities and Inputs

GTZ Overall goal Project purpose Results/outputs Activities and Inputs

European Union Overall objectives Project Purpose Results Activities

FAO & UNDP Development Immediate objectives Outputs Activities and Inputsobjective

NORAD Development Intermediate objectives Outputs Activities and Inputsobjectives

World Bank Long-term Short-term objectives Outputs Inputsobjectives

Page 45: Measuring Results from Climate Change Programs: Performance Indicators for GEF

38

Table B.2. DFID Logframe Guide

Source: As received via CARE UK, December 1997.

Table B.3. Performance Indicators Used by a Selection of Other Organizations

Objectives Measurable Indicators Means of Verification Key Assumptions

Goal: Wider problem theproject will help to resolve

Quantitative ways ofmeasuring or qualitativeways of judging claimedachievement of goal

Cost-effective methodsand sources to quantify orassess indicators

(Goal to Supergoal)External factors necessaryto sustain objectives in thelong run

Purpose: The immediateimpact on the project areaor target group, i.e., thechange or benefit to beachieved by the project

Quantitative ways ofmeasuring or qualitativeways of judging claimedachievement of purpose

Cost-effective methodsand sources to quantify orassess indicators

(Purpose to Goal) Externalconditions necessary ifachieved project purposeis to contribute to reachingproject goal

Outputs: The specificallydeliverable resultsexpected from the projectto attain the purpose

Quantitative ways ofmeasuring or qualitativeways of judging timedproduction of outputs

Cost-effective methodsand sources to quantify orassess indicators

(Outputs to Purpose)Factors outside of projectcontrol that, if present,could restrict progressfrom outputs to achievingproject purpose

Activities: The tasks to bedone to produce theoutputs

Inputs: A summary of theproject budget (sub-budgets and total)

Financial outcomereportas agreed in grantagreement

(Activity to Output) Factorsoutside of projectcontrolthat, if present,could restrict progressfrom activities to achievingoutputs

Organization Indicators

New South Wales Sustainable Development Authority(Australia)

Activity and output indicators for energy efficiencyprojects

Northwest Energy Efficiency Alliance (United States) Output and outcome indicators for energy efficiencyprojects

USAID Activity, output, and outcome indicators for the ClimateChange Initiative project

USAID Global Bureau, Environment Center Outcome indicators

Energy Savings Trust (United Kingdom) Output indicators for efficiency projects

Electrobras (Brazil) Outcome indicators for National Electricity ConservationProgram (PROCEL)

National Energy Conservation Center (ENERCON)(Pakistan)

Output and outcome indicators for transportationefficiency project of ENERCON and UNDP

Deutsche Gesellschaft für Technische Zusammenarbeit(GTZ)

Activity and output indicators for selected energyefficiency and renewable energy projects

Page 46: Measuring Results from Climate Change Programs: Performance Indicators for GEF

39

Annex C. Examples of Performance Indicators

Possible performance indicators that can be used inassessing climate change project and program perfor-mance are listed here. Project designers should selectproject-specific indicators to be consistent with ex-pected outputs and project objectives. Also, to conductcross-cutting evaluations of several projects to provideinformation about climate change program impacts,evaluators should select an appropriate set of indicatorsbased on the nature of the cluster of projects selected forstudy. The possible indicators that follow were sug-gested by the review of projects conducted for this paperand are provided as a resource.

Possible Indicators for Energy Efficiency (EE)Projects

These indicators were suggested by a review of thefollowing groupings of projects:

• Cluster 5, EE-Product Manufacturing and Mar-kets (primary template)

• Cluster 6, EE Investments in Industry

• Cluster 7, Building Codes/Construction

• Cluster 8, District Heating EE Improvements.

Financial

1. Number and dollar volume of completed EEproject transactions using financing (e.g., amountsborrowed) [broken out by sector: utility, govern-ment, industry, consumers, etc.]

2. Number and dollar volume of completed EEprojects (total installed costs) [broken out by sec-tor: utility, government, industry, consumers, etc.]

3. Number of innovative financial and contractingmechanisms (packages) [broken out by sector:utility, government, industry, consumers, etc.]

4. Market acceptance of innovative financial andcontracting mechanisms (packages) [broken outby sector: utility, government, industry, consum-ers, etc.]

5. Number of commercial financial institutions par-ticipating in EE projects

6. Pipeline of EE (portfolios of) projects ready forimplementation and financing by commercialparties

7. Level of EE investment activity in domestic andcommercial sectors in country [broken out by sec-tor: utility, government, industry, consumers, etc.]

8. Level of EE investment activity in other countries[broken out by sector: utility, government, indus-try, consumers, etc.]

9. Revolving fund to support financing of incremen-tal investment costs

10. Equipment leasing program

Page 47: Measuring Results from Climate Change Programs: Performance Indicators for GEF

40

11. Number and type of financial incentives offered[broken out by sector: utility, government, indus-try, consumers, etc.]

12. Amount of financing leveraged [broken out bysector: utility, government, industry, consumers,etc.]

13. Number of grants issued [broken out by sector:utility, government, industry, consumers, etc.]

ESCOs

1. Number of ESCOs formed

2. Level of ESCO activity (number and value of con-tracts with customers)

3. Business advisory services for facilitating devel-opment of an ESCO industry

4. Pilot energy performance contracting program(loan guarantees to support performance contracts)

5. Training of ESCOs and NGOs on providing EEservices [could go under training]

6. Cluster 6: Number of business plans developedfor ESCOs

EE Programs, Projects, and Measures

1. Number of buildings retrofitted (percent of build-ing population)

2. Number (percent) of EE measures installed

3. Number of EE projects implemented and evaluated

4. Number of DSM programs

Labeling [could go under policymaking]

1. Number and type of EE products tested, labeled,and certified

2. Nationally certified EE label [could go underpolicymaking]

3. Labeling of new equipment

4. Information or labeling system for equipment[could go under information]

Awareness and Acceptance

1. Level of awareness and understanding of EE prod-ucts, services, and/or actions [broken out by sec-tor: utility, government, industry, consumers, etc.]

2. Awareness of business opportunities in EE field

3. Increased awareness of codes and standards byconsumers, architects, engineers, constructioncompanies, and building owners and developers

4. Public acceptance of EE measures

Availability

1. Number (percent) and type of EE products andservices available

2. Cluster 6: Use of locally made EE equipment

Information

1. Information network (clearinghouse; newsletters,Internet, and conferences/workshops) developed

2. Dissemination of results of EE projects

3. Number of demonstration projects: (1) of EE mea-sures; (2) with key actors (e.g., ESCOs) [broken outby sector; includes monitoring and verification]

4. Number of energy audits (in particular facilities,or by sector)

5. Guidelines on identifying and developing EEprojects

6. Number of workshops for retailers and distribu-tors to encourage sales of EE equipment

7. Cluster 7: Communication plan

Capability Building

1. Support office created to coordinate and supportinstitutional and capacity-building activities in EE

Page 48: Measuring Results from Climate Change Programs: Performance Indicators for GEF

41

2. Offices established for identifying EE opportuni-ties in operations and developing and implement-ing programs for EE

3. Number of training programs for staff, experts,industry personnel, energy managers, and ESCOs

4. Training of architects, engineers, owners of con-struction companies, and building owners anddevelopers

5. Strengthened institutional capabilities (includinginformation management, evaluation, and dissemi-nation) of organization promoting EE

6. Improvements in marketing capabilities

7. Cluster 6: Energy auditing instituted as a regularactivity [could also apply to other activities: e.g.,energy labeling]

8. Cluster 6: Accreditation program for energyauditors

9. Cluster 7: Least-cost planning and DSM methodsfor energy sector operators

Regulatory Policymaking

1. Model energy policy and guidelines (for munici-palities)

2. Strengthened environmental standards

3. (Stricter) EE equipment standards

4. Elimination of subsidies encouraging energy con-sumption

5. Cluster 8: End user metering for district heatingsystems

6. Time-of-use tariff to encourage load shifting

7. Reduction of custom duties on EE equipment

8. Design, implementation, and enforcement of EEcode for new buildings (or at least a code ofpractice)

9. Life cycle government procurement policies thatconsider energy costs associated with purchase ofenergy-using equipment

10. Legal, financial, institutional, and regulatory poli-cies instituted to ensure large-scale, sustainablefinancing of EE investments [could go under fi-nancing]

Studies/Proposals

1. Number of feasibility studies

2. Number of proposals

Market Infrastructure Development

1. Number (percent) of EE measures manufacturedby in-country manufacturers

2. Number (percent) of dealers and distributors stock-ing and selling EE equipment

3. Number of manufacturers producing EE models

4. Number of dealer incentive programs

5. Increased local manufacturer capabilities to pro-duce equipment in compliance with standards

6. Number and type of EE models stocked by appli-ance and equipment vendors

7. Manufacturer investments in/production of EEequipment

8. Cluster 7: Number of contractors purchasing EEbuilding materials

9. Cluster 7: Number of contractors incorporating EEbuilding design in their activities

Other

1. Improvements in engineering, operations, produc-tion management

2. Increased utility support of EE technologies

Page 49: Measuring Results from Climate Change Programs: Performance Indicators for GEF

42

3. Independent testing laboratories

4. Consumer purchases of EE equipment

Possible/Additional Indicators for RenewableEnergy Projects

Additional indicators were suggested by review ofthe following types of renewable energy and fuelswithcing projects:

• Capital costs per kW of demonstrated renewableor low-GHG-emitting technology (in host/othercountries)

• Installations of demonstrated technologies outsideof project (in host/other countries)

• Regulatory policymaking: specific tariffs/policiesto provide incentives/remove disincentives toadoption of renewable technologies

Possible/Additional Indicators for Cross-Cutting Types of Projects

Cluster: Sustainable Energy TechnologyInvestment Funds

1. Indicators for particular projects financed throughthe funds

2. Indicators for the financial performance of thefunds themselves:

• Number of subprojects and/or business plansfunded

• Repayment rates for loans extended

• Financial returns to financial intermediaries(average and aggregate)

Cluster: Capacity or Knowledge Building

1. Capability-building indicators from above

2. Additional indicators:• Declining use of external consultants (amount

of donor receipts used for foreign/expatriatetechnical assistance)

• For NGOs: enhanced capacity to do researchand advocacy work (number of staff withthese skills)

• Organizational restructuring from bureau-cratic to networking organizational paradigm(business, government)

Possible/Additional Indicatorsfor Social Impacts

1. Affected population (number and percentage of endusers and/or beneficiaries to total population)10

2. Off-grid renewable energy systems in rural areas:

• Number of affected households or villages andpercent with installed renewable energy ser-vices relative to total relevant population

• Changes in energy use (e.g., reduction in useof kerosene and woodfuels)

• Estimated changes in livelihood and income,including other social parameters (e.g., in-crease or decrease in household income, re-duction in women’s labor time for gatheringfuel, power provided to rural village healthand education facilities, etc.)

• Acceptance and satisfaction among (1) directbeneficiaries, (2) those indirectly affected

3. Off-grid renewable energy systems with povertyalleviation components:

• Linkage of rural renewable energy project/scheme to national poverty alleviation programs

• Anticipated impacts on the poor (e.g., improv-ing service delivery for health, family plan-ning, education; improving infrastructure forfarm- or fishery-based livelihoods, etc.)

The following tools are examples of those that canbe used to measure these additional social impactindicators:

• Collection of secondary socioeconomic informa-tion: published government reports (statisticalyearbooks, subnational government surveys and

Page 50: Measuring Results from Climate Change Programs: Performance Indicators for GEF

43

censuses); community information, village histo-ries, reports from NGOs, churches, etc.; publishedresearch reports (books, articles, technical papers,etc.)

• Key informant interviews: checklist format; infor-mal, consensus building techniques; formal meet-ings; life histories

• Community mapping, including production andlivelihood diagramming and gender analysis

• Social surveys: non-random sampling; randomsampling; purposive, stratified sampling

Page 51: Measuring Results from Climate Change Programs: Performance Indicators for GEF

44

Page 52: Measuring Results from Climate Change Programs: Performance Indicators for GEF

45

Bibliography

Global Environment Facility (GEF). 1995. Report bythe GEF to the First Conference of the Parties ofthe Framework Convention on Climate Change.Washington, DC.

——. 1996a. Operational Strategy. Washington, DC.

——. 1996b. Public Involvement in GEF-FinancedProjects. Washington, DC.

——. 1996c. Promoting Strategic Partnerships be-tween GEF and the NGO Community: A Reportfrom the GEF-NGO Working Group. Washington,DC.

——. 1996d. Incremental Costs. Washington, DC.

——. 1997a. Operational Programs. Washington, DC.

——. 1997b. Project Implementation Review 1996.Washington, DC.

——. 1997c. The Global Environment Facility SmallGrants Program. Washington, DC.

——. 1997d. Medium-Sized Projects. Washington, DC.

——. 1998a. Operational Report on GEF Programs.Washington, DC.

——. 1998b. Study of GEF Project Lessons (SummaryReport). Washington, DC.

Ahuja, Dilip. 1993. The Incremental Cost of ClimateChange Mitigation Projects. Washington, DC:Global Environment Facility.

Anderson, Dennis, and R.H. Williams. 1993. The Cost-Effectiveness of GEF Projects. Global Environ-ment Facility Working Paper Number 6. Wash-ington DC: Global Environment Facility.

Ashford, Michael S. 1999. Contingent Finance as aGEF Financing Modality. Part 1: Conceptual Is-sues. World Bank Environment Department WhitePaper. Washington, DC: World Bank.

Deutsche Gesellschaft für TechnischeZusammenarbeit. 1996. ZOPP: Objectives-Ori-ented Project Planning. Eschborn, Germany.

——. 1997. Project Cycle Management (PCM) andObjectives-Oriented Project Planning (ZOPP).Eschborn, Germany.

Feldman, Shel. 1995. “Measuring Market Effects.”Proceedings of the 1995 AESP Annual Meeting.Boca Raton, FL: Association of Energy ServiceProfessionals.

Feldman, Shel. 1996. On Estimating the Value Addedthrough Market Transformation. Oak Ridge, TN:Oak Ridge National Laboratory.

Geller, Howard, and Steven Nadel. 1994. “MarketTransformation Strategies to Promote End-UseEfficiency.” Annual Review of Energy and theEnvironment 19:301-46.

Page 53: Measuring Results from Climate Change Programs: Performance Indicators for GEF

46

——. 1998c. Project Implementation Review 1997.Washington, DC.

——. 1998d. Proceedings of the First Assembly of theGlobal Environment Facility, New Delhi, 1-3 April1998. Washington, DC.

——. 1999a. Project Performance Report 1998. Wash-ington, DC.

——. 1999b. “Engaging the Private Sector in GEF Ac-tivities.” Information note presented to GEF Coun-cil May 5-7. GEF/C.13/Inf.5. Washington, DC.

Herman, Patricia, Shel Feldman, Shahana Samiullah,and Kirsten Stacey Mounzih.1997. “MeasuringMarket Transformation: First You Need a Story…”The Future of Energy Markets: Evaluation in aChanging Environment. Proceedings of the Inter-national Energy Program Evaluation Conference,Chicago, pp. 319-26. Washington, DC: AmericanCouncil for an Energy-Efficient Economy.

Intergovernmental Panel on Climate Change. 1996a.Climate Change 1995: Impacts, Adaptations andMitigation of Climate Chang: Scientific-Techni-cal Analyses. Contribution of Working Group IIto the second assessment report of the Intergov-ernmental Panel on Climate Change. New York:Cambridge University Press.

——. 1996b. Technologies, Policies and Measures forMitigating Climate Change. Working Group II ofthe Intergovernmental Panel on Climate Change.Geneva.

——. 1996c. Climate Change 1995: Economic andSocial Dimensions of Climate Change. Contribu-tion of Working Group III to the second assess-ment report of the Intergovernmental Panel onClimate Change. New York: Cambridge Univer-sity Press.

——. 2000. Special Report on Methodological andTechnological Issues in Technology Transfer.Working Group II of the Intergovernmental Panelon Climate Change. Geneva. Forthcoming.

Kammen, Daniel M. 1999. “Promoting AppropriateEnergy Technologies in the Developing World.”Environment 41(5):11-15, 34-41.

Kozloff, Keith, and Olatokumbo Shobowale. 1994.Rethinking Development Assistance for Renew-able Energy. Washington, DC: World ResourcesInstitute.

Kushler, Martin, J. Schlegel, and R. Prahl. 1996. “ATale of Two States: A Case Study Analysis of theEffects of Market Transformation.” Proceedingsof the 1996 ACEEE Summer Study on Energy Ef-ficiency in Buildings. Washington, DC: AmericanCouncil for an Energy-Efficient Economy.

Lee, Allen D., and R. Conger. 1996. “Market Trans-formation: Does it Work?—The Super EfficientRefrigerator Program.” Proceedings of the 1996ACEEE Summer Study on Energy Efficiency inBuildings. Washington, DC: American Council foran Energy-Efficient Economy.

Liebenthal, Andres, Subodh Mathur, and HerbertWade. 1994. Solar Energy: Lessons from the Pa-cific Island Experience. World Bank TechnicalPaper No. 244, Energy Series. Washington, DC:World Bank.

Martinot, Eric. 1998. Monitoring and Evaluation ofMarket Development in World Bank/GEF Climate-Change Projects: Framework and Guidelines.World Bank Environment Department Paper 66.Washington, DC: World Bank.

——. 2000. “Renewable Energy Markets and the Glo-bal Environment Facility.” Financial Times Renew-able Energy Report 12(February):18-22.

Martinot, Eric, and Omar McDoom. 2000. PromotingEnergy Efficiency and Renewable Energy: GEFClimate Change Projects and Impacts. Washing-ton, DC: Global Environment Facility.

Nadel, Steven, and H. Geller. 1994. “Market Trans-formation Programs: Past Results, Future Direc-tions.” Proceedings of the 1994 ACEEE SummerStudy on Energy Efficiency in Buildings. Wash-ington, DC: American Council for an Energy-Ef-ficient Economy.

Porter, Gareth, Raymond Clemencon, Waafas Ofosu-Amaah, and Michael Philips. 1998. Study of GEF’sOverall Performance. Washington, DC: GlobalEnvironment Facility.

Page 54: Measuring Results from Climate Change Programs: Performance Indicators for GEF

47

Prahl R, and J. Schlegel. 1993. “Evaluating MarketTransformation.” Energy Program Evaluation:Uses, Methods, and Results, pp. 469-77. Chicago:National Energy Program Evaluation Conference.

Reed, John H., and Nicholas P. Hall. 1997. “Methodsfor Measuring Market Transformation.” The Fu-ture of Energy Markets: Evaluation in a Chang-ing Environment, pp. 177-84. Chicago: Interna-tional Energy Program Evaluation Conference.

Richards, David, Kane Thierno, Moises Leon, and TinaLiamzon. 1995. Report of the Independent Evalu-ation of the Global Environment Facility SmallGrants Programme (GEF/SGP) Pilot Phase. NewYork: UNDP/GEF.

Rogers, E. 1995. Diffusion of Innovations. New York:Free Press.

Rugh, James. 1998. An Overview of Program Design,Monitoring and Evaluation (DME) in CARE.

Saxonis, William P. 1997. “Market Transformation:Real Problems, Real Answers.” The Future ofEnergy Markets: Evaluation in a Changing Envi-ronment, pp. 171-76. Chicago: International En-ergy Program Evaluation Conference.

Sjöberg, Helen. 1994. From Idea to Reality: The Cre-ation of the Global Environment Facility. GEFWorking Paper 10. Washington DC: Global Envi-ronment Facility.

Suozzo, Margaret, and S. Nadel. 1996. “Learning theLessons of Market Transformation Programs.”Proceedings of the 1996 ACEEE Summer Studyon Energy Efficiency in Buildings. Washington,DC: American Council for an Energy-EfficientEconomy.

United Kingdom, Department for International Devel-opment. 1993. Project Evaluation: A Guide forNGOs. Glasgow, UK: Overseas DevelopmentAdministration.

——. 1994. A Guide to the ODA Evaluation System.Glasgow, UK: Overseas Development Adminis-tration.

United Nations. 1992. Framework Convention on Cli-mate Change. New York.

——. 1993. Agenda 21: Program of Action for Sus-tainable Development. New York.

United Nations Development Program. 1997. NGOParticipation in UNDP-GEF Projects. New York:UNDP/GEF.

United Nations Development Program, United NationsEnvironment Programme, and World Bank. 1994.Global Environment Facility: Independent Evalu-ation of the Pilot Phase. Washington, DC: WorldBank.

Vine, Edward, and Jayant Sathaye. 1997. The Moni-toring, Evaluation, Reporting, and Verification ofClimate Change Mitigation Projects: Discussionof Issues and Methodologies and Review of Exist-ing Protocols and Guidelines. Berkeley, CA:Lawrence Berkeley National Laboratory.

——. 1999. Guidelines for the Monitoring, Evalua-tion, Reporting, Verification, and Certification ofEnergy-Efficiency Projects for Climate ChangeMitigation. Report LBNL-41543. Berkeley, CA:Lawrence Berkeley National Laboratory.

Violette, Daniel, Shannon Ragland, and FranklinSterm. 1998. Evaluating Greenhouse Gas Miti-gation through DSM Projects: Lessons Learnedfrom DSM Evaluation in the United States. Pre-pared for the World Bank. Washington, DC:Hagler-Bailly.

Wells, Michael P., Delfin J. Ganapin, and FrancineTrempe. 1998. Report on the Second IndependentEvaluation of the GEF Small Grants Program(SGP/GEM). New York: UNDP/GEF.

Williams, Robert, Stephen Karakezi, Jyoti Parikh, andChihiro Watanabe. 1998. The Outlook for Renew-able Energy Technologies. GEF Working Paper 14.Washington DC: Global Environment Facility.

Wirtshafter, Robert M., and A. Sorrentino. 1994.“Proving Causality in Market Transformation Pro-grams: Issues and Alternatives.” Proceedings ofthe 1994 ACEEE Summer Study on Energy Effi-ciency in Buildings. Washington, DC: AmericanCouncil for an Energy-Efficient Economy.

Page 55: Measuring Results from Climate Change Programs: Performance Indicators for GEF

48

World Bank. 1994. Greenhouse Gas Abatement In-vestment Project Monitoring and EvaluationGuidelines. Environment Department White Pa-per. Washington, DC.

——. 1996. Rural Energy and Development: Improv-ing Energy Supplies for 2 Billion People. Wash-ington, DC.

——. 1998. Greenhouse Gas Assessment Handbook:A Practical Guidance Document for the Assess-ment of Project-Level Greenhouse Gas Emissions.Environment Department Working Paper 64.Washington, DC.

Page 56: Measuring Results from Climate Change Programs: Performance Indicators for GEF

49

Endnotes

climate change efforts—not only projects, but alsoenabling and adaptation activities and short-term re-sponse measures. Thus, communications enabled,GHG inventories supported, and research performedcould be reflected here. In addition, a productionreport could be an apt vehicle for capturing the activi-ties of projects that have important objectives in addi-tion to or other than market development. Forexample, the United Nations EnvironmentProgramme has been the implementing agency forGEF projects aimed at building countries’ methodsand capacities for assessing climate change mitiga-tion and adaptation strategies rather than at directlyremoving market barriers to sustainable energy tech-nologies.

6 For example, the target values of an indicator maybe based on the assumption that growth in a country’sgross domestic product will fall between 2 and 4percent per year on a rolling average basis over thenext decade. If economic growth is much stronger orweaker than that, targets should be revised. Anotherexample might be if the entity and staff positionassigned measurement responsibility were abolishedby a government restructuring; measurement respon-sibility would subsequently need to be reassigned.These are simple illustrative examples of assump-tions. It is not practical to reflect all of the conditionsthat bear on project outcomes in explicitly identifiedkey assumptions. Rather, these identified assump-tions recognize conditions that are believed to beclosely related to project outcomes and that areknown to be subject to risk of variation. Other unan-ticipated developments that significantly affect the

1 Annex C includes a menu of project-level indicators.Some of these are forms of the core indicators thatmay be useful at the project level. Many are additionalindicators that can be used to express project perfor-mance in more detail.

2 Martinot and McDoom (2000) define and utilizethese nine clusters in describing the climate changeproject portfolio. Clustering was also suggested bymany of the interviews mentioned in the first sectionof this paper.

3 Two sources for electricity data are Projected Costsof Generating Electricity, prepared by the NuclearEnergy Agency and the International Energy Agency(update 1998); and Developing Countries & GlobalClimate Change: Electric Power Options for Growth,1999, prepared for the Pew Center on Global ClimateChange. Data from these sources are being put into aTechnology & Environment Data Base that will beavailable through the Stockholm Environment Insti-tute-Boston (www.leap2000.org). A source coveringall energy forms is International Energy Outlook1999 from the U.S. Department of Energy’s EnergyInformation Administration.

4 As a multiyear multicountry project, the EfficientLighting Initiative will be subject to ongoing assess-ment by evaluation consultants hired by the IFC. Werefer here, however, to cross-cutting evaluation thatwould include additional lighting project markets.

5 The scope of a production report can be all of theactivity that occurs within the boundaries of all GEF

Page 57: Measuring Results from Climate Change Programs: Performance Indicators for GEF

50

validity of performance indicators must also be takeninto account if and as they occur.

7 Ideally, these baseline studies should be performedbefore a project is fully designed. One solution to thispotential conundrum may be the use of project devel-opment funds to ensure that targets for full projects—and, indeed, their strategic designs as well—arebased on essential market information. Another ap-proach could be two-phase projects in which perfor-mance targets are adjusted after baseline research isdone during the first phase.

8 A similar approach could also be taken for theimplementation success ratings that GEF collects forits PIRs, moving those reports too from a subjectiveto an objective basis.

9 “Country level” can also include impacts at regional(subnational) levels that also extend beyond theproject level.

10 End users or consumers are differentiated frombeneficiaries in that the latter is a broader stakeholderterm that may include capacity-building recipients ingovernment and NGOs, for example, who may notnecessarily be receiving energy services directly.

Page 58: Measuring Results from Climate Change Programs: Performance Indicators for GEF

Global Environment Facility1818 H Street, NW

Washington, DC20433 USA

tel: 1(202)473-0508fax: 1(202)522-3240/3245

www.gefweb.org