world bank group impact...
TRANSCRIPT
1
WORLD BANK GROUP IMPACT EVALUATIONS: RELEVANCE AND EFFECTIVENESS
Approach Paper
INTRODUCTION
1. There has been a rapid expansion in recent years in the production of impact
evaluation as a method to assess the impacts of development projects, which is largely driven by an increasing demand for credible evidence of development results. Much of the
development community perceives impact evaluation (IE) as a tool that provides rigorous and
objective estimates of the effects of specific interventions. This had led to expectations that IE
may help build the knowledge base of what does and does not work in development and where
resources may be best allocated. As part of its results and knowledge agenda, the World Bank
Group (WBG) has made important efforts to expand and deepen its IE work.1 Major initiatives
include the Development Impact Evaluation Initiative (DIME), the Africa Impact Evaluation
Initiative (AIM), the Spanish Trust Fund for Impact Evaluation (SIEF), and IFC‟s Advisory
Services Unit in the Development Impact Department (previously Results Measurement Unit).
Recognizing these trends, in the fiscal year 2005, OPCS established IE as a new product line
under the analytic and advisory activities (AAA) umbrella.
2. There has been a growing base of support for applying impact evaluation to
improve development practices,2 but the contribution of impact evaluation has not been
systematically evaluated. The Bank has increased the number of IEs in Bank-supported projects
as a way to “improve the quality of Bank‟s operations, strengthen country institutions for
evidence-based policy making, and generate knowledge in 15 strategic development areas.”3
The recent DIME strategy paper for World Bank IE moreover proposes to mainstream IE as a
core instrument in the Bank‟s knowledge agenda and analytic toolkit (Box 1). The IFC has also
been conducting IEs on private sector projects, particularly for testing pilots and different
approaches of its advisory services, while an institutional strategy has not been established for
building out an IE program. Since 2005, the WBG has spent at least $44 million on IE, financed
both by internal and trust fund resources.4 Little is known about whether WBG IEs have been
conducted (i) in areas where there are strategic analytic and policy knowledge gaps, (ii) are of
high quality, and (iii) have influenced resource allocation, project design/implementation, future
evaluation, strategy or policy making. The timing of this evaluation may thus provide inputs to
the WBG‟s IE strategy moving forward.
3. This joint evaluation will examine the experience of impact evaluation at the WBG
(World Bank and IFC) and its contribution to the development practices of the WBG along
several dimensions. It aims to evaluate the relevance of IEs being conducted by the WBG
(completed and ongoing), their quality, and their use and influence from both the institution‟s
and client‟s perspectives. This includes assessing the institutional organization of IEs, the current
and ideal scope for impact evaluation in WBG-supported projects (in a selected sector), and their
relevance to operations, strategic planning, the knowledge and M&E agendas, and dialogue with
clients. On the country level, this includes the influence of IEs in policymaking and their use for
evaluation purposes. This study recognizes systematic differences in the motivation, production,
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
2
and application of IEs between the WB and IFC and will make the distinction whenever
appropriate.
4. This report is in line with the current effort of IEG to better understand and
evaluate the contribution of AAA to the development effectiveness of the WBG. This
evaluation is similar to other recent IEG evaluations of AAA: the 2010 PSIA evaluation, the
2008 ESW and TA evaluation, the 2003 Sharing Knowledge evaluation, and other evaluations of
particular AAA reports within country assistance reviews.5 It follows IEG‟s goal of evaluating
AAA products in terms of their explicit and implicit objectives and derive findings that can
enhance their relevance and effectiveness.
5. The findings will be derived from several evaluation tools: (1) desk review of impact
evaluations; (2) review of relevant documents of the evaluated projects; (3) electronic surveys of
IE authors and WBG staff who have worked on the evaluated projects; (4) interviews with
internal and external stakeholders in selected countries; and (5) sector case studies. The study
will triangulate the inputs from the perspectives of different actors (such as World Bank staff,
evaluators, government officials, and private sector representatives). Efforts will be also made to
identify information about the costs of the IEs to give a rough estimate of the resources allocated
to this type of technical analysis. The IEs included in this report must incorporate a credible
counterfactual analysis and have the WBG‟s involvement as an evaluator, funder, or technical
assistance provider. This study looks mostly at IEs of WBG projects that have been completed in
the last decade (2000-10) as well as ongoing IEs.6 The evaluation will also review the relevance
and quality of IEs of programs that have not received WBG financing in addition to the
relevance and effectiveness of IE-related activities, such as workshops, guidance notes and IE
clinics undertaken by DIME, SIEF, AIM and other initiatives at the WBG.
BACKGROUND
DEFINITION AND ROLE OF IMPACT EVALUATION
6. Impact evaluation is a method of assessing the impact attributable to an
intervention, where the outcomes are compared with a counterfactual situation – what
would have happened without the program.7 A key challenge of any evaluation is that the
counterfactual cannot be observed. IE seeks to overcome this by creating a control group (or
counterfactual) that is similar to the group which receives the treatment. The specific techniques
for identifying the control group vary according to the program setting and available data, but
can be broadly classified into two categories: experimental and quasi-experimental (i.e. non-
experimental). Experimental evaluations are possible when treatment and control groups are
selected randomly prior to, or during, program implementation. When randomization is not
possible, various quasi-experimental techniques can be used as an attempt to remove the
selection bias that could arise from observable and unobservable differences between the control
and treatment groups.8 In contrast to other evaluation approaches that, for instance, assess
whether targets have been achieved, impact evaluation is structured around counterfactual
analysis to compare what actually happened with and without the intervention.
3
7. Like any other evaluation approach, impact evaluations have strengths and
limitations in assessing the effectiveness of, and learning from, development programs. In
addition to establishing the link between an intervention and an outcome of interest, IE can
quantify the added value of variations in the interventions. It can in principle provide credible
information not only on what works but also on how, for whom, under which circumstances, and
at what cost. However, IE has also been criticized for not fully answering what factors induce the
change and in which manner, for being highly project- and context-specific, for lacking
efficiency analysis, and for their limited applicability to some sectors. Likewise, IEs may be less
appropriate to assess the effectiveness of thematic, sector and country strategies.
Box 1. Development Impact Evaluation Initiative: A World Bank-Wide Strategic Approach to
Enhance Developmental Effectiveness
Since 2009, DIME has taken on the new mandate as an organizing framework for IE at the Bank. Its
current portfolio of WB-supported IEs includes 170 completed and 280 active studies in 72 countries.
While not all evaluations in the Bank have been actively supported by DIME, DIME‟s objectives are
closely aligned with the Bank‟s strategy for making research relevant to policy decision-making and the
Bank‟s operational work. In particular, by increasing the number of Bank-supported projects with IEs,
DIME seeks to accomplish three objectives: (1) to improve the quality of the Bank‟s operations through
iterative learning, (2) to generate knowledge on critical development questions, and (3) to strengthen client
institutions for evidence-based policy-making. The model aims at increasing the share of IEs supported by
DIME so that they can benefit from the approaches, community of practice, and capacity building
activities provided by DIME. The overall effort is led by a high level steering group, coordinated by a
secretariat based in DEC, guided thematically by the networks, implemented by the regions, and supported
by the research and data groups in DEC with analytical and data services.
DIME‟s objectives are expected to be achieved through three main strategies, reflecting each of the three
functions that IEs are supposed to provide (project assessment, knowledge generation and evaluation
promotion) and that are planned to be assessed in this evaluation: (1) promoting prospective IEs, an
evaluative process planned before implementation (or expansion) of the intervention, to improve
operational quality –bringing IEs into the results agenda, providing just-in-time advice to incorporate
evidence in project designs, measuring project results, improving impacts during implementation, and
introducing formative evaluation based on experimental methods; (2) producing IEs to systematically
learn from development experience—aligning thematic IE programs with Bank‟s learning priorities and
actively disseminating the large DIME portfolio via cross-country workshops, seminars, conferences,
network weeks, web and journal publications, and thematic policy reports; (3) building evaluation
capacity and supporting results-based policy making through joint evaluations—transferring
technical skills, promoting sound data collection strategies, networking with a large community of
practitioners, and learning-by-doing through joint government-Bank evaluations.
THE WORLD BANK GROUP ’S ENGAGEMENT IN IMPACT EVALUATION
8. The WBG has been rapidly expanding its efforts in impact evaluation, increasing
the number of WBG-supported evaluations by seven fold between 2004 and 2008 through
several initiatives.9 The WBG‟s IE promotion efforts include conducting IEs, reaching out to
WBG‟s operations, supporting countries in implementing IEs of their programs and
strengthening their monitoring and evaluation (M&E) capacity, developing best practices, and
facilitating global learning from IE results on effective development interventions. The initiatives
to mainstream IE within and outside the Bank include:
4
Multi-country dedicated IE funds: (1) the Africa Impact Evaluation Initiative (AIM),
which started in 2004 in the Africa Region to support capacity development for IE, and
provides technical assistance to produce and support country-specific IEs of public
programs;10
and (2) the Spanish Trust Fund for Impact Evaluation (SIEF), established by
the Human Development Network (HDN) in 2007 and currently the largest trust fund at
the Bank that focuses on providing funding for IEs of human development interventions,
training, and result dissemination.11
Centralized efforts: (1) the Development Impact Evaluation Initiative (DIME) created in
2004 as a Bank-wide effort to increase the number of Bank projects with IE components,
enhance staff capacity, and build a learning process based on the active participation of
different stakeholders in the design of new IEs and evidence of completed evaluations;12
and (2) IFC effort since 2005 to support IEs of its advisory services projects.
Decentralized efforts: (1) the Poverty Impact Evaluation Thematic Group, managed by
PREM, focuses on IE of governance and institutional reforms;13
(2) the Finance and
Private Sector Research team in DECRG, which uses IE techniques in the areas of
finance and private sector development; and (3) collaboration between DEC researchers
and Bank‟s operations to incorporate IEs into some projects.
9. The DIME report on the Bank’s IE strategy promotes IE as playing an important
role in the WBG’s results, knowledge, and M&E agendas. It characterizes the growth in IE
as the Bank‟s response to increasing demands of donors and client countries for more meaningful
and reliable analyses that assess the effectiveness of development interventions. The potential
contributions of IE to these agendas include:
i) Results agenda: The agenda is at the core of efforts to enhance the WBG‟s development
effectiveness. By generating project performance information, the Bank identifies IEs as an
evaluation tool that contributes to improving decision making in a results-based management
framework.14
ii) Knowledge agenda: The WBG has prioritized strengthening its comparative advantage
in development knowledge.15
The pool of evidence from IEs is seen by the Bank as adding to
the knowledge base of the WBG where it could help connect local experiences, motivate
open dialogues on results and learning, and develop development solutions applicable to
different contexts.
iii) M&E effort: This agenda aims to ensure that all WBG instruments include a results
framework and that the WBG conducts self-evaluations of its work. Moreover, the Bank
assists its clients in developing and implementing M&E methodologies in all government
areas.16
The Bank deems IE to be an important component in a rigorous M&E system and,
with its emphasis on data quality and technical skills, it is argued to provide key inputs for
building and promoting capacity for results-based evidence policy-making.
EVALUATION FRAMEWORK AND QUESTIONS
10. With the goal to evaluate the impact evaluation experience at the WBG and its
contribution to enhancing development effectiveness, this study aims to address the overarching
question “To what extent and why (or why not) have impact evaluations been relevant and
5
effective to inform development practice of the WBG and its clients? The evaluation will
examine various aspects of IE relevance, quality and applicability along a process chain from
initiation through production and uptake of IEs (Figure 1). The IE process chain put forth here
proposes that factors that play a role in the initiation of IEs -such as motivation and purpose, area
of focus, timing and funding of the IE-may affect aspects of IE production, such as technical
quality, costs, analysis of impact distribution, among others. These in turn may influence the
dissemination and uptake of IEs, by affecting project decisions, knowledge generation for future
projects, and the promotion of capacity and resources for evaluation (Figure 1).17
Figure 1. Process Chain from IE Initiation to Uptake
11. There are three main evaluative questions that emerge from this overarching issue: (A)
To what extent and why (or why not) are impact evaluations relevant to close knowledge gaps
and aligned with WBG’s and clients’ priorities and strategies? (B) To what extent and why (or
why not) do impact evaluations meet the expected quality standards and address the questions of
interest? (C) To what extent and why (or why not) are impact evaluations used to influence
development practice? While there have been many guidelines and expectations of what IEs
should accomplish, there are no explicit objectives or ‟best practice‟ set out to evaluate IEs.
Therefore, to the extent possible, this report will rely on evaluation literature and other related
literature within and outside the WBG to guide the analytic framework required to approach
these questions.
Initiation of IEs
Area of focus (sector, country, and intervention)
Motivation and purpose (demand or supply driven)
Involvement of WBG staff, management, country clients, or external actors
Timing in project cycle (prospective, mid-term, ex-post)
Design and evaluative questions
Funding sources
Production of IEs
Technical elements (methods used)
Cost
Articulation of theories linking interventions to outcomes
Inclusion of other analytical issues (analysis of impact distribution, channels behind the
impacts, cost benefit analysis, etc.)
Dissemination and Uptake of IEs
Inform project decisions (design, implementation, expansion, termination, etc.)
Generate knowledge for future projects, policies, debates, and institutional strategies
Promote capacity and resources for evaluation
6
A. Relevance of Impact Evaluations
12. To what extent and why (or why not) are impact evaluations relevant to closing the
knowledge gap and alignment with WBG’s and client’s priorities and strategies? This
question refers to the first step in the process chain – initiation of the impact evaluation. It aims
to describe the areas in which IEs are (or are not) done, and if these IEs are indeed relevant in
terms of analytic and strategic priorities. Relevance will be assessed by the process and
motivation for selecting IE topics and the extent to which IEs are aligned with country, sector or
project strategies of the WBG and its clients. This will require understanding the factors behind
the adoption of IEs as explained by the following sub-questions:
Who and what reasons (operational, analytic, sector strategies) motivated the IE?
What kinds of decisions, if any, did it plan to inform (depending on the purpose, timing
and questions of the IE)?
Who and what factors determine decisions regarding the design and conduct of the IE
as well as the analytical questions examined in the evaluation?
Is the information provided by the IEs aligned with the purposes/objectives of the
projects, WBG‟s and countries‟ strategies, and learning priorities?
What is the ideal coverage of IEs? Which projects and topics for which IE would be
helpful are not being evaluated?
The answers to these questions will be obtained from IE reports as well as surveys and
interviews with the IE teams. They will then be compared with the priorities of the WBG and its
clients as outlined in the PADs/PDSs, PRSPs, CASs, and sector board strategies to detect
alignment and gaps.
B. Quality of Impact Evaluations
13. To what extent and why (or why not) do impact evaluations meet the expected quality
standards and address the questions of interest? The question refers to the second step in the
process chain – production of the impact evaluation. It aims to assess the quality of IE techniques
and scope in order to better inform the discussion on their use and influence. As mentioned
before, there are no „best-practice standards‟ and this evaluation does not attempt to rate or rank
IEs according to their technical quality. It will, however, employ existing guidelines (for
instance, the ones produced by 3IE and NONIE) as well as the guidelines employed by recent
IEG‟s meta-reviews of impact evaluations in nutrition and safety nets. For this purpose the
following sub-questions are pertinent:
What techniques did the IE use to attribute the impact to the intervention? Are these
appropriate methods to attribute causality of impacts?
Did the IE examine the robustness of their findings to methodological issues and other
aspects of the program and context that may affect the internal validity of the analysis?
Which other aspects of program impacts were captured and how (channels that explain
the impacts, distribution of impacts, cost effectiveness, sustainability of impacts)?
7
Are IEs seen to contribute much more than other analytic and evaluative tools?
14. The answers to these questions will be derived mainly from thorough desk reviews of IE
reports and documents related to the IEs (if they are ongoing). They will be linked with
information from the first question to gauge how the techniques and findings are appropriate for
the motivation of the IEs. They will also expose factors inherent to IEs that may explain the
extent to which they are relevant, utilized, and influential. Undertaking an accurate cost-
effectiveness analysis of IEs is very difficult due to the complexities of measuring both their
direct and indirect costs and benefits (multiple sources of financial support, a general lack of
data, and the difficulty of attributing benefits or financial returns to IEs). While not a full cost-
effectiveness analysis, the report will explore what sources of information about costs are
available and examine the perceived contribution of IEs relative to other evaluation approaches
from the client's and other perspectives.
C. Use and Influence of Impact Evaluations
15. To what extent and why (or why not) are impact evaluations used to influence
development practice? The question refers to the third step in the process chain – dissemination
and uptake of the impact evaluation. It aims to measure if there has been a strategic uptake of IEs
in decision making by the WBG and its clients, and, if not, what are the constraints. IEs are often
expected to be used for three broad functions: (1) as a project assessment tool to evaluate
performance, influence design and operational decisions of the evaluated program, and provide
an objective basis for shared accountability; (2) for knowledge generation beyond the program
evaluated to provide evidence about what works and does not work in development, why, for
whom, and under which conditions; and (3) to promote evaluation capacity and evidence-based
decision making mechanisms by emphasizing high-quality data collection and strong technical
skills (Annex 1).18
In assessing these three functions and the factors that may underline the
applicability of IEs, this report seeks to answer the following questions:
To what extent and how have IEs contributed to decisions throughout the life cycle of
the project, from design (choice of beneficiaries, delivery mechanisms, timing for data
collection) to implementation to completion (decisions to continue, expand, modify or
terminate, and funding decisions)?
To what extent and how have IEs encouraged learning to improve future projects and
contributed to the institutional strategies and policy debates of the WBG and its clients?
What are the „feedback loops‟ in WBG processes that support this learning function?
To what extent and how have IEs and their related activities (workshops, guidance
notes, IE clinics) contributed to promoting the capacity and demand for M&E?
What factors (motivation, methodological quality, active engagement with counterparts,
timing, focus on stakeholder‟s priority issues, dissemination, interest of decision
makers, etc.) are associated with the use and influence of IEs? 19
The answers to these questions will be acquired from project document reviews, structured
surveys with the IE teams and WBG staff related to the evaluated projects, interviews with WBG
clients in the country, and sector case studies.
8
DATA AND METHODOLOGY
16. In order to answer the above questions, this evaluation will employ five evaluation tools:
(1) desk review of impact evaluations; (2) desk review of relevant project documents; (3)
electronic surveys of evaluators and Bank staff related to the evaluated project; (4) interviews
with internal and external stakeholders in selected countries; and (5) sector case studies. (Annex
2 details how these tools serve to answer each sub-question).
Desk review of impact evaluation reports: The team has compiled a database of IEs
(completed and ongoing) supported by the WBG to serve as the basis of the evaluation. Since
IE is a decentralized effort in the WBG, it is difficult to generate a comprehensive list. The
bulk of the database is from the DIME initiative, which documents IEs across the WB, and is
complemented with the IE list from the IFC. In order to ensure that the inventory is close to
being exhaustive, the team has conducted different searches in Business Warehouse, Image
Bank, and Operations Portal/ I-docs. The compiled database reveals that there are 329 IEs of
WB projects since 2000 (119 completed and 210 ongoing) and they tend to concentrate in
certain sectors and regions (Annex 3).20
The reports of all the completed IEs will be reviewed
to extract information for the questions regarding the relevance and quality of IEs
(motivation and objectives, timeliness, involved actors, factors affecting the design,
methodology, and findings).
Desk review of project documents and policy notes: A random sample of
approximately 40 WB projects corresponding to 50-60 completed IEs and 20 WB projects
associated to 30 ongoing IEs will be selected for this in-depth document review (Annex 4
describes the sampling methodology). In the case of IFC all IEs (approximately 25) will be
included in the analysis.21
The documents reviewed will include PADs/PDS of the evaluated
and follow-up projects (to understand the objectives of the projects, if the IE was foreseen in
the project, and if the IE findings contributed to the design of the follow-up project), ISRs,
ICRs/PCRs of the evaluated projects (to check if IE findings were cited in these
assessments), as well as PPARs, PRSs, CASs, and sector board strategies (to provide context
for the priorities of stakeholders). Results from this exercise will shed light on whether the
IEs are aligned with the project, country, and/or sector strategies of the WBG and its clients,
as well as whether the IEs have been used for project assessment, knowledge generation, and
M&E promotion. Information from the review will be matched with the surveys and
interviews to create a more comprehensive picture of the IEs and related projects. Finally, a
subsample of Bank projects in a specific sector will be selected to review project documents
and determine the actual and ideal scope conceptually feasible for impact evaluation work in
connection with these projects.
Surveys: For the same randomly selected sample of projects with completed and/or
ongoing IEs, the team will distribute structured surveys to IE authors and WBG staff related
to the evaluated projects. The survey questions for IE authors seek to gain information about
the motivation and costs of a specific IE and their awareness (with concrete verifiable
examples) of the IE use and influence. The survey questions to WBG project staff will ask
their view as users (with quantitative ratings) about the usefulness and influence of the IE
related to a specific project as well as factors that may explain this (Annex 5).22
Survey
questions will be designed in a neutral way such that they do not prompt the respondents to
9
overstate the importance of IEs. If one person is related to more than one IE or project, they
will be asked about only one, which is chosen randomly. To increase a likely low response
rate, the team will actively follow up and contact senior management to ensure cooperation.
Interviews with internal and external stakeholders in selected countries: These
interviews aim to provide a country perspective (from both the WBG and clients) across IE
applications and a detailed evaluative view of their development policy impact. For this
selection of IEs, countries have been ranked according to the total number of completed IEs.
The top 10 are Nicaragua, Peru, Indonesia, Colombia, China, India, Vietnam, Brazil,
Ecuador, and Bangladesh (with at least 5 completed IEs). Due to time and budget constraints,
countries and interviewees will be purposely selected through the following process. First,
four to six of the top 10 countries will be selected to ensure coverage of most regions and that
each country offers cases of completed and ongoing IEs for which it is possible to
systematically apply a consistent set of questions. An extra project with an IFC-supported IE
with learning complementarities to WB‟s case studies will be identified. Second, within each
country, one or two main sectors will be selected to minimize overstretching the interview
team. The goal of the exercise is to interview IE authors, WBG staff and clients (for example,
government officials and private sector representatives) related to the evaluated projects
(they will be excluded from the survey) about aspects aligned with the three main questions
outlined in the previous section. In addition, the team will conduct structured interviews with
sector managers, business line leaders/product specialists, M&E staff and stakeholders
benefited from other IE-related activities supported by the WBG (e.g. workshops, guidance
notes and IE clinics) to capture their views on relevance and usefulness of IEs and related
products. The findings from these interviews will feed into the conclusions and
recommendations for the overall evaluation and provide concrete examples of cases in which
IEs are (or not) used and influential in different country contexts.
Sector Case studies: Sector case studies will complement the analysis through
examining specific sector IE experience across different country circumstances and
applications. One of the studies will be in a sector with a significant amount of evidence
(likely education) and another in a sector where the IE evidence base is starting to grow (for
instance, health and/or water and sanitation). The case study structure is outlined in Annex 6.
TEAM
17. The joint evaluation will be under the guidance of Mark Sundberg (Manager, IEGPS) and
Stoyan Tenev (Chief Evaluation Officer, IEGPR). The evaluation team will be managed by
Javier E. Baez (Economist, IEGPS) and Izlem Yenice (Evaluation Officer, IEGPR), and include
Andaleeb Alam, Carmem Domigues, and Tu Chi Nguyen (IEGPS), as well as other consultants
to be identified. The team will be assisted by Yezena Zemene Yimer.
TIMELINE
18. The approach paper was reviewed at the IEG “one stop” on November 9, 2010 and will
be submitted to CODE for review in January 2011. The evaluation will be reviewed within IEG
and by peer reviewers by July 15, 2011. It will be submitted to CODE by mid-September 2011.
10
Figure 2. Proposed Work Program
Activities 2010 2011
Oct Nov Dec Jan Feb Mar Apr May June July Aug Sep
Background work and drafting of approach
paper
One-stop review of approach paper
CODE review of approach paper
Review of impact evaluations
Desk review of project documents
Design, launch, and track of surveys
Analysis of surveys
Selection and design of country-based
interviews
Interviews in Washington DC
Field-based interviews
Drafting of sector case studies
Drafting of evaluation report
Review by IEG management and peer
reviewers
Review by WBG management
Final modification and submission to
CODE
PEER REVIEW
19. Peer reviewers for the evaluation are David McKenzie (Senior Economist, DECFP,
World Bank) and Howard White (Executive Director, International Initiative for Impact
Evaluation).
11
Annex 1
Conceptualizing Use and Influence of IE
The literature on theory of evaluation utilization, in general, identifies four types of uses: instrumental
use (making decisions about programs based on evaluation results); conceptual use (affecting how
people think about development issues); strategic use (persuading others or using evaluation findings
to reach particular outcomes); and process use (promoting evaluation activities).23
This evaluation
groups the different functions of IEs under three categories to align their potential use and influence
with the WBG‟s results, knowledge, and M&E agendas outlined above:
I. Project Assessment: Within the evaluated project, IEs are claimed to serve as an assessment
tool. This may be used by various actors, within and outside the WBG, to assess performance and
influence design and operational decisions at the following phases:
Project design: Incorporating a prospective IE into a program may affect decisions regarding
the design of a project in aspects such as targeting of beneficiaries, type of benefits, delivery
mechanisms, and timing and quality of data collection, among others. IEs may also help identify
other related development issues to be addressed, evaluate pilot projects, and test different design
features to improve subsequent phases of the project.
Project implementation: Knowledge generated along the production of an IE may be useful
for strengthening the project‟s implementation. For example, data collected for the IE could
provide information for a process review that reveals areas if and where implementation is not
carried out as planned, hence allowing needed in-time correction.
Project completion and assessment: The results from IEs could demonstrate the impacts (or
lack thereof) of the program, which could be central to operational decisions to continue, modify,
expand or terminate the evaluated project. In addition, if combined with information about costs,
IE could provide an opportunity to assess the cost-effectiveness of programs, contributing to more
efficient resource allocation.
II. Knowledge Generation: Beyond the project evaluated, evidence from IEs may also
contribute to knowledge generation for enhancing development practices. As IEs are being
conducted on an increasing number of sectors and countries, the evidence from IEs could contribute
to build the stock of knowledge, which may be argued to help inform decision makers of future
projects and the development community in general.24
Future project design and implementation: When designing and implementing a new
project, managers/policymakers may take into consideration lessons learned from IEs of other
projects (for example, what approaches have worked in similar settings).
Bank’s strategies, dialogues, and policy decisions: As the results of IEs are disseminated,
they add to the general pool of knowledge of development challenges and solutions, which may
help motivate debates and dialogues, establish best practices, influence Bank‟s strategies
(including resource allocation), and promote evidence-based policymaking.
III. Evaluation Promotion: IE has the potential role of contributing to develop capacity for and
institutionalizing evidence-based evaluations, both within the WBG and at the country/sector level.
IEs are supposed to help narrow the gap between the demand for and supply of knowledge on
development effectiveness, both directly and indirectly by supporting and supplementing the
production of other evaluations.25
Conducting IEs could promote the capacity for evaluations,
12
including the collection of systematic high quality data, specialized skills, and demand for follow-up
or new evaluations.26
Within the WBG: As the number of IEs of Bank-supported projects grows, it is expected
that a higher fraction of WBG staff will learn the skills needed to carry out sound IEs.
Furthermore, the inputs used by IEs such as rich data and information systems are often claimed
to be beneficial for other M&E and analytical purposes undertaken by the WBG.
At the country/sector level: The Bank‟s promotion of IEs produced in a collaborative
manner with local counterparts such as policymakers, private sector, local researchers, and
development institutions (as well as NGOs and civil society) may help transfer the necessary
technical skills, internalize the value of IEs, understand their limitations and costs, align M&E
and data collection strategies, and build and strengthen evaluation capacity at the country/sector
level.
13
Use & Influence Flowchart
Function 2 – Knowledge Generation
Function 1 - Evaluated Project
Function 3 - Evaluation Promotion
Project Design Project Implementation Project Completion and Assessment
Decision to Conduct
Prospective IEs
IE Production (data collection, analysis
undertaken)
IE Results (prospective or ex-
post)
Future Project Design and
Implementation
WBG’s Strategies, Countries’
Policies, Policy Dialogues
WBG’s M&E Capacity and Framework
Country’s M&E Capacity and Framework
External Stakeholders
Acquire Information
Extent and means of interaction between evaluators and project
managers; Timeliness of IE relative to decision;
Bank’s policies and incentives, stakeholder incentives;
IE quality and credibility; Relevance of results
Extent and means of result dissemination;
Institutional and external
stakeholder incentives;
IE quality and credibility;
Relevance of results
Extent and means of IE dissemination;
Bank’s policies and incentives; IE quality and credibility
Factors that may affect the use and influence of IEs
14
Annex 2 - Evaluation Design Matrix
Evaluation Questions Sub-questions Information Sources and
Collection Methods
Relevance of IEs: To what
extent and why (or why
not) are IEs relevant to
close knowledge gaps and
aligned with WBG’s and
clients’ priorities and
strategies?
Who and what motivated the IE? IE reports
Surveys with IE teams
Interviews with selected IE
teams
Priorities of the WBG and its
clients as outlined in the
PADs/PDS, PRSP, CAS, and
sector board strategies
What kinds of decisions, if any, did it plan to inform (depending on the purpose, timing
and questions of the IE)?
Who and what factors were involved in decisions regarding the design and conduct of the
IE?
Quality of IEs: To what
extent and why (or why
not) do IEs meet the
expected quality standards
and address the questions
of interest?
What techniques did the IE use to attribute the impact to the intervention? Are these
appropriate methods to attribute causality of impacts?
IE reports
Documents related to the IEs (if
they are ongoing) Do the IEs examine the robustness of their findings to methodological issues and other
aspects of the program and context that may affect the internal validity of the analysis?
Which other aspects of program impacts were captured and how (distribution of impacts,
cost effectiveness, sustainability of impacts)?
How cost-effective have IEs been? Are they seen to contribute much more than other
analytic and evaluative tools?
Use and Influence of IEs:
To what extent and why (or
why not) are IEs used to
influence development
practice?
Project assessment tool: To what extent and how have IEs contributed to decisions
throughout the life cycle of the project, from design (choice of beneficiaries, delivery
mechanisms, timing for data collection) to implementation to completion (decisions to
continue, expand, modify or terminate, and funding decisions)?
Project document reviews
Surveys with IE teams and WBG
staff related to the evaluated
projects
Interviews with WBG staff and
clients in selected countries
Sector case studies
Knowledge generation: To what extent and how have IEs encouraged learning to improve
future projects and contributed to the institutional strategies and policy debates of the
WBG and its clients?
Evaluation capacity and evidence-based decision making: To what extent and how have
IEs and their related activities (workshops, guidance notes, IE clinics) contributed to
promoting the capacity and demand of M&E for the WBG and its clients?
Factors affecting use and influence: What factors (methodological quality, active
engagement with counterparts, timing, focus on priority stakeholder issues, dissemination,
interest of decision makers, etc) are associated with the use and influence of IEs?
15
0
10
20
30
40
50
60
70
19
80
19
91
19
92
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
An
alysi
s in
pro
gre
ss
Fo
llo
w-u
p d
ata
coll
ecte
d
Bas
elin
e d
ata
coll
ecte
d
Eval
uat
ion
des
ign
ed
Un
der
dis
cuss
ion
Completed Ongoing
# o
f IE
s
WB-suported IEs of WB Projects per Completion Status
and Year
Period evaluated
Annex 3
Summary of the Preliminary Database (including WB-supported IEs of World Bank projects, not including IFC-supported IEs)
Note: The years shown are the years in which IEs were completed
0
20
40
60
80
100
FPD HDN PREM SDN
# o
f IE
s
WB-suported IEs since 2000 of WB
Projects per Network (completed and
ongoing)
Completed Ongoing
0
20
40
60
80
100
120
Africa East Asia
and the
Pacific
Europe
and
Central Asia
Latin
America
and the Caribbean
Middle
East and
North Africa
South
Asia
# o
f IE
s
WB-suported IEs since 2000 of WB
Projects per Region (completed and
ongoing)
Completed Ongoing
16
Annex 4
Sampling Methodology
Data: The WB database consists of 329 IEs linked to 260 World Bank operations. There are
120 completed IEs that correspond to 80 operations, while the 209 ongoing IEs are linked to
189 Bank operations. Of these 189 operations, 5 have already had an IE that was completed
in the past decade. Project-based breakdown of IEs is as follows:
Number of IEs Per Project 1 2 3 4 5 Number of Projects 216 27* 12 2 3**
* 4 of these projects have both one ongoing IE and one completed IE
**1 of these projects has one ongoing IE and 4 completed IEs
Sample Design: For the purposes of this evaluation, a stratified random sample of projects
will be studied. The rationale for sampling on projects as opposed to IEs is because within a
particular project, the effect of one IE cannot be isolated from another and hence, all IEs
within the project need to be analyzed. No design effects or non-response effects are assumed
in sample size calculations.
Universe: The universe consists of 260 projects linked to 329 IEs in the database
Domains: The universe of projects is classified into two mutually exclusive domains: (1) 80
projects with at least one completed IE (2) 184 projects with ongoing IEs but no prior
completed IE. Sample size for each domain is determined separately to ensure
representativity at the domain level. For the completed IEs domain, the sample is significant
at a 95 percent confidence level with ±10 percent margin of error. Given the higher number
of projects in the second domain, the focus of this evaluation on completed IEs, and the time
and resource constraints, the sample of ongoing IEs is significant at a 95 percent confidence
level with ±20 percent margin of error.
Stratification: The two domains are stratified into (1) Projects with only one IE, and (2)
Projects with multiple IEs. In the completed IE domain, there are 55 projects with single IEs
that have been completed, and 21 projects with multiple IEs at least one of which is
completed. In the ongoing IE domain, 16 projects have multiple ongoing IEs (and no prior
completed IEs), while 168 projects have single ongoing IEs (and no prior completed IEs).
Each stratum is further stratified into SDN (Sustainable Development Network), HDN
(Human Development Network), FPD (Finance and Private Sector Development Network),
and PREM (Poverty Reduction and Economic Management Network) on the basis of the
sectoral distribution of the projects and its correspondence with network. A random sample
of projects will be selected from each strata.
Sample Allocation: The sample size is allocated to each of the 8 strata based on Probability
Propotional to Size method. Although the incidence of PREM projects is low compared to
SDN, HDN and FPD projects, time and resource constraints do not allow for oversampling
PREM projects to ensure sufficient power for comparing PREM with other networks.
17
Annex 5
Surveys Structure
Surveys will target a wide range of key players to Impact Evaluation within the Bank. Since different
types of actors have different responsibilities and experiences in conducting IEs, the questions posed
to each of them will vary according to their roles. Survey questions will be grouped along themes or
topic areas, and will be asked to the relevant actors, with the possibility of a given topic area being
addressed by multiple types of actors. Some tentative groupings with a sample of questions of each
section are described below (multiple choice answers or ratings where applicable will also be
included in the actual surveys).
Actors Groups of relevant questions asked
TTLs Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9
Evaluators Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9
Country Directors Q1, Q4, Q5, Q6, Q7, Q9
Sector Managers Q1, Q4, Q5, Q6, Q7, Q9
Question
Group
Topic area of
questions Sample questions Information for
Q1 Decision and
support to
conduct the IE
What led/contributed to the initiation of the IE?
Who supported the IE being conducted?
What was the purpose of conducting the IE?
Relevance
Factors that influence
utilization
Q2 Production of
IEs and
linkages with
the project
Who was in charge of designing/implementing the IE?
What were the major factors that affected the design and
implementation of the IE?
What was the time lag between project implementation and the
IE?
Relevance
Function 1 (use of IE as
an assessment tool)
Factors that influence
utilization
Q3 Funding of IE Who funded the IE?
Did WBG supervision pay any researcher time? Relevance
Factors that influence
utilization
Q4 Results and
dissemination
When were the results available in relation to the timing of
decisions regarding the project?
Who was responsible for disseminating the results?
Through what channels were the results disseminated?
Function 2 (use of IE
for knowledge
generation)
Factors that influence
utilization
Q5 Knowledge
generation
Did this IE contribute new information to the development
knowledge base? Function 2
Factors that influence
utilization
Q6 Use and
influence
Have the results of the evaluation been utilized? If so, by
whom? And in what ways?
Did the IE contribute to the wider WBG Strategy? If so, how?
Did the IE contribute to the policies of the country? If so, how?
What were the factors that affected their use and influence?
Function 1
Function 2
Factors that influence
utilization
Q7 Promotion of
evidence-based
decision
making
How have IEs contributed to improvements in data collection,
staff evaluation capacity, institutionalization of evaluation
efforts?
Function 3 (use of IE
for M&E capacity)
Factors that influence
utilization
Q8 Relationship
among actors
How was the sharing of results between the implementers and
the WBG team?
Was the relationship between evaluators and the operational
team a smooth one?
Function 2
Function 3
Factors that influence
utilization
Q9 Lessons from
conducting the
IE
What lessons would you point to for your colleagues who
would like to incorporate an IE into their projects?
Have you ever tried to implement an IE into another WBG
project and not succeed? If so, which one? Why did it not
succeed? What were the lessons from that?
Factors that influence
utilization
Lessons and
recommendations
18
Annex 6
Sector Case Study Structure
In order to ensure that the team can make comparisons between case studies in an accurate fashion,
both case studies will follow a specified format (with room for certain adaptations to accommodate
inherent differences between sectors) and will be based on a structured set of questions that will guide
team members in their review of relevant documents and conduct of interviews. The final report that
will be generated for each case study is expected to follow the format specified below.
1. SECTOR BACKGROUND
This section describes the sector context including policymaking, development issues, motivation for
projects, WBG‟s engagement and M&E system in terms of the structure, resources and key actors.
The point of this section is to provide a background for how impact evaluations may fit in.
Policymaking context. This sub-section describes the broad policymaking context in the sector, such
as major actors and their interests, and what factors influence the policymaking process.
Background of the development issues/motivation for the projects. This sub-section describes the
main development issues within the sector, what the government has been doing to address them, and
the main motivations for the conduct of the projects evaluated.
Bank’s engagement. Describe the involvement of the WBG in the given sector, including the size of
its portfolio and type of WBG engagement, including with policymakers.
2. BACKGROUND ON IMPACT EVALUATION
This section describes all the completed, planned, or ongoing impact evaluations supported by the
World Bank Group in the sector. The purpose is to provide an overview of the evidence from impact
evaluations available for the sector as well as potential for the evidence to be used.
Motivation for impact evaluations. This sub-section describes what led to the conduct of the IEs,
including the purpose, key players, and alignment with Bank‟s and countries‟ priorities. It analyses
whether the results of the IEs were made available to make informed operational decisions about the
programs evaluated.
Impact evaluation design and implementation. This sub-section describes the design and
implementation of the IEs, including the kind of involvement the WBG and the country have in the
IEs, and whether there were issues in program implementation that had implications for evaluations
and vice-versa.
Impact evaluation findings and dissemination. This sub-section describes the results generated by
the IEs, as well as their dissemination both within the WBG and the sector where the IEs were
conducted.
3. ANALYSIS AND ASSESSMENT
This section analyzes the application of IEs in the WBG‟s operations and the country‟s policies and
what factors can make them more relevant and effective.
19
Applicability of IEs for WBG’s operations. This sub-section describes how IEs are used within the
WBG to influence project design and implementation.
Knowledge generation. This sub-section describes how IEs stimulate critical thinking to identify and
test the relevant and influential policy and operational questions in highest demand from governments
and the WBG.
Determinants behind the utilization of IEs. This sub-section describes what factors explain the
prevalence (or lack of) IEs in the sector.
4. LESSONS LEARNED AND RECOMMENDATIONS
This section synthesizes the findings of the case study, and makes recommendations to different
actors on how to improve the use and influence of IEs.
Lessons learned. This part draws conclusions about the extent to which IEs are relevant, utilized and
influential, and the factors that affect their relevance and applicability within the WBG and in the
sector context.
Recommendations for enhancing IE relevance and effectiveness. This sub-section makes
recommendations to evaluators, project managers, country directors, policy makers, and other
relevant actors for improving the relevance, take-up and effectiveness of IEs based on the analysis
above.
20
REFERENCES
Center for Global Development. 2006. “When Will We Ever Learn? Improving Lives through Impact
Evaluation.” Washington, DC: Center for Global Development.
Cummings, R. 2002. “Rethinking Evaluation Use”. Australasian Evaluation Society International Conference
Fiszbein, A. 2006. “Development Impact Evaluation: New Trends and Challenges.” In Evidence and Policy
2(3): 385-393.
Jones, N., C. Walsh, H. Jones, and C. Tincati. 2008. “Improving Impact Evaluation Coordination and Uptake: A
Scoping Study Commissioned by the DFID Evaluation Department on behalf of NONIE.” London, UK:
Overseas Development Institute.
Overseas Development Institute. 2009. “Improving Impact Evaluation Production and Use.” Working Paper
300. London, UK: Overseas Development Institute.
Ravallion, M. 2009. “Evaluation in the Practice of Development.” In The World Bank Research Observer 24(1).
Sandison, P. 2005. “The Utilization of Evaluations.” In ALNAP Review of Humanitarian Action. London, UK:
ALNAP.
Shadish, William R. & Cook, Thomas D. & Leviton, Laura C. 1991. Foundations of program evaluation :
theories of practice / by William R. Shadish, Jr., Thomas D. Cook, Laura C. Leviton Sage Publications,
Newbury Park, CA
Thomas, V. 2009. “Evaluation to Help Improve Development Results.” Washington, DC: The World Bank.
World Bank. 2009. “Making Smart Policy: Using Impact Evaluation for Policy Making – Case Studies on
Evaluations that Influenced Policy.” Doing Impact Evaluation No. 14, Thematic Group on Poverty Analysis,
Monitoring and Impact Evaluation. Washington, DC: The World Bank.
World Bank-DIME. 2010. “Development Impact Evaluation Initiative: A World Bank-Wide Strategic
Approach to Enhance Developmental Effectiveness.”, DIME, Mimeo. Washington, DC: The World Bank.
World Bank - IEG. 2010. “Analyzing the Effects of Policy Reforms on the Poor: An Evaluation of the
Effectiveness of World Bank Support to Poverty and Social Impact Analysis.” Washington, DC: The World
Bank.
___. 2008. “Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank
Economic and Sector Work and Technical Assistance, 2000-2006.” Washington, DC: The World Bank.
___. 2003. “Sharing Knowledge: Innovations and Remaining Challenge.” Washington, DC: The World Bank.
Worthen B., J. Sanders, and J. Fitzpatrick. 1997. “Program Evaluation: Alternative Approaches and Practical
Guidelines”. Longman, N.Y, USA.
21
ENDNOTES 1. OPCS, Impact Evaluations at the World Bank: Guidance Note
2. Thomas, 2009; CGD, 2006; Fiszbein, 2006
3. World Bank-DIME, 2010
4. This is a conservative estimated based on the DIME Strategy report (During fiscal 2000-2004, the Bank spent $160,000 a
year on impact evaluation in total. After DIME was established, spending rose to $4 million a year in fiscal 2005-2008,
evenly split between internal and trust fund resources. Since DIME was re-launched as a Bank-wide initiative and with the
maturing of the IE portfolio, spending tripled to $13-14 million a year, mostly financed by trust funds).
5. World Bank-IEG, 2010, 2008, 2003
6. The 2000-10 period covers 92 percent of all completed IEs. The team excludes IEs completed earlier because it will be
difficult to track their use and influence. IEs that do not evaluate WBG-supported projects were excluded given that the
evaluation tools available to the team will not be able to document the contribution of these IEs to the operational aspects of
projects that had no Bank involvement (function 1 of third question).
7. World Bank-IEG website
8. Quasi-experimental techniques include propensity score matching, difference-in-differences, regression discontinuity,
instrumental variable, and structural modeling.
9. World Bank-DIME, 2010
10. World Bank-AIM website
11. World Bank-SIEF website
12. James Adams, OPCS VP, Announcement July 14, 2005
13.World Bank-Poverty Impact Evaluation Website
14. Results agenda website
15. Knowledge agenda website
16. World Bank-DIME, 2010; World Bank-IEG website
17. Throughout the evaluation other factors not mentioned here may emerge as influencing one of the three parts of the
theory of change, and if this is the case, they will be duly incorporated to explain how IEs have been relevant and effective
in informing development practices.
18. World Bank-DIME, 2010; World Bank, 2009; Sandison, 2005; ODI, 2009; World Bank-IEG website
19. In fact, the literature identifies some characteristics that may hinder or facilitate IE use. Some factors include: (i) the
methodological quality of the evaluation and credibility of the international evaluators; (ii) timing and focus on priority
stakeholder issues; (iii) identifying users and change agents early in the evaluation; (iv) frequent contacts with users; (v)
providing interim results; (vi) translating findings into actions and recommendations; (vii) disseminating results through
informal meetings, oral briefs, non-technical summaries; (viii) challenging fundamental assumptions about problems and
policies and by circulating results through networks of opinion makers; (ix) extent to which incentives are aligned with
efficacy; (x) the interest of decision makers and community in the evaluation, (xi) active engagement with counterparts; (xii)
the extent to which findings can go against entrenched interests or prevailing ideology; (xiii) constraints on decision-makers
related to budget constraints, past decisions and conflicting demands.
20. The sectors that have seen the highest uptake are in human development, especially social protection and labor,
education, and health programs, followed by programs in the agriculture and rural development and finance sectors.
Geographically, sub-Saharan Africa is where the highest number of IEs has taken place, followed by Latin America, then
South and East Asia respectively. Most IEs used nonexperimental methods. It is difficult, however, to determine the degree
of Bank involvement in each of these IEs, which could vary from funding to technical assistance to implementation.
21. The sample of completed IEs is significant at a 95 percent confidence interval at ± 10 percent margin of error and the
sample of completed IEs is significant at a 95 percent confidence interval at ± 20 percent margin of error.
22. The database will provide information about the staff related to the IEs and evaluated projects, including evaluators,
TTLs, Sector Managers, Country Directors, and other key players involved in the projects or sectors being evaluated.
23. Cumming, 2002; Shadish, William, Thomas Cook and Laura Leviton, 1991; Worthen, Sanders and Fitzpatrick, 1997
24. ODI, 2009; Sandison, 2005; IEG website
25. CGD, 2006; Ravallion, 2009
26. World Bank-DIME, 2010; World Bank, 2009