![Page 1: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/1.jpg)
![Page 2: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/2.jpg)
3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions
![Page 3: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/3.jpg)
3ie
Methodological developments Results agenda› US: Government Results and Performance Act, GRPA -
1993 – USAID adopted six ‘strategic development goals’ e.g. broad-based economic growth, defining outcome indicators for each e.g. per capita growth
› UK: Public Service Agreements and Service Delivery Agreements (very similar to Report Cards and Development Indicators in South Africa)
MDGs› Focus on outcomes› Raises questions about attribution
![Page 4: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/4.jpg)
3ie
Goals were ‘so broad, and progress affected by many factors other than USAID programmes, that the indicators cannot realistically serve as measures of the agency’s specific efforts’
In 2000 USAID abandoned the goal outcome measures as a basis for monitoring its performance
![Page 5: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/5.jpg)
3ie Timescale, exacerbated by data lags Decentralized programs not aligned with
higher-level objectives Data quality Impact?
Development agencies may not know exactly what impact their efforts are having given the wide range of other agencies and external political, economic and social factors involved.
![Page 6: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/6.jpg)
3ie
Use existing data systems, but with a view to data quality and timeliness
Invest in improving existing data systems rather than making new (often parallel) ones
Know the advantages and limitations of different sorts of data (administrative versus survey data)
Avoid indicator proliferation Don’t be all M and no E
![Page 7: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/7.jpg)
3ie
Outcome monitoring is not a valid basis for rigorous performance measurement
… hence the need for impact evaluation
![Page 8: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/8.jpg)
3ie International agency response
› World Bank: DIME and IEG› Inter-American Development Bank*
At country level response strongest in Latin America› Progressa› In Mexico and Colombia social programs
legally require impact evaluations to secure funding
Note: * See IDS Bulletin March 2008 for discussion
![Page 9: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/9.jpg)
3ie
After decades in which development agencies have disbursed billions of dollars for social programs, and developing countrygovernments and nongovernmental organizations (NGOs) have spent hundreds of billions more, it is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs.
![Page 10: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/10.jpg)
3ie UNICEF review (1995) found 15% had
‘impact assessments’ but ‘many evaluations were unable to properly assess impact because of methodological shortcomings’
ILO review of 127 studies of 258 community health programs found only 2 which could give robust conclusion on impact on access
Recent (2008) NORAD review found many reports draw impact conclusions on basis of weak evidence (or none at all)
![Page 11: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/11.jpg)
3ie
Billions of dollars are being spent with no idea if they are the best intervention to achieve the desired outcome – or even if the help achieve it at all. Rigorous impact evaluation – and only rigorous impact evaluation – will provide the evidence needed for optimal resource allocation
![Page 12: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/12.jpg)
3ie When will we ever learn?
› An approach which can attribute change in outcomes to the intervention
› Requires a well-defined control group› Strongest design is developed at project design
stage› Where feasible, a randomized approach should be
considered Poverty action lab
› Active promotion of randomized control trials (RCTs)› Evaluations in conjunction with NGOs especially in
India and Kenya
![Page 13: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/13.jpg)
3ie
Joint AfrEA, NONIE, 3ie impact evaluation conference Cairo May 2009
NONIE 3ie
History Origins in DAC meeting November 2006
Evaluation gap working group
Purpose Demand generationGuidanceCoordination
Promote quality impact studies, including financingAdvocacy
Membership Evaluation networks Governments and NGOsIE practitioners are associates
Engagement OPSC represented Sign up, membership and studies
![Page 14: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/14.jpg)
3ie Websites
› World Bank (PREM, IEG and NONIE)› Poverty Action Lab› 3ie – www.3ieimpact.org
Training courses› PAL› IPDET› Africa Impact Evaluation Initiative
Opportunities to conduct IEs› DIME› 3ie› Own resources
![Page 15: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/15.jpg)
3ie Each year two themes and open window Themes being determined – see 3ie website for
details and to submit proposed questions Southern-driven, issues-based with southern-led
evaluation teams Under each of these three windows
› 10-12 ‘quick wins’› 6-8 large studies› 6 baseline studies› 6 synthetic reviews
![Page 16: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/16.jpg)
3ie Only promoting RCTs – not true RCTs seen as gold standard – not exactly true,
issues-led not methods-led, but where feasible an RCT should certainly be considered (best available method)
A positivist approach – true, but evidence-based policy making is an inherently positivist approach
Another northern initiative – partly true, but 3ie offers great potential for (1) Southern ownership of evaluation agenda, (2) harmonization around quality standards
![Page 17: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/17.jpg)
3ie
The current policy framework in South Africa implicitly demands quality impact evaluations, and resources are available to help meet the supply
![Page 18: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/18.jpg)
3ie The move to performance-based management
leads to questions being asked about attribution Impact evaluation is the approach to be adopted
to address attribution Impact evaluation has to adopt certain technical
standards if it is to provide proper evidence The new international initiatives exist to support
efforts to expand programs of impact evaluation
![Page 19: 3ie Context: the results agenda The call for evidence Rigorous impact evaluation The international response Reactions](https://reader035.vdocuments.us/reader035/viewer/2022070412/56649eb25503460f94bb88b6/html5/thumbnails/19.jpg)
3ie
VISITwww.3ieimpact.org