The Evaluation GapAn International Initiative to Build Knowledge

Presentation byWilliam D. Savedoff

Sr. Partner, Social InsightProject Director, Center for Global Development

Meeting of the DAC Evaluation NetworkParis

November 10, 2004

The Evaluation GapOverview

• Who are we?

• What do we mean by an evaluation gap?

• Why does this evaluation gap occur?

• What other initiatives are being taken?

• What are we doing?

• Where are we going?

Who are we?

Funding

Project Director

Secretariat

Working Group Members

Members do not represent their institutions but participate in their individual capacity, bringing a wide range of experience from:

• Evaluation Offices• NGOs• Development Agencies• Universities• Philanthropic Foundations

Who are we?Who are we?

Working Group Members

• Nancy Birdsall• Francois Bourguignon• Esther Duflo• Paul Gertler• Judith Gueron• Indrani Gupta• Jean-Pierre Habicht • Dean Jamison• Patience Kuruneri

• Ruth Levine• Richard Manning • Stephen Quick• William D. Savedoff• Raj Shah• Smita Singh• Miguel Szekely• Cesar Victora

Who are we?Who are we?

What is “the evaluation gap”?Range of Analytical Work

• Project preparation studies

• Monitoring implementation

• Process & Operational evaluation

• Outcome evaluation

• Impact evaluation

Who are we?What is the gap?

Impact Evaluation

Studies that measure changes in a target population that can be attributed to a

particular program or policy

Who are we?What is the gap?

Importance of Attribution

0

5

10

15

20

Period

HIV

Pre

va

len

ce

(%

)

• Different studies answer different questions

• Attribution should not be ignored

• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …

First RoundBeneficiaryCommunities

Importance of Attribution

0

5

10

15

20

Period

HIV

Pre

va

len

ce

(%

)

• Different studies answer different questions

• Attribution should not be ignored

• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …

First RoundBeneficiaryCommunities

Second RoundCommunities

Results from Millions Saved Cases of public health interventions

Excluded• 27 b/c impact could

not be documented• 12 b/c too early or

small scale

Included• 17 identified and

documented

What is the gap?

N

Results from Millions Saved Cases of public health interventions

“The gap in evaluation inhibits the documentation of successes, and prevents policymakers from being able to tell the difference between a well told story and a hard fact as they make decisions about which programs to support.”

What is the gap?

Community Health Insurance

• Community Health Insurance was proposed as early as 1978 in a WHO Technical Expert Report

• What have we learned about community health insurance in the 36 years that have followed?

What is the gap?

Community Health InsuranceILO/Universitas Review

What is the gap?

127 Studies of 258 Cases

24 Cases Measured Utilization

Two Cases Had Internal Validity

None of the studies measured impact on health

Community Health InsuranceEkman, Health Policy & Planning (2004)

What is the gap?

Descriptive Non-Statistical

(22)

Other (2)

Multivariatestatistical (5)

Descriptive (14)Review of studies that analyzed:1. Resource Mobilization2. Financial Protection

Why does this gap occur?

• Knowledge from impact evaluations is a public good

• Costs are visible, benefits seem far-off & resources are limited

• Demand is diffuse (institutions & time)• “It Pays to be Ignorant” • Low-quality studies crowd out the good• Methodological challenges

Why?

Guarded optimism

• Changing profile of agency staff• Worldwide capacity for good studies• Recognition of impact evaluation value• Pressure from skeptical donors • Methodological & practical advances in research

design (also highly visible & successful)– PROGRESA/Oportunidades, IMCI, Guinea Worm,

etc.

Why?

Other Initiatives

• Increasing access to existing information

• Developing aggregate indicators

• Improving capacity

• Promoting evaluation w/funds & data

• Conducting research & demonstrating good evaluation practices

Other Initiatives

Some Examples

• OECD/DAC Evaluation Network

• World Bank Research Department

• Health Metrics Network

• UN M.E.R.G.

• USAID/MACRO Surveys

• MIT Poverty Action Lab

Other Initiatives

We need an initiative that:

• Focuses specifically on the Public Good aspect of impact evaluation

• Develops a collective response to the problem

• Mobilizes & appropriately channels new funds

• Acts selectively where the most can be learned

The Evaluation GapWorking Group Process

• Preliminary interviews & research

• Convene working group

• Meetings, teleconference, and e-list debates

• Consultation group

• Draft paper & action plan

• Dissemination and broader debate

What are we doing?

Likely characteristics of recommendations

• Identifying a reliable source of funding

• Establishing collective mechanism for selecting “enduring questions”, guidance, involvement & commitment

• Developing institutional mechanisms for channeling funds into appropriate studies & projects

Where are we going?

What do you think?

• What is the fundamental problem from your perspective?

• Do you have other examples (reviews?)

• What are your current initiatives?

• What kinds of solutions should be considered?

• Who else should we consult?

Where are we going?

Contact Us

Center for Global Developmentwww.cgdev.org

• Nancy Birdsall, Presidentnbirdsall@cgdev.org

• Ruth Levine, Senior Fellowrlevine@cgdev.org

• William D. Savedoff, Project Directorsavedoff@socialinsight.org

Where are we going?

References

• Christensen, Jon. Asking the Do-Gooders to Prove They Do Good. The New York Times . 1-3-2004.

• Development Assistance Committee. Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1991. Paris, OECD. 2004.

• Development Assistance Committee. Review of the DAC Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1-120. 1998. Paris, OECD. 2004.

• Development Assistance Committee. Glossary of Key Terms in Evaluation and Results Based Management. 6, 1-37. 2002. Paris, OECD. Evaluation and Aid Effectiveness.

References

• Dugger, Celia. “World Bank Challenged: Are the Poor Really Helped?” The New York Times . 7-28-2004. New York.

• Ekman, Björn. "Community-based health insurance in low-income countries: a systematic review of the evidence." Health Policy and Planning, 2004, 19 (5), 249-270.

• France, Ministère de l'Economie des Finances et de l'Industrie. Partners in Development Evaluation: Learning and Accountability. Partners in Development Evaluation: Learning and Accountability. 3-25-2003. Paris.

References

• International Labour Office. 2002. “Extending Social Protection in Health Through Community Based Health Organizations: Evidence and Challenges”, Dicussion Paper, Universitas Programme, ILO, Geneva.

• Kremer, Michael. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons." American Economic Review Papers and Proceedings, 2003, 93 (2), 102-115.

• Pritchett, Lant. "It Pays to Be Ignorant: A Simple Political Economy of Rigorous Program Evaluation." The Journal of Policy Reform, December 2002, 5 (4), 251-269.

References

• Victora, Cesar G, Habicht, Jean-Pierre, Bryce, Jennifer. "Evidence-Based Public Health: Moving Beyond Randomized Trials." American Journal of Public Health, March 2004, 94 (3), 400-405.

• World Bank. Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs. 1-24. 2004. Washington, DC, World Bank.

• World Health Organization. Financing of Health Services. 625. 1978. Geneva. Technical Report Series.

EG Working Group Members• Nancy Birdsall, President, Center for Global Development• Francois Bourguignon, Chief Economist & Sr. Vice President, World Bank• Esther Duflo, Associate Professor of Economics, MIT • Paul Gertler, Professor of Economics, Haas School of Business • Judith Gueron, President, MDRC• Indrani Gupta, Reader, Institute of Economic Growth• Jean-Pierre Habicht, Professor, Cornell University • Dean Jamison, Senior Fellow, National Institutes of Health• Patience Kuruneri, Senior Policy Analyst, World Health Organization• Ruth Levine, Senior Fellow, Center for Global Development• Richard Manning, Chair, Development Assistance Committee • Stephen Quick, Director, Inter-American Development Bank• William D. Savedoff, Senior Partner, Social Insight• Raj Shah, Senior Policy Officer & Senior Economist, Bill & Melinda Gates Foundation• Smita Singh, Special Advisor for Global Affairs, William & Flora Hewlett Foundation• Miguel Szekely, Ministry of Social Development, Mexico• Cesar Victora, Professor, Universidade Federal de Pelotas, Brazil