the future of monitoring and evaluation: lessons from a decade of impact evaluations

The Future of Monitoring and Evaluation: Lessons from a Decade of Impact Evaluations

Eric Foster-MooreThe World Bank | Africa Water Resources

efostermoore@worldbank.org

Development works

Our project will directly benefit 400,000 people…

Our project will benefit the people of Mozambique…

Number of women benefitted

Growth rate in country X increased by 2 percentage points

Art credit: @ChrisPlanicka

“No it won’t.”

“$195 billion in aid per year for 20 years would

eliminate poverty.”

Jeffrey SachsBig Push, Millennium Villages Project, The End of Poverty

William Easterly

The White Man’s BurdenDead Aid by Dambisa MoyoCan the West Save Africa?The white band’s burden…

1. Is aid effective?2. How do we know? 3. How do we apply this to our own work?

Three questions

Boone (1996): Nope.

Does aid increase growth? A macro perspective

Burnside and Dollar (2002). It depends. Effective when combined with “good policy.”

Easterly, Levine, and Roodman (2003). Not really.

Original Burnside-Dollar results

Easterly, Levine, and Roodman

Boone (1996): Nope.

Does aid increase growth? A macro perspective

Burnside and Dollar (2002). It depends. Effective when combined with “good policy.”

Easterly, Levine, and Roodman (2003). Not really.

Clemens, Radelet, Bhavnani, and Bazzi (2012). It still depends.

(Chris Blattman)

What about at the micro level?

Often just make them up: “This project will directly benefit 300,000 people.”

Move beyond correlations. Establish causality. Do an experiment. Some are natural, some are contrived

Natural experiment: half of a community displaced by a reservoir

Contrived experiment: the Fiala (2013) paper

Grants, loans, training, or a combo?

Fiala (2013):

Stimulating Microenterprise Growth: Results from a Loans, Grants, and Training Experiment in Uganda

Loans Loans plus training Cash Cash plus training Control

Treatment groups:

Follow up surveys after 6 and 9 months.

Loan only: effect is gone after nine months

Loans and training: 54 percent increase in profits for men Grant only: no effect whatsoever. For men, Fiala speculates, “Knowing that

the loan had to be repaid appears to have led them to use the money more effectively in the business.”

Grants and training: no effect

Women: no positive effects. “Family pressure on women appears to have significant negative effects on business investment decisions: married women with family nearby perform worse than the control group in a number of the interventions.”

Results

What is the effect of institutions on economic performance?

Acemoglu, Johnson, and Robinson (2000): The Colonial Origins of Comparative Development

Methods

Stronginstitutions

Strong economic performance

Other stuff

Instrumental variable

Hypothesis

“Differences in institutions account for roughly three quarters of the differences in income per capita.”

Not driven by outliers such as the United States, Canada, Australia, etc.

Not driven by simply being on the African continent.

Not driven by distance from the equator (i.e. geography).

Results

Compare program alternatives Quantify effects of program X in place Z at time T Force you to identify and test assumptions Microfinance, cash transfers, education, health, job

training, etc.

What RCTs do really well:

What you see: Policy and legal frameworks

strengthened Cash collection ratio; debts repaid Number of people in project area with

access to “Improved Water Sources” Percent of biological samples failing Number of direct beneficiaries, percent

of which are female Number of staff trained

Monitoring and evaluation: some examples

What you don’t see: Household income Business profitability Governance Strength of political parties Comparison to other

interventions Opportunity cost Articulation of assumptions

1. There’s an entire sector of projects that don’t get tested and where assumptions that are probably decades old dominate. Let’s bring the lessons of impact evaluation to these projects.

2. Need to be strategic about their application in order to learn something about human behavior.

Two problems with RCTs

1. Is aid effective? 2. How do we know? 3. How do we apply this to our own work?

Three questions

Backup

Solow model of growth

From E-L-R (2003)

From AJR