Howard White

International Initiative for Impact Evaluation (3ie)

An impact evaluation seeks to attribute all, or part, of the observed change in outcomes to a specific intervention.

Level Indicators

Inputs Resources: Funds and personnel

Activities Teacher trainingSchool improvementsDecentralized management and local school management committees

Outputs Trained teachersBetter school facilitiesFunctioning school management committees

Intermediate outcomes Higher school enrolments at all levelsTeacher and parent satisfactionBetter managed schools

Final outcomes Improved learning outcomes

Impact Higher productivity and earningsEmpowerment

Focus on final welfare outcomes, e.g. Infant mortality Income povertySecurity

Usually long-term, but not necessarily so (but then sustainability is an issue)

Projects (or specific interventions) Individual projects are the ‘back bone’ of impact

analysis But even then may only be able to do rigorous

impact analysis of some components Programmes

Sector wide programs can be conceived of as supporting a range of interventions, many of which can be subject to rigorous impact evaluation.

Policies In general different approaches are required,

such as CGEs – these are not being discussed today

Pick a named intervention for an impact evaluation and make a short list of indicators (using the log frame) for evaluation of this intervention

Before After

Project (treatment)

66

Control

But we don’t know if they were similar before… though there are ways of doing this

Before After

Project (treatment) 66

Control 55

Sometimes this can work … but usually not

Before After

Project (treatment)

40 66

Control

Before After

Project (treatment) 40 66

Control 44 55

Ex ante design preferred to ex post: impact evaluation design is much stronger if baseline data are available (but may still be possible even if they are not)

Means collecting data before intervention starts, and can be affecting the design of the intervention

But can sometimes use secondary data, that is an existing survey

1. Confounding factors

2. Selection effects

3. Spillovers and contagion

4. Impact heterogeneity

5. Ensuring policy relevance

Other things happen – so before versus after rarely sufficient

So get a control group… but different things may happen there

So collect data on more than just outcome and impact indicators

And collect baseline data But …

Program placement and self-selection Program beneficiaries have particular

characteristics correlated with outcomes – so impact estimates are biased

Need to use experimental or quasi-experimental methods to cope with this; this is what has been meant by rigorous impact evaluation

But it is just one facet of impact evaluation design

Other things can also bias impact estimates

Experimental (randomized):Limited application, but there are

applications and it is a powerful approachMany concerns (e.g. budget and ethics) and

not valid Quasi-experimental design (regression

based):Propensity score matching is most commonRegression discontinuity Interrupted time series Regression modelling of outcomes

Spillover – positive and negative impacts on non-beneficiaries

Contagion – similar interventions in control areas

Need to collect data on these aspects and may need to revise evaluation design

WHAT ARE THE MAJOR CONFOUNDING FACTORS FOR YOUR OUTCOME AND IMPACT INDICATORS?

HOW MIGHT SELECTION BIAS, SPILLOVER AND CONTAGION AFFECT THE EVALUATION OF THE INTERVENTION YOU HAVE SELECTED?

Impact varies by intervention (design),

beneficiary and context

‘Averages’ can be misleading

Strong implications for evaluation

design

Is the impact of X and Y, bigger, equal to or less than the impacts of doing X and Y separately?

For example, hygiene promotion and sanitation facilities

Evidence suggestions they are substitutes- either one reduces incidence child diarrhea by 40-50%, but not by more if the two are combined

Irreparable damage to physical and cognitive development results from nutritional deprivation in the first two years of life

Hence interventions to infants have greater long-run impact on many outcomes than do those aimed at older children (such as school feeding programs)

Expected impact

Good year Low

Average-bad year High

Very bad year None

What sort of differences in impact would you expect for your intervention with respect to intervention (design), context and beneficiary?

ProcessStakeholder engagementPackaging messages

DesignTheory-based approachMixed methodsCapture all costs and benefits, including

cross-sectoral effects Cost effectiveness and CBA

Make explicit underlying theory about how inputs lead to intended outcomes and impacts

Documents every step in causal chain Draws on multiple data sources and

approaches Stresses context of why or why not

working

Assumption Findings

Provide nutritional counselling to care givers

Mothers are not decision makers, especially if they live with their mother-in-law

Women know about sessions and attend

90% participation, lower in more conservative areas

Malnourished and growth faltering children correctly identified

No – community nutrition practitioners cannot interpret growth charts

Women acquire knowledge Those attending training do so

And knowledge is turned into practice

No there is a substantial knowledge-practice gap

Supplementary feeding is additional food for intended beneficiary

No, considerable evidence of substitution and leakage

Adopted changes are sufficient to improve intended outcomes

Only sometimes (not for pregnant women)

Need to collect survey data at the unit of intervention (child, firm etc)

Will need also facility/project data Need data across the log frame and for

confounding factors – and for your instrumental variables (lack of valid instruments is the major obstacle to performing IE)

Designing data collection instruments takes time and should be iterated with qualitative data

Study Data sources

Rural electrification 3 rural electrification surveys11 DHS2 LSMS

India irrigation and rural livelihoods

Own surveyDistrict-level government dataCensus data

Bangladesh maternal and child health and nutrition

DHSProject data + national nutrition survey

Ghana basic education 1988/89 GLSS (LSMS)Own follow up survey

Kenya agricultural extension 2 previous rural surveysOwn follow up survey

OUTLINE YOUR PROPOSED EVALUATION DESIGN (TIMING OF DATA COLLECTION, IDENTIFICATION OF CONTROL, IF ANY)

WHAT DATA SOURCES WOULD YOU USE FOR YOUR PROPOSED EVALUATION?

VISIT

WWW.3IEIMPACT.ORG