africa program for education impact evaluation dakar, senegal december 15-19, 2008 experimental...
Post on 21-Jan-2016
215 Views
Preview:
TRANSCRIPT
Africa Program for Education Impact EvaluationDakar, SenegalDecember 15-19, 2008
Experimental Methods
Muna MekyEconomist
Africa Impact Evaluation Initiative
2
Motivation
• Objective in evaluation is to estimate the CAUSAL effect of intervention X on outcome Y– What is the effect of a housing upgrade on
household income?
• For causal inference, we need to understand exactly how benefits are distributed– Assigned / targeted– Take-up
3
Causation versus Correlation
• Correlation is NOT causation– Necessary but not sufficient condition– Correlation: X and Y are related
• Change in X is related to a change in Y
• And….
• A change in Y is related to a change in X
– Example: age and income
– Causation – if we change X how much does Y change• A change in X is related to a change in Y
• Not necessarily the other way around
4
Causation versus Correlation
Three criteria for causation:
– Independent variable precedes the dependent variable.
– Independent variable is related to the dependent variable.
– There are no third variables that could explain why the independent variable is related to the dependent variable.
5
• Statistical analysis: Typically involves inferring the causal relationship between X and Y from observational data– Many challenges & complex statistics
– We never know if we’re measuring the true impact
• Impact Evaluation: – Retrospectively:
• same challenges as statistical analysis
– Prospectively:• we generate the data ourselves through the program’s design
evaluation design• makes things much easier!
Statistical Analysis & Impact Evaluation
6
How to assess impact
• What is the effect of a housing upgrade on household income?
• Ideally, compare same individual with & without programs at same point in time
• What’s the problem?
• The need for a good counterfactual– What are the requirements?
7
Case study: housing upgrade
• Informal settlement of 15,000 households
• Goal: upgrade housing of residents
• Evaluation question:
What is the impact of upgrading housing on household income? on employment?
• Counterfeit counterfactuals
8
Gold standard:Experimental design
• Only method that ensures balance in unobserved (and observed) characteristics Only difference is treatment
• Equal chance of assignment into treatment and control for everyone
• With large sample, all characteristics average out
• Experimental design = Randomized evaluation
9
“Random”
• What does the term “random” mean here?– Equal chance of participation for everyone
• How could one really randomize in the case of housing upgrading?
• Options– Lottery– Lottery among the qualified– Phase-in– Encouragement– Randomize across treatments
10
Kinds of randomization
• Random selection: external validity– Ensure that the results in the sample represent the
results in the population – What does this program tell us that we can apply to
the whole country?
• Random assignment: internal validity– Ensure that the observed effect on the outcome is
due to the treatment rather than other factors – Does not inform scale-up without assumptions
• Example: Housing upgrade in Western Cape vs Sample from across country
11
Randomization
Randomization
External Validity
(sample)
Internal Validity
(identification)
External vs Internal
12
Example of Randomization
• What is the impact of providing free books to students on test scores?
• Randomly assign a group of school children to either:- Treatment Group – receives free books
- Control Group – does not receive free books
13
Randomization
Random Assignment
14
How Do You Randomize?
1) At what level? – Individual – Group
• School• Community • District
15
When would you use randomization?
• Universe of eligible individuals typically larger than available resources at a single point in time– Fair and transparent way to assign benefits– Gives an equal chance to everyone in the sample
• Good times to randomize:– Pilot programs– Programs with budget/capacity constraints – Phase in programs
16
Basic Setup of an Experimental Evaluation
Target Population
Potential Participants
Evaluation Sample
Random Assignment
TreatmentGroup
ControlGroup
Participants No-Shows Based on Orr (1999)
All informal settlement dwellers
Communities that might participate or a targeted sub-group
Select those you want to work with right now
17
Examples…
18
Beyond simple random assignment
• Assigning to multiple treatment groups– Treatment 1, Treatment 2, Control– Upgrade housing in situ, relocation to better housing,
control– What do we learn?
• Assigning to units other than individuals or households– Health Centers (bed net distribution)– Schools (teacher absenteeism project)– Local Governments (corruption project)– Villages (Community-driven development projects)
19
Unit of randomization
• Individual or household randomization is lowest cost option
• Randomizing at higher levels requires much bigger samples: within-group correlation
• Political challenges to unequal treatment within a community– But look for creative solutions: e.g., uniforms in Kenya
• Some programs can only be implemented at a higher level – e.g., strengthening school committees
20
Efficacy & Effectiveness
• Efficacy– Proof of Concept– Pilot under ideal conditions
• Effectiveness – At scale– Normal circumstances & capabilities– Lower or higher impact?– Higher or lower costs?
21
Advantages of experiments
• Clear causal impact
• Relative to other studies– Much easier to analyze– Cheaper! (smaller sample sizes)– Easier to convey– More convincing to policymakers– Not methodologically controversial
22
What if randomization isn’t possible?
It probably is…• Budget constraints: randomize among the
needy
• Roll-out capacity: randomize who receives first
• Randomly promote the program to some
23
When is it really not possible?
• The treatment has already been assigned and announced
and no possibility for expansion of treatment
• The program is over (retrospective)
• Universal eligibility and universal access– Example: free education, exchange rate
regime
top related