1 different methods of impact evaluation. how to measure impact? assessing causality impact of...
TRANSCRIPT
1
Different Methods of Impact Evaluation
How to measure impact?
• Assessing causality
Impact of program = outcome 1 – outcome 2In practice, we compare two groups, one of
which benefited from the program, the other one did not 2
Event 1e.g. Education
program/ treatment
Event 2 (Effects)
Outcome 1Causes
Event 2 occurs if and only if Event 1 occurred before
Event 1No Education
Program
Event 2Outcome 2Causes
Constructing the counterfactual
• The counterfactual– what would have happened in the absence of the
program? …– … for the people who benefitted from the program: we
don’t observe it – thus, impact evaluation will have to mimic it
• Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation
• Methods differ by the way the counterfactual is constructed
3
Non-Experimental Methods
4
Non-Experimental Methods
1. Simple Difference2. Multivariate (Multiple) Regression3. “Difference in difference” (Multiple
Regression with Panel Data)4. Matching5. Randomization/RCT’s (advanced topic
covered separately)
5
Simple difference
• Simple difference is a first measure – why is it not sufficient?
• There may be differences between the two groups (age, location, gender, initial endowment, bargaining power)
• So we may want to control for these differences
6
Multi-variate regression
• Suppose we can observe these differences.– Age composition of the group, initial infrastructure
in the school, level of education, …
• We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +…– A regression provides the linear combination of
observable variables that best “mimics” the outcome
– Each coefficient represents the effect of each variable
– Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal
7
Multi-variate regression
• Problems of the regression– You may want to include many many variables,
to control for as many characteristics as possible– Problem of sample size (degrees of freedom)
• More important: do you have measures of everything?– Bargaining power, Pro-activeness, Intrinsic
motivation, hopelessness
• There are unobservables8
Panel data
• Simple difference: before/after– Counterfactual = same group before the program– Can we trust this? Assumption = would have
remained the same– Ex: Police project
• Double difference– Control for the situation before the program– Ex: Group 1 = Treatment Group; Group 2 = Control
Group– 2006: Group 1 : 30 Group 2 : 60
2009: Group 1 : 50 (+66%) Group 2 : 90 (+50%)Effect = +16%
- Assumption: they would have grown at the same pace- Not sure…
9
Matching
• We compare pairs of 2 individuals for which the values taken by ALL variables are the same
10
Matching
• Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile”
• Problems– This matching method requires a big dataset:
find pairs on a sufficient number of variables– What about unobservables?
11
An example
• Case study: US Congress elections, 2002– 60 000 phone calls to potential voters to
encourage them to vote; 25 000 reached– Outcome: did they actually go vote?– 1st method: compare the 25 000 (reached) vs. the
35 000 (not reached)– 2nd method: introduce co-variates in a regression– 3rd method: introduce baseline data (vote in 1998)– 4th method: do a matching
12
1st comparison: we suspect a selection bias
Reached Not Reached
Difference
Female 56.2% 53.8% 2.4 pp*Newly Regist. 7.3% 9.6% -2.3 pp*From Iowa 54.7% 46.7% 8.0 pp*
Voted in 2000
71.7% 63.3% 8.3 pp*
Voted in 1998
46.6% 37.6% 9.0 pp*
13
Selection Bias: differences in observable / unobservable characteristics → differences in
outcome not due to the treatment
Non-Experimental Methods
14
Method Estimated Impact
1 – Simple Difference 10.8 pp *
2 – Multiple regression 6.1 pp *
3 – Multiple regression with panel data (diff-in-diff)
4.5 pp *
4 – Matching 2.8 pp *
5 – Randomized Experiment 0.4 pp
• For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References
15