impact evaluation methods regression discontinuity design and difference in differences
DESCRIPTION
Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences. Slides by Paul J. Gertler & Sebastian Martinez. Measuring Impact. Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (diff in diff) Other options. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/1.jpg)
Impact Evaluation MethodsRegression Discontinuity Design and
Difference in Differences
Slides by Paul J. Gertler & Sebastian Martinez
![Page 2: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/2.jpg)
2
Measuring Impact
• Experimental design/randomization
• Quasi-experiments
– Regression Discontinuity
– Double differences (diff in diff)
– Other options
![Page 3: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/3.jpg)
3
Case 4: Regression Discontinuity
• Assignment to treatment is based on a clearly defined index or parameter with a known cutoff for eligibility
• RD is possible when units can be ordered along a quantifiable dimension which is systematically related to the assignment of treatment
• The effect is measured at the discontinuity – estimated impact around the cutoff may not generalize to entire population
![Page 4: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/4.jpg)
4
• Anti-poverty programs targeted to households below a given poverty index
• Pension programs targeted to population above a certain age
• Scholarships targeted to students with high scores on standardized test
• CDD Programs awarded to NGOs that achieve highest scores
Indexes are common in targeting of social programs
![Page 5: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/5.jpg)
5
• Target transfer to poorest households
• Construct poverty index from 1 to 100 with pre-intervention characteristics
• Households with a score <=50 are poor
• Households with a score >50 are non-poor
• Cash transfer to poor households
• Measure outcomes (i.e. consumption) before and after transfer
Example: Effect of Cash Transfer on Consumption
![Page 6: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/6.jpg)
6
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
![Page 7: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/7.jpg)
7
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
Non-Poor
Poor
![Page 8: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/8.jpg)
8
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
![Page 9: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/9.jpg)
9
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
Treatment Effect
![Page 10: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/10.jpg)
10
• Oportunidades assigned benefits based on a poverty index
• Where
• Treatment = 1 if score <=750
• Treatment = 0 if score >750
Case 4: Regression Discontinuity
![Page 11: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/11.jpg)
11
Case 4: Regression Discontinuity
Fitt
ed v
alu
es
puntaje estimado en focalizacion276 1294
153.578
379.224
2
Baseline – No treatment
0 1 ( )i i iy Treatment score
![Page 12: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/12.jpg)
12
Estimated Impact on CPC
** Significant at 1% level
Case 4 - Regression DiscontinuityMultivariate Linear Regression
30.58**(5.93)
Fitt
ed v
alu
es
puntaje estimado en focalizacion276 1294
183.647
399.51 Treatment Period
Case 4: Regression Discontinuity
![Page 13: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/13.jpg)
13
Potential Disadvantages of RD• Local average treatment effects – not always
generalizable
• Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size
• Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: – Nonlinear Relationships– Interactions
![Page 14: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/14.jpg)
14
Advantages of RD for Evaluation
• RD yields an unbiased estimate of treatment effect at the discontinuity
• Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy
– No need to “exclude” a group of eligible households/individuals from treatment
![Page 15: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/15.jpg)
15
Measuring Impact
• Experimental design/randomization
• Quasi-experiments
– Regression Discontinuity
– Double differences (Diff in diff)
– Other options
![Page 16: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/16.jpg)
16
Case 5: Diff in diff
• Compare change in outcomes between treatments and non-treatment
– Impact is the difference in the change in outcomes
• Impact = (Yt1-Yt0) - (Yc1-Yc0)
![Page 17: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/17.jpg)
17
TimeTreatment
Outcome
Treatment Group
Control Group
Average Treatment Effect
![Page 18: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/18.jpg)
18
TimeTreatment
Outcome
Treatment Group
Control Group
Estimated Average Treatment Effect
Average Treatment Effect
![Page 19: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/19.jpg)
19
Diff in Diff
• Fundamental assumption that trends (slopes) are the same in treatments and controls
• Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)
![Page 20: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/20.jpg)
20
Not Enrolled Enrolled t-statMean ΔCPC 8.26 35.92 10.31
Case 5 - Diff in Diff
Linear Regression Multivariate Linear Regression
Estimated Impact on CPC 27.66** 25.53**(2.68) (2.77)
** Significant at 1% level
Case 5 - Diff in Diff
Case 5: Diff in Diff
![Page 21: Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences](https://reader035.vdocuments.us/reader035/viewer/2022062423/5681474b550346895db48d42/html5/thumbnails/21.jpg)
21
Case 1 - Before and After
Case 2 - Enrolled/Not
Enrolled
Case 3 - Randomization
Case 4 - Regression
Discontinuity
Case 5 - Diff in Diff
Multivariate Linear
RegressionMultivariate Linear
Regression
Multivariate Linear
Regression
Multivariate Linear
Regression
Multivariate Linear
Regression
Estimated Impact on CPC 34.28** -4.15 29.79** 30.58** 25.53**
(2.11) (4.05) (3.00) (5.93) (2.77)** Significant at 1% level
Impact Evaluation Example – Summary of Results