![Page 1: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/1.jpg)
CAUSAL INFERENCE
Shwetlena Sabarwal
Africa Program for Education Impact EvaluationAccra, Ghana, May 2010
![Page 2: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/2.jpg)
Motivation
2
Goal of any evaluation is to estimate the causal effect of intervention X on outcome Y.
Example: does an education intervention improve test scores (learning)?
Reducing class size Teacher training In-school nutrition
![Page 3: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/3.jpg)
Causation is not correlation!
3
Any two variable (X and Y) can move together1. Male teachers & academic performance of
students.2. Health and income.
But, they may have nothing to do with each other.
Other explanations?
![Page 4: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/4.jpg)
Evaluation problem: Potential Outcomes Approach
Ideal way to evaluate the impact of an intervention:observe agent in and out of program, at a
point in time.
But, think about the only way in which we can evaluate the impacts of an intervention: observe agent in or out of program, at any
point in time.
![Page 5: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/5.jpg)
How to assess causality?
Let Y= outcome of interest (test score)P= participation in program = 1 if
in= 0 if out
Formally, program impact is:
α = (Y | P=1) - (Y | P=0)
Program Impact: difference in outcomes for individuals in and out of program.
Outcome w/ program
Outcome w/out program
![Page 6: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/6.jpg)
Another Way to Think of Evaluation Problem
The problem we face is that: (Y | P=0) is not observed for program
participants. (Y | P=1) is not observed for non-participants
Missing Data Problem:
Counterfactual not observed.
what would have happened to agent without the intervention?
![Page 7: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/7.jpg)
Solving the evaluation problem
Generate the counterfactual find a control or comparison observation
for agent facing the intervention.
Criteria for selecting comparison observation:1.Observationally similar, at baseline (and
after intervention).
2.Face same contemporaneous “shocks” as the treatment group.
![Page 8: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/8.jpg)
“Counterfeit” Counterfactuals1. Before and after:
Same individual before the treatment
2. Non-Participants:Those who choose not to enroll in
programThose who were not offered the program
![Page 9: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/9.jpg)
“Counterfeit” CounterfactualNumber 1: Before and After
9
Consider how you might evaluate an agricultural assistance program. Suppose program offers free/subsidized fertilizer. Compare rice yields before and after
Q: If you find no change in rice yield, can you conclude the program failed? What else changed?
Drought? Lots of rainfall?
![Page 10: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/10.jpg)
Scholarship Program and School Enrollment, Before and After
Time
YAfter
A
B
t-1 t
O
Before
Ultimate goal is to estimate α
(Yit | P=1) - (Yi,t| P=0)
Estimate the impact on treated individuals:
"A-O"=(Yi,t| P=1) - (Yi,t-1| P=1)
Second, estimate counterfactual
"B-O"=(Yi,t| P=0) - (Yi,t-1| P=0)
“Impact” = A-B
α’
![Page 11: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/11.jpg)
Scholarship Program and School Enrollment, Before and After
Time
YAfter
A
B
t-1 t
O
Before
But, impact "A-B" may misrepresent true counterfactual.
Suppose C is the correct counterfactual.
Here, the impact of the intervention is "A-C".
α’’
C
![Page 12: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/12.jpg)
“Counterfeit” CounterfactualNumber 2: Non-Participants….
12
Compare non-participants to participants
Counterfactual: non-participant outcomes
Impact estimate: αi = (Yit | P=1) - (Yj,t| P=0)
Assumption: (Yj,t| P=0) = (Yi,t| P=0)
Issue: why did the j’s not participate?
![Page 13: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/13.jpg)
Non-participants Example : Job Training and Employment
13
Compare employment & earning of individuals who sign up for training to those who do not.
Who signs up?
Those who are most likely to benefit, i.e. those with more ability
Would have higher earnings than non-participants without job training
Poor estimate of counterfactual
![Page 14: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/14.jpg)
Non-participants Example 2: Health Insurance and Demand for Medical Care14
Compare health care utilization (# doctor visits) of those who got insurance to those who did not.
But, who buys insurance? those who expect large medical expenditures
(unhealthy)
Those who do not buy insurance have less need for medical care.
Poor estimate of counterfactual
![Page 15: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/15.jpg)
The problem is selection bias.
15
Selection bias: People choose to participate in program for specific reasons.
Problem occurs when reasons for participation are related to the outcome of interest: Job Training: ability and earning Health Insurance: health status and medical-
care utilization.
Cannot separately identify impact of the program from these other factors/reasons
![Page 16: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/16.jpg)
Need to know…16
Know all reasons why someone gets the program and others not
reasons why individuals are in the treatment versus control group
If reasons correlated w/ outcome cannot identify/separate program impact
from other explanations of differences in outcomes
![Page 17: CAUSAL INFERENCE Shwetlena Sabarwal Africa Program for Education Impact Evaluation Accra, Ghana, May 2010](https://reader035.vdocuments.us/reader035/viewer/2022062804/56649ed15503460f94bdf764/html5/thumbnails/17.jpg)
Possible Solutions…17
We need to understand the data generation process How beneficiaries are selected and how
benefits are assigned
Guarantee comparability of treatment and control groups, so ONLY unaccounted for difference is the intervention.