shawn cole harvard business school threats and analysis
Post on 12-Jan-2016
231 Views
Preview:
TRANSCRIPT
Shawn ColeHarvard Business School
Threats and Analysis
Course Overview
• What is Evaluation?• Measuring Impacts• Why Randomize?• How to Randomize• Sampling and Sample Size• Threats and Analysis• Project from Start to Finish
Course Overview
• What is Evaluation?• Measuring Impacts• Why Randomize?• How to Randomize• Sampling and Sample Size• Threats and Analysis• Project from Start to Finish
Lecture Overview
• Attrition• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
Lecture Overview
• Attrition• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
Attrition
• Is it a problem if some of the people in the experiment vanish before you collect your data?– It is a problem if the type of people
who disappear is correlated with the treatment.
• Why is it a problem?• Why should we expect this to
happen?
Attrition bias: an example
• The problem you want to address:– Some children don’t come to school because they
are too weak (undernourished)• You start a school feeding program and want
to do an evaluation– You have a treatment and a control group
• Weak, stunted children start going to school more if they live next to a treatment school
• First impact of your program: increased enrollment.
• In addition, you want to measure the impact on child’s growth– Second outcome of interest: Weight of children
• You go to all the schools (treatment and control) and measure everyone who is in school on a given day
• Will the treatment-control difference in weight be over-stated or understated?
Before Treatment After Treament
T C T C
20 20 22 2025 25 27 2530 30 32 30
Ave.
Difference Difference
Before Treatment After Treament
T C T C
20 20 22 2025 25 27 2530 30 32 30
Ave. 25 25 27 25
Difference 0 Difference 2
What if only children > 21 Kg come to school?
What if only children > 21 Kg come to school?
Before Treatment After Treament
T C T C
20 20 22 2025 25 27 2530 30 32 30
A. Will you underestimate the impact?
B. Will you overestimate the impact?
C. NeitherD. AmbiguousE. Don’t know
A. B. C. D. E.
19%
52%
10%10%10%
Before Treatment After TreamentT C T C
[absent] [absent] 22 [absent]25 25 27 2530 30 32 30
Ave. 27.5 27.5 27 27.5
Difference 0 Difference -0.5
What if only children > 21 Kg come to school?
When is attrition not a problem?
A. B. C. D. E.
0%
35%
40%
0%
25%
A. When it is less than 25% of the original sample
B. When it happens in the same proportion in both groups
C. When it is correlated with treatment assignment
D. All of the aboveE. None of the above
When is attrition not a problem?
A. B. C. D. E.
0%
9%
83%
0%
9%
A. When it is less than 25% of the original sample
B. When it happens in the same proportion in both groups
C. When it is correlated with treatment assignment
D. All of the aboveE. None of the above
Attrition Bias
• Devote resources to tracking participants after they leave the program
• If there is still attrition, check that it is not different in treatment and control. Is that enough?
• Also check that it is not correlated with observables.
• Try to bound the extent of the bias– suppose everyone who dropped out from
the treatment got the lowest score that anyone got; suppose everyone who dropped out of control got the highest score that anyone got…
– Why does this help?
Lecture Overview
• Attrition• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
What else could go wrong?
Target Populati
on
Not in evaluation
Evaluation Sample
TotalPopulation
Random Assignment
Treatment Group
Control Group
Spillovers, contamination
Target Populati
on
Not in evaluation
Evaluation Sample
TotalPopulation
Random Assignment
Treatment Group
Control Group
Treatment
Spillovers, contamination
Target Populati
on
Not in evaluation
Evaluation Sample
TotalPopulation
Random Assignment
Treatment Group
Control Group
Treatment
Example: Vaccination for chicken pox
• Suppose you randomize chicken pox vaccinations within schools– Suppose that prevents the
transmission of disease, what problems does this create for evaluation?
– Suppose externalities are local? How can we measure total impact?
Externalities Within School
Without ExternalitiesSchool ATreated? OutcomePupil 1 Yes no chicken poxTotal in Treatment with chicken poxPupil 2 No chicken pox Total in Control with chicken poxPupil 3 Yes no chicken poxPupil 4 No chicken pox Treament EffectPupil 5 Yes no chicken poxPupil 6 No chicken pox
With ExternalitiesSuppose, because prevalence is lower, some children are not re-infected with chicken pox
School ATreated? OutcomePupil 1 Yes no chicken poxTotal in Treatment with chicken poxPupil 2 No no chicken poxTotal in Control with chicken poxPupil 3 Yes no chicken poxPupil 4 No chicken pox Treatment EffectPupil 5 Yes no chicken poxPupil 6 No chicken pox
0%
100
%-100%
0%67%
-67%
Externalities Within School
Without ExternalitiesSchool ATreated? OutcomePupil 1 Yes no chicken poxTotal in Treatment with chicken poxPupil 2 No chicken pox Total in Control with chicken poxPupil 3 Yes no chicken poxPupil 4 No chicken pox Treament EffectPupil 5 Yes no chicken poxPupil 6 No chicken pox
With ExternalitiesSuppose, because prevalence is lower, some children are not re-infected with chicken pox
School ATreated? OutcomePupil 1 Yes no chicken poxTotal in Treatment with chicken poxPupil 2 No no chicken poxTotal in Control with chicken poxPupil 3 Yes no chicken poxPupil 4 No chicken pox Treatment EffectPupil 5 Yes no chicken poxPupil 6 No chicken pox
How to measure program impact in the presence of spillovers?
• Design the unit of randomization so that it encompasses the spillovers
• If we expect externalities that are all within school:– Randomization at the level of the
school allows for estimation of the overall effect
Example: Price Information
• Providing farmers with spot and futures price information by mobile phone
• Should we expect spilloves?• Randomize: individual or village level?• Village level randomization
– Less statistical power– “Purer control groups”
• Individual level randomization– More statistical power (if spillovers small)– Ability to measure spillovers
Example: Price Information
• Can we do both?• Randomly assign villages into one of four groups,
A, B, C, and D• Group A Villages
– SMS price information to all individuals with phones
• Group B Villages– SMS price information to randomly selected 75% of
individuals with phones• Group C Villages
– SMS price information to randomly selected 25% of individuals with phones
• Group D Villages– No SMS price information
Lecture Overview
• Attrition• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
Non compliers
27
Target Populati
on
Not in evaluation
Evaluation Sample
Treatment group
Participants
No-Shows
Control groupNon-
Participants
Cross-overs
Random Assignment
No!
What can you do?
Can you switch them?
Non compliers
28
Target Populati
on
Not in evaluation
Evaluation Sample
Treatment group
Participants
No-Shows
Control groupNon-
Participants
Cross-overs
Random Assignment
No!
What can you do?
Can you drop them?
Non compliers
29
Target Populati
on
Not in evaluation
Evaluation Sample
Treatment group
Participants
No-Shows
Control groupNon-
Participants
Cross-overs
Random Assignment
You can compare the original groups
Sample selection bias
• Sample selection bias could arise if factors other than random assignment influence program allocation– Even if intended allocation of
program was random, the actual allocation may not be
Sample selection bias
• Individuals assigned to comparison group could attempt to move into treatment group– School feeding program: parents could
attempt to move their children from comparison school to treatment school
• Alternatively, individuals allocated to treatment group may not receive treatment– School feeding program: some students
assigned to treatment schools bring and eat their own lunch anyway, or choose not to eat at all.
Lecture Overview
• Attrition• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
ITT and ToT
• Vaccination campaign in villages
• Some people in treatment villages not treated– 78% of people assigned to receive
treatment received some treatment
• What do you do?– Compare the beneficiaries and non-
beneficiaries?– Why not?
Which groups can be compared ?
Treatment Group:
VaccinationControl Group
Acceptent :
TREATEDNON-TREATED
Refusent :
NON-TREATED
What is the difference between the 2 random groups?
Treatment Group Control Group
1: treated – not infected2: treated – not infected3: treated – infected
5: non-treated – infected6: non-treated – not infected7: non-treated – infected8: non-treated – infected
4: non-treated – infected
Intention to Treat - ITT
Treatment Group: 50% infected
Control Group: 75% infected
●Y(T)= Average Outcome in Treatment Group
●Y(C)= Average Outcome in Control Group
ITT = Y(T) - Y(C)
●ITT = 50% - 75% = -25 percentage points
Intention to Treat (ITT)
• What does “intention to treat” measure?“What happened to the average child who is in a treated school in this population?”
• Is this difference the causal effect of the intervention?
When is ITT useful?
• May relate more to actual programs
• For example, we may not be interested in the medical effect of deworming treatment, but what would happen under an actual deworming program.
• If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant.
What NOT to do!Intention
School 1 to Treat ? Treated?Pupil 1 yes yes 4Pupil 2 yes yes 4Pupil 3 yes yes 4Pupil 4 yes no 0Pupil 5 yes yes 4Pupil 6 yes no 2Pupil 7 yes no 0Pupil 8 yes yes 6 School 1:Pupil 9 yes yes 6 Avg. Change among Treated (A)Pupil 10 yes no 0 School 2:
Avg. Change among Treated A= Avg. Change among not-treated (B)
School 2 A-BPupil 1 no no 2Pupil 2 no no 1Pupil 3 no yes 3Pupil 4 no no 0Pupil 5 no no 0Pupil 6 no yes 3Pupil 7 no no 0Pupil 8 no no 0Pupil 9 no no 0Pupil 10 no no 0
Avg. Change among Not-Treated B=
Observed Change in
weight
What NOT to do!
3
3 0.9
2.1
0.9
IntentionSchool 1 to Treat ? Treated?
Pupil 1 yes yes 4Pupil 2 yes yes 4Pupil 3 yes yes 4Pupil 4 yes no 0Pupil 5 yes yes 4Pupil 6 yes no 2Pupil 7 yes no 0Pupil 8 yes yes 6 School 1:Pupil 9 yes yes 6 Avg. Change among Treated (A)Pupil 10 yes no 0 School 2:
Avg. Change among Treated A= Avg. Change among not-treated (B)
School 2 A-BPupil 1 no no 2Pupil 2 no no 1Pupil 3 no yes 3Pupil 4 no no 0Pupil 5 no no 0Pupil 6 no yes 3Pupil 7 no no 0Pupil 8 no no 0Pupil 9 no no 0Pupil 10 no no 0
Avg. Change among Not-Treated B=
Observed Change in
weight
From ITT to effect of treatment on the treated (TOT)
• The point is that if there is leakage across the groups, the comparison between those originally assigned to treatment and those originally assigned to control is smaller
• But the difference in the probability of getting treated is also smaller
• Formally this is done by “instrumenting” the probability of treatment by the original assignment
41
Treatment Control0
0.2
0.4
0.6
0.8
1
1.2
Act
ual
ly T
reat
ed
Estimating ToT from ITT: Wald
Treatment Control0
0.2
0.4
0.6
0.8
1
1.2
Act
ual
ly T
reat
ed
Interpreting ToT from ITT: Wald
Estimating TOT
• What values do we need?• Y(T)• Y(C)
• Prob[treated|T]• Prob[treated|C]
Treatment on the treated (TOT)
• Starting from a simple regression model:
• [Angrist and Pischke, p. 67 show]:
Treatment on the treated (TOT)
IntentionSchool 1 to Treat ? Treated?
Pupil 1 yes yes 4Pupil 2 yes yes 4Pupil 3 yes yes 4 A = Gain if TreatedPupil 4 yes no 0 B = Gain if not TreatedPupil 5 yes yes 4Pupil 6 yes no 2Pupil 7 yes no 0 ToT Estimator: A-BPupil 8 yes yes 6Pupil 9 yes yes 6Pupil 10 yes no 0 A-B = Y(T)-Y(C)
Avg. Change Y(T)= Prob(Treated|T)-Prob(Treated|C)
School 2Pupil 1 no no 2 Y(T)Pupil 2 no no 1 Y(C)Pupil 3 no yes 3 Prob(Treated|T)Pupil 4 no no 0 Prob(Treated|C)Pupil 5 no no 0Pupil 6 no yes 3Pupil 7 no no 0 Y(T)-Y(C)Pupil 8 no no 0 Prob(Treated|T)-Prob(Treated|C)Pupil 9 no no 0Pupil 10 no no 0
Avg. Change Y(C) = A-B
Observed Change in
weight
TOT estimator
TOT estimator
3
30.960%20%
2.140%
0.9 5.25
IntentionSchool 1 to Treat ? Treated?
Pupil 1 yes yes 4Pupil 2 yes yes 4Pupil 3 yes yes 4 A = Gain if TreatedPupil 4 yes no 0 B = Gain if not TreatedPupil 5 yes yes 4Pupil 6 yes no 2Pupil 7 yes no 0 ToT Estimator: A-BPupil 8 yes yes 6Pupil 9 yes yes 6Pupil 10 yes no 0 A-B = Y(T)-Y(C)
Avg. Change Y(T)= Prob(Treated|T)-Prob(Treated|C)
School 2Pupil 1 no no 2 Y(T)Pupil 2 no no 1 Y(C)Pupil 3 no yes 3 Prob(Treated|T)Pupil 4 no no 0 Prob(Treated|C)Pupil 5 no no 0Pupil 6 no yes 3Pupil 7 no no 0 Y(T)-Y(C)Pupil 8 no no 0 Prob(Treated|T)-Prob(Treated|C)Pupil 9 no no 0Pupil 10 no no 0
Avg. Change Y(C) = A-B
Observed Change in
weight
1. First stage regression:
2. Predict treatment status using estimated coefficients
3. Regress outcome variable on predicted treatment status
4. gives treatment effect
Generalizing the ToT Approach: Instrumental Variables
• First stage– Your experiment (or instrument)
meaningfully affects probability of treatment
• Exclusion restriction– Your expeirment (or instrument)
does not affect outcomes through another channel
Requirements for Instrumental Variables
The ITT estimate will always be smaller (e.g., closer to zero) than the ToT estimate
A. TrueB. FalseC. Don’t Know
Spillovers, contamination
Target Populati
on
Not in evaluation
Evaluation Sample
TotalPopulation
Random Assignment
Treatment Group
Control Group
Treatment
Negative Spillovers
Target Populati
on
Not in evaluation
Evaluation Sample
TotalPopulation
Random Assignment
Treatment Group
Control Group
Treatment
TOT not always appropriate…
• Example: send 50% of MIT staff a letter warning of flu season, encourage them to get vaccines
• Suppose 50% in treatment, 0% in control get vaccines
• Suppose incidence of flu in treated group drops 35% relative to control group
• Is (.35) / (.5 – 0 ) = 70% the correct estimate?
• What effect might letter alone have?
Lecture Overview
• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
Multiple outcomes
• Can we look at various outcomes?• The more outcomes you look at,
the higher the chance you find at least one significantly affected by the program– Pre-specify outcomes of interest– Report results on all measured
outcomes, even null results– Correct statistical tests (Bonferroni)
Covariates
Rule: Report both “raw” differences and regression-adjusted results
Rule: Report both “raw” differences and regression-adjusted results
• Why include covariates?– May explain variation, improve statistical
power
• Why not include covariates?– Appearances of “specification searching”
• What to control for?– If stratified randomization: add
strata fixed effects–Other covariates
Lecture Overview
• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Cost Effectiveness
Threat to external validity:
• Behavioral responses to evaluations
• Generalizability of results
Threat to external validity: Behavioral responses to evaluations• One limitation of evaluations is that
the evaluation itself may cause the treatment or comparison group to change its behavior– Treatment group behavior changes: Hawthorne
effect– Comparison group behavior changes: John
Henry effect
●Minimize salience of evaluation as much as possible
●Consider including controls who are measured at end-line only
Generalizability of results
• Depend on three factors:– Program Implementation: can it be
replicated at a large (national) scale?– Study Sample: is it representative?– Sensitivity of results: would a similar,
but slightly different program, have same impact?
Lecture Overview
• Spillovers• Partial Compliance and Sample
Selection Bias• Intention to Treat & Treatment on
Treated• Choice of outcomes• External validity• Conclusion
Conclusion
• There are many threats to the internal and external validity of randomized evaluations…
• …as are there for every other type of study
• Randomized trials:– Facilitate simple and transparent analysis
• Provide few “degrees of freedom” in data analysis (this is a good thing)
– Allow clear tests of validity of experiment
Further resources
• Using Randomization in Development Economics Research: A Toolkit (Duflo, Glennerster, Kremer)
• Mostly Harmless Econometrics (Angrist and Pischke)
• Identification and Estimation of Local Average Treatment Effects (Imbens and Angrist, Econometrica, 1994).
top related