studying variation in the effect of program participation · 3. instrumental variables in...
TRANSCRIPT
![Page 1: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/1.jpg)
Studying Variation in the Effect of Program Participation
Stephen W. Raudenbush
Presentation at the
“Workshop on Learning from Variation in Program Effects”
Palo Alto, July 19, 2016
The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Sean Reardon and Takako Nomi for their collaboration on these ideas.
![Page 2: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/2.jpg)
Outline1. Pervasiveness of
• Multi-site trials• Non-compliance
2. Instrumental variables in a single-site study• Under homogeneity of impact• Under heterogeneity of impact• Examples
3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses• Method 2: Two-stage generalized least squares• Method 2= “Between-Site Regression!”
4. Design Considerations
5. Modeling Program Participation and Program Impact on Participants
![Page 3: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/3.jpg)
1. Pervasiveness of Multi-Site Trials
Since 2002, IES has funded 175 group-randomized trials
Vast majority are multi-site trials (Spybrook, 2013)
Other recent examples
* National Head Start Evaluation (US Dept of HHS, 2010)
* Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006)
* School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009).
* Tennessee STAR (Finn and Achilles, 1990)
* Ending Social Promotion (Jacob and Lefgren, 2009)
* Double-Dose Algebra (Nomi and Allensworth, 2009)
* Welfare to Work (Bloom, Hill, Riccio, 2003)
* Small Schools of Choice (MDRC)
![Page 4: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/4.jpg)
Some Recent MS TrialsStudy Levels Assigned
UnitsSites Fixed or
Random sites
National Head Start Eval.
2 Children 198 Program Sites
Random
Moving to Opportunity
2 Families 5 cities Fixed
BostonCharter School Lotteries
2 Children Lottery pools Random
Tennessee STAR
3 Teachers 79 Schools Random
4 R’s 3 Classrooms 18 Schools Random
Double-Dose Algebra
2 Children 60 Schools Random
![Page 5: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/5.jpg)
2. Estimating the Impact of Program Participation in a One- Site Study
![Page 6: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/6.jpg)
T=Random assignment M=Participation
Y=outcome
Figure 1: Conventional Instrumental Variable Model (Homogeneous Treatment Effects)
0/
:EffectITTTotal
so
![Page 7: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/7.jpg)
Single site, heterogeneous treatment effects
TM
Y
Person-specific Causal Model
)(E
)(E
),()(
CovBEB iii
![Page 8: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/8.jpg)
“No Compliance-Covariance” Assumption is Strong!
0/0),(
),()(
ifCovif
CovBE
![Page 9: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/9.jpg)
Alternative Approachfor binary M
“Local Average Treatment Effect” (LATE)or“Complier Average Treatment Effect”
(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)
![Page 10: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/10.jpg)
Principal StratificationStratum M(1) M(0) Г=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction
of popAverageEffect
Compliers 1 0 1 Y(1)=Y(0) γcompliers δcompliers
Always-takers
1 1 0 Y(1)-Y(1)=0 γ always 0
Never-takers
0 0 0 Y(0)-Y(0)=0 γnever 0
Defiers 0 1 -1 Y(0)-Y(1) 0 0
![Page 11: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/11.jpg)
Complier-average treatment effect(“Local average treatment effect”)
randomizedisTifMonITTTMETME
YonITTTYETYENote
so
BE
complierscompliers
neveralwayscomplierscompliers
"")0|()1|("")0|()1|(
/
*0*0)(
![Page 12: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/12.jpg)
In SumWe can estimate the Population-Average
Effect of Participating if we assume Cov(Г,Δ)=0
We can estimate LATE if we assumePr(Г≥0)=1
The latter is a weaker assumption, but does not eliminate the selection problem!
![Page 13: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/13.jpg)
Multiple SitesHow do we take this to multiple sites to
* Estimate average Impact of Program Participation
* Estimate variation in the Impact of Program Participation
* Two methods using simulated data:“Small Schools of Choice” Design (J=200, 80<n<120)
)430.0,194.1:( 2 valuestrue
![Page 14: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/14.jpg)
Method 1: Combine 2 ITT analyses
Step 1: Estimate the Impact of Treatment Assignment on the Outcome
Results
2)(,)(
.).(.
B
jijjijjjij
BVarBE
eeTTBYY
)75.1,21.(499.*96.1770.
forintervalvalueplausible%95
499.249.0ˆ53.0,770.0ˆ
22
j
B
B
se
![Page 15: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/15.jpg)
Step 2: Estimate the Impact of Treatment Assignment on Program Participation
Results2)(,)(
.).(.
G
jijjijjjij
GVarGE
TTGMM
)86,55(.
forintervalvalueplausible%95
078.0061.0ˆ703.0ˆ
22
j
G
G
![Page 16: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/16.jpg)
Step 3: Combine Results
)248.1,267.(483.*96.1095.1)(%95
483.)703(.)078(.
)078(.*)095.1()249(.ˆ
caseourIn
095.1703./770.ˆ/ˆˆ/
22
222
22
2222
DPV
DGif
caseourIn
D
D
![Page 17: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/17.jpg)
In sumTrue Values
Our estimates
430.0,19.1: 2 valuesTrue
487.0ˆ,10.1ˆ 2
![Page 18: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/18.jpg)
Method 2: Two-Stage Generalized Least Squares: Theoretical Model
2
2
)(,)(
.).(.:2Stage
)(,)(
.).(.:1Stage
D
jijjijjjij
G
jijjijjjij
DVarDE
MMDYY
GVarGE
TTGMM
![Page 19: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/19.jpg)
Method 2 in Practice
2
**
2
**
)(,ˆ)(
).(ˆ.
originalusingvariancesRecompute.3
)(,)(
).ˆ(.:.2
).(ˆ.ˆ:.1
D
ijijjijjjij
D
ijijjijjjij
jijOLSjij
DVarDE
MMDYY
M
DVarDE
MMDYYHLMDo
TTGMMCompute
![Page 20: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/20.jpg)
![Page 21: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/21.jpg)
Method 2=“Between Site Regression!”
Cj
Ejjjjj
jjj
jjj
GDGB
whichfrom
GDG
GDB
ˆ)(ˆˆ.....
)(
0)](ˆ[ jj DGERequires
![Page 22: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/22.jpg)
Results
)863.,562(.)746.,316(.%95
6968.
4856.ˆ
0730.0,095.1ˆ
22
2
2
CI
se
![Page 23: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/23.jpg)
Envisioning Variation:LATE Effect
“Head Start” Design (J=200, 10<n<20)
“Small Schools of Choice” Design (J=200, 80<n<120)
“Welfare to Work” Design (J=60, 200<n<1400)
![Page 24: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/24.jpg)
Program Participation Model (“LATE”)
)430.0,194.1:(
280.0ˆ,007.1ˆ:
487.0ˆ,095.1ˆ0ˆ986.0ˆ:
),(~
.).(.:
)(,)(
.).(:
2
2
2
2
2
2
valuestrue
WtW
SSC
HS
ND
MMDYYModelImpact
GVarGE
vvTTGMMmodelionParticipat
j
jijjijjjij
jijjijjjij
![Page 25: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/25.jpg)
Profile Likelihood for LATE: “HS” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00
0
1.00
2.00
3.00
Tau
Bet
a
![Page 26: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/26.jpg)
Profile Likelihood for LATE: “SSOC” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01
-0.50
1.00
2.50
Tau
Bet
a
![Page 27: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/27.jpg)
Profile Likelihood for LATE: “WtW” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00
0
1.00
2.00
3.00
Tau
Bet
a
![Page 28: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/28.jpg)
Posterior intervals for site-specific LATE Effects
![Page 29: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/29.jpg)
Posterior Intervals for LATE: “HS” Design
-0.55
0.27
1.09
1.90
2.72
MH
AT
0 50.50 101.00 151.50 202.00
![Page 30: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/30.jpg)
Posterior Intervals for LATE: “SSC” Design
0 50.50 101.00 151.50 202.00-1.21
-0.13
0.96
2.04
3.12
MH
AT
![Page 31: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/31.jpg)
Posterior Intervals for LATE: “W to W” Design
0 15.50 31.00 46.50 62.00-0.37
0.41
1.18
1.95
2.73
MH
AT
![Page 32: Studying Variation in the Effect of Program Participation · 3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses • Method 2: Two-stage generalized](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5c20175eb2b22f126d90b/html5/thumbnails/32.jpg)
Moving Toward Explanationmodeling participation, modeling impact
Total Impact of Assignment=Impact of Assignment on Participation * Impact of participation on Outcome
Within site:
Which persons are most likely to participate?Which persons are most likely to benefit from participation?
Between Sites:
How do we improve site-average participation rate?How do we enhance average benefit of participating?
Models are needed at both levels because sites vary not only in organizational effectiveness but also in client composition
The picture can't be displayed.
jjj DGB