regression discontinuity design
DESCRIPTION
Regression Discontinuity Design. Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides. RD Designs. A pretest- posttest , program-comparison group strategy Review: Advantages of Pre-tests ? Detect differences between groups - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/1.jpg)
Regression Discontinuity Design
Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides
![Page 2: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/2.jpg)
RD DesignsA pretest-posttest, program-comparison
group strategy Review:
Advantages of Pre-tests?Detect differences between groupsDetect potential vulnerability to internal validity
threatsHelps with statistical analysis
Advantages of Comparison groups?Helps control sources of errorHelps support the counterfactual inference
![Page 3: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/3.jpg)
Underuse of RD? Why?It’s new.Key criteria must be met for use.Perhaps it’s just misunderstood.
![Page 4: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/4.jpg)
Overview of RD
OA C X O2
OA C O2
“pre” is ANY continuous variable that correlates with the outcome of interest
Assignment based on cutoff scoreRegression line should have vertical displacement at
the cutoff score if there is an effect
![Page 5: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/5.jpg)
No Treatment Effect
![Page 6: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/6.jpg)
Positive Effect
![Page 7: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/7.jpg)
Examples1. Campell & Stanley’s Ivy League Education
Example2. Trochim’s Hospital Administration ExampleHospital Quality of Care
![Page 8: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/8.jpg)
More about assignmentAssignment variables:
Must be continuous (or ordinal) Can be a pretest on a dependent
variable Can be by order of entry into study Cannot be caused by treatment May or may not be related to the
outcome
If implementing an RD design in your area of research, what variables would you choose for assignment?
![Page 9: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/9.jpg)
Choosing the Cutoff ScoreNow referring to the assignment variable(s) you
identified, how would you arrive at a cutoff score?
Substantive grounds: professional judgmentNeed or MeritClinical diagnosis
Practical grounds:Available data setsAvailable resources
![Page 10: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/10.jpg)
Choosing the Cutoff ScoreMean of the distribution of assignment scoresPolitically defined thresh-holdsComposite scores of assignment variables
Important: Assignment must be controlled! (It is as important as proper random assignment.)
![Page 11: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/11.jpg)
Additional ConsiderationsFunctional form relating the assignment and
outcome variablesA defined population in which it is possible for all
units in the study to receive Tx regardless of the choice of a certain cutoff point
Intent-to-Treat? : Tx diffusion and cross-over participants
![Page 12: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/12.jpg)
Variations1. Compare 2 treatment groups2. Compare 3 conditions3. Different dose treatment groups4. Multiple cutoff points5. …and many more creative ways to think of
![Page 13: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/13.jpg)
Theory of RD – How does this work?
RDs as Treatment Effects in REs RE pretest means of Tx and Control groups nearly
identical at would be the cutoff score in an RD design through random assignment
Cutoff-based assignments creates groups with different pretest means and non-overlapping pretest distributions
RD compares regression lines, not meansBoth RDs and REs control for selection bias
Unknown variables do not determine assignment Pretests have no error IF used as the selection variable Regression lines are not affected by posttest errors
![Page 14: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/14.jpg)
![Page 15: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/15.jpg)
Adherence to the CutoffOverrides of the cutoffCrossoversAttrition“Fuzzy” regression discontinuity
![Page 16: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/16.jpg)
Threats to ValidityStatistical Conclusion Validity
Nonlinearity Interactions
Internal Validity – must occur at the cutoff pointHistoryMaturationMortalitySelection-instrumentation
![Page 17: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/17.jpg)
Interaction
![Page 18: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/18.jpg)
Group Exercise: RD Design
![Page 19: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/19.jpg)
Analytical AssumptionsNo exceptions to the cutoffAdhere to true function of the pre-post
relationshipUniform delivery of pretest and program
![Page 20: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/20.jpg)
Combining RD with Randomized Experiments
7 combo examples3 advantages:
Increased power Allows estimation of both
groups at the overlap interval
Adds clarity to the cutoff point
![Page 21: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/21.jpg)
RD – Quasi-experiment?shortfalls are not yet clearRequires more “demanding statistical analysis”Less statistical powersee table 7.2 in SCC (pg. 243)
![Page 22: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/22.jpg)
Analysis ProblemsThe Curvilinear Problem
![Page 23: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/23.jpg)
Steps to AnalysisTransform the pretestExamine the relationship visuallySpecify high order terms and interactionsEstimate the initial modelRefine
![Page 24: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/24.jpg)
Comparing RD Designs with Experimental Designs
IN theory, both designs should produce similar results when all exemplary conditions of each method type exist
Question remains do they produce similar results and standard errors in practice (real world settings)?
Under exemplary conditions, experiments are 2.75 times more efficient than RDDs
If otherwise, the degree of this efficiency will varyCentral Question: How to compare these two design
options in field settings?Cook, Shadish & Wong 2008
![Page 25: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/25.jpg)
Statistical Power for GRT and RDD
the RDD has approximately 36 per cent the efficiency of the GRT.
This implies that the RDD will require approximately 2.75 times more groups than a GRT with the same power.
The same result was found by Schochet [16] for hierarchical models in education
and by Cappelleri and Trochim [31] for trials targeting individuals rather than groups.
Pennell et al., 2010
![Page 26: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/26.jpg)
![Page 27: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/27.jpg)
Within Study Comparisons:
Proposed methodology from LaLondeCausal estimates derived from an experiment
compared with estimates from a non-experimentSame Tx GroupDifferent Control Group
Modifications needed to use for RDDs
![Page 28: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/28.jpg)
7 Criteria to Improve Interpretation of Within-Study Comparisons
1. Must demonstrate variation in types of methods being contrasted
2. Both assignment mechanisms cannot be correlated with other factors related to outcome variables
3. The RE must “deserve” its status of the causal “Gold Standard”
4. The non-experiment design must also be good
AND
![Page 29: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/29.jpg)
7 Criteria to Improve Interpretation of Within-Study Comparisons
AND
5. Both study types should estimate the same causal quantity
6. Explicit criteria must be raised on how the two design estimates relate to each other
7. Blind that data analyst!
![Page 30: Regression Discontinuity Design](https://reader033.vdocuments.us/reader033/viewer/2022061617/568162b9550346895dd3413b/html5/thumbnails/30.jpg)
Further Discussion?Nagging Questions?
…orInspirations?