from experimental to quasi- experimental research · from experimental to quasi-experimental...
TRANSCRIPT
Research Methods for Political Science
Dr. Thomas ChadefauxAssistant Professor in Political Science
From experimental to quasi-experimental research
Experiments…
… Address threats to internal validity:– Omitted variables– Trends in outcomes (e.g., inflation, aging, etc.)– Mis-specified variances (e.g., autocorrelation or
group correlation)– Mismeasurement– Endogeneity– Selection effects.• E.g., effect of training programs on earning
2
Experiments…
• … are less able to address threats to externalvalidity:– Unrepresentative treatment group of the
population– The effect of the treatment might differ
depending on the setting (lab vs real life)– The effect of the treatment may differ over time
3
Experiments and quasi-experiments
• Experiments are the gold standard because of random assignment
• But in the real-world, random assignment is often not possible.
4
Quasi-experiments
• Quasi-experiments try to be as close as possible to experiments, but without the experimenter’s ability to assign subjects to one group or another
• -> take advantage of situations occurring naturally that correct for issues of internal and external validity
5
Main types of quasi-experiments
• Difference in differences
• Natural experiments• Instrumental variables• Regression-discontinuity designs
6
DIFFERENCE IN DIFFERENCES
7
Difference in differences
8
An example
9
New Jersey
Pennsylvania
Empl
oym
ent
February NovemberApril
Disadvantage of DiD
• Parallel trend assumption
10
Outcome
TimeYear0Yeart-1Yeart-2Yeart-3Yeart-4
Treatment
Comparison
Start of intervention
Matched Comparison
Matchedcounterfactual
Even better: matched counterfactuals: find a good match, or a synthetic comparison
Another example: Effect of anti-speeding measuresin Connecticut on fatalities
before and after measures
12
Source for this and upcoming images: Campbell, D. T. (1968). The Connecticut Crackdown on Speeding: Time-Series Data in Quasi-Experimental Analysis. Law and Society Review, 3(1), 33–54.
Time series2. interrupted time series
13
Time series3. Multiple time series
14
15
Time series3. Multiple time series
NATURAL EXPERIMENTS
16
Natural Experiments
• A classic example:1854 cholera outbreak in London
17
Types of natural experiments• Lottery studies
– Vietnam draft– Lottery winners and socio-economic outcomes.
• Among lottery winners, winnings are randomly assigned– Public policies
• In Bogotá, vouchers that covered the cost of private secondary school were allocated by lottery– Evidence of persistent effects on academic achievement
– Term lengths in legislatures. Does length improve performance (e.g., abstain less, propose more bills?)• Argentine, 1983. To introduce a staggered system, some legislators were
assigned 2-year initial terms– Effect of indiscriminate violence on insurgent attacks -> drunk Russian
soldiers– soldiers “get drunk as pigs, lob out a few shells, claim combat pay and
get drunk again”
18
Jurisdictional borders
• Borders that separate similar populations, firms, etc.
• An intervention affects one side, but not the other
• E.g., Posner on Chewas and Tumbukas
19
INSTRUMENTAL VARIABLES
20
Instrumental variables
• Same idea as standard natural experiments: find an additional variable (an ‘instrument’) that is correlated with the independent variable but could not be influenced by the DV or correlated with its other causes
21
Instrumental variablesExample 1
22
Size of Tea Party Rally
How conservativedoes represen-tative vote in thelegislature?
Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.
Instrumental variables
23
Size of Tea Party Rally
How conservativedoes represen-tative vote in thelegislature?
Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.
Politicalpreferences
Instrumental variables
24
Size of Tea Party Rally
How conservativedoes represen-tative vote in thelegislature?
Rainfall on the day of the rally (conditional on chance of rain)
Rainfall cannot be the consequence of political beliefs, culture of activism or previous voting behaviour. But it is likely that it affects turnout at rallies.
Politicalpreferences
Instrumental variablesExample 2
• An example: effect of resource scarcity on conflict? Problem of reverse causality.
• Solution: use as-if random assignment to scarcity using rainfall patterns.
25
Instrumental variablesExample 2
26
Resource scarcity conflict
Instrumental variablesExample 2
27
Resource scarcity Conflict
Rainfall
Instrumental variablesExample 3
• Vietnam war draft– Angrist, Joshua D. (1990). "Lifetime Earnings and the Vietnam Draft Lottery:
Evidence from Social Security Administrative Records". American Economic Review. 80 (3): 313–336
29
Example 3
30
Soldier Earnings
Personality
Example 3
31
Soldier Earnings
Personality
Draft (lottery)
REGRESSION DISCONTINUITY DESIGNS
32
Discontinuity design
33
34
Regression-discontinuity designs
35The Impact of Secondary Schooling in Kenya: A Regression Discontinuity Analysis. Owen Ozier, 2011