from experimental to quasi- experimental research · from experimental to quasi-experimental...

Research Methods for Political Science

Dr. Thomas ChadefauxAssistant Professor in Political Science

Thomas.chadefaux@tcd.ie1

From experimental to quasi-experimental research

Experiments…

… Address threats to internal validity:– Omitted variables– Trends in outcomes (e.g., inflation, aging, etc.)– Mis-specified variances (e.g., autocorrelation or

group correlation)– Mismeasurement– Endogeneity– Selection effects.• E.g., effect of training programs on earning

2

Experiments…

• … are less able to address threats to externalvalidity:– Unrepresentative treatment group of the

population– The effect of the treatment might differ

depending on the setting (lab vs real life)– The effect of the treatment may differ over time

3

Experiments and quasi-experiments

• Experiments are the gold standard because of random assignment

• But in the real-world, random assignment is often not possible.

4

Quasi-experiments

• Quasi-experiments try to be as close as possible to experiments, but without the experimenter’s ability to assign subjects to one group or another

• -> take advantage of situations occurring naturally that correct for issues of internal and external validity

5

Main types of quasi-experiments

• Difference in differences

• Natural experiments• Instrumental variables• Regression-discontinuity designs

6

DIFFERENCE IN DIFFERENCES

7

Difference in differences

8

An example

9

New Jersey

Pennsylvania

Empl

oym

ent

February NovemberApril

Disadvantage of DiD

• Parallel trend assumption

10

Outcome

TimeYear0Yeart-1Yeart-2Yeart-3Yeart-4

Treatment

Comparison

Start of intervention

Matched Comparison

Matchedcounterfactual

Even better: matched counterfactuals: find a good match, or a synthetic comparison

Another example: Effect of anti-speeding measuresin Connecticut on fatalities

before and after measures

12

Source for this and upcoming images: Campbell, D. T. (1968). The Connecticut Crackdown on Speeding: Time-Series Data in Quasi-Experimental Analysis. Law and Society Review, 3(1), 33–54.

Time series2. interrupted time series

13

Time series3. Multiple time series

14

15

Time series3. Multiple time series

NATURAL EXPERIMENTS

16

Natural Experiments

• A classic example:1854 cholera outbreak in London

17

Types of natural experiments• Lottery studies

– Vietnam draft– Lottery winners and socio-economic outcomes.

• Among lottery winners, winnings are randomly assigned– Public policies

• In Bogotá, vouchers that covered the cost of private secondary school were allocated by lottery– Evidence of persistent effects on academic achievement

– Term lengths in legislatures. Does length improve performance (e.g., abstain less, propose more bills?)• Argentine, 1983. To introduce a staggered system, some legislators were

assigned 2-year initial terms– Effect of indiscriminate violence on insurgent attacks -> drunk Russian

soldiers– soldiers “get drunk as pigs, lob out a few shells, claim combat pay and

get drunk again”

18

Jurisdictional borders

• Borders that separate similar populations, firms, etc.

• An intervention affects one side, but not the other

• E.g., Posner on Chewas and Tumbukas

19

INSTRUMENTAL VARIABLES

20

Instrumental variables

• Same idea as standard natural experiments: find an additional variable (an ‘instrument’) that is correlated with the independent variable but could not be influenced by the DV or correlated with its other causes

21

Instrumental variablesExample 1

22

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.

Instrumental variables

23

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Madestam, Andreas, D Shoag, S. Veuger, and D. Yanagizawa-Drott. 2013. “Do Political Protests Matter? Evidence from the Tea Party Movement.” The Quarterly Journal of Economics 128(4): 1633–85.

Politicalpreferences

Instrumental variables

24

Size of Tea Party Rally

How conservativedoes represen-tative vote in thelegislature?

Rainfall on the day of the rally (conditional on chance of rain)

Rainfall cannot be the consequence of political beliefs, culture of activism or previous voting behaviour. But it is likely that it affects turnout at rallies.

Politicalpreferences

Instrumental variablesExample 2

• An example: effect of resource scarcity on conflict? Problem of reverse causality.

• Solution: use as-if random assignment to scarcity using rainfall patterns.

25

Instrumental variablesExample 2

26

Resource scarcity conflict

Instrumental variablesExample 2

27

Resource scarcity Conflict

Rainfall

Instrumental variablesExample 3

• Vietnam war draft– Angrist, Joshua D. (1990). "Lifetime Earnings and the Vietnam Draft Lottery:

Evidence from Social Security Administrative Records". American Economic Review. 80 (3): 313–336

29

Example 3

30

Soldier Earnings

Personality

Example 3

31

Soldier Earnings

Personality

Draft (lottery)

REGRESSION DISCONTINUITY DESIGNS

32

Discontinuity design

33

34

Regression-discontinuity designs

35The Impact of Secondary Schooling in Kenya: A Regression Discontinuity Analysis. Owen Ozier, 2011