01 research design 2009
TRANSCRIPT
-
7/28/2019 01 Research Design 2009
1/23
Introduction:Research design
17.871
1
-
7/28/2019 01 Research Design 2009
2/23
Statistics
1. a. In early use, that branch of political sciencedealing with the collection, classification, and
discussion of facts (especially of a numericalkind) bearing on the condition of a state or community. In recent use, the department of study that has for its object the collection and
arrangement of numerical facts or data, whether relating to human affairs or to naturalphenomena. (OED)First usage: 1770
2
-
7/28/2019 01 Research Design 2009
3/23
Etymology of statistics
From German Statistik , political science,from New Latin statisticus , of state affairs,from Italian statista , person skilled instatecraft, from stato , state, from OldItalian, from Latin status , position, form of
government.
-American Heritage Dictionary of the American Language
3
-
7/28/2019 01 Research Design 2009
4/23
The Biggest Problem in Research:
Establishing CausalityExample: HIV and circumcision
Observational studies suggest that male circumcisionmay provide protection against HIV infection
4
-
7/28/2019 01 Research Design 2009
5/23
Why is causality such a problem?
In observational studies, selection intotreatment and control cases rarely random
Schooling examples (private vs. public)Voting examples (pro-choice versus pro-life)
Treatment and control cases may thus differ inother ways that affect the outcome of interest
The two primary drivers of selection areConfounding variablesReverse causation
5
-
7/28/2019 01 Research Design 2009
6/23
How to Establish Causality(i.e., how to rule out alternatives)
How do we establish causality? By rulingout alternative explanations
Legal analogy: prosecutor versus defenseRun a field experiment! (best approach)HIV and circumcision: field experimentpossible?
6
-
7/28/2019 01 Research Design 2009
7/23
Post-test only experiment
Donald Campbell and Julian Stanley, Experimental and Quasi-Experimental Designs for Research (1963)Summary:
R X OR O
No prior observationClassical scientific and agricultural experimentalism
7
-
7/28/2019 01 Research Design 2009
8/23
Field Experiment example:
HIV and male circumcision3,274 uncircumcised men, aged 18 24, volunteered!Randomly assigned to a control or an intervention group withFollow-up visits at months 3, 12, and 21
Did it work?
Control group: 2.1 per 100 person-yearsTreatment group: 0.85 per 100 person-years
Problems?Internally valid!
Because of randomization intervention, no bias from nonrandom selection intothe treatment group. That is,
No differences between the treatment and control group on confounding variables(only comparing apples with apples, no apples with oranges)No possibility of reverse causation
Alternative interpretations of the treatment?External validity?Could the difference have occurred by chance?
Unlikely: p < 0.001 on difference
8
-
7/28/2019 01 Research Design 2009
9/23
HIV and male circumcision
When controlling for behavioral factors, including sexualbehavior that increased slightly in the intervention group,condom use, and health-seeking behavior, the protectionwas
61% (95% CI: 34% 77%).Male circumcision provides a degree of protectionagainst acquiring HIV infection, equivalent to what avaccine of high efficacy would have achieved.Male circumcision may provide an important way of reducing the spread of HIV infection in sub-Saharan
Africa.PLoS Medicine Vol. 2, No. 11
9
-
7/28/2019 01 Research Design 2009
10/23
How to Establish Causality(i.e., how to rule out alternatives)
But, running an experiment is often impossibleTry anyway: e.g., HIV and circumcision
If you cant run an experiment: naturalexperiment
Exploit something that is exogenous Accidental deathsTiming of Senate elections
Imposition of new voting machines9/11 terrorist attacksGeographical boundaries
Exploit a discontinuitySumma Cum Laude s effect on incomeRegression discontinuity (RD) design 10
-
7/28/2019 01 Research Design 2009
11/23
11
-
7/28/2019 01 Research Design 2009
12/23
Regression discontinuityExample from Brazil
12
-
7/28/2019 01 Research Design 2009
13/23
13
-
7/28/2019 01 Research Design 2009
14/23
-
7/28/2019 01 Research Design 2009
15/23
How to Establish Causality(i.e., how to rule out alternatives)
If you cant run an experiment or find anatural experiment/discontinuity
Control for confounding variablesDifference-in-differences (DD)MatchingControlling for variables with parametric models,e.g., regression
Eliminate reverse causationExploit time with panel data, i.e., measure the
outcome before and after some treatment 15
-
7/28/2019 01 Research Design 2009
16/23
Difference-in-differences
Media effects exampleEndorsement changes in the 1997 British
electionIllustrates
difference-in-differences, which reduces bias fromconfounding variablesPanel data, which can help rule out reversecausation
16
-
7/28/2019 01 Research Design 2009
17/23
% L
a b o u r v o
t e a
m o n g v o
t e r s
2 5
3 0
3 5
4 0
4 5
5 0
5 5
6 0
1992 1997
Read paper before 1997 that switched to Labour (n=185)
Did not read paper before 1997 that switched to Labour (n=1408)
5.3
12.6
17
-
7/28/2019 01 Research Design 2009
18/23
% L
a b o u r v o
t e a
m o n g v o
t e r s
2 5
3 0
3 5
4 0
4 5
5 0
5 5
6 0
1992 1997
Read paper before 1997 that switched to Labour (n=185)
Did not read paper before 1997 that switched to Labour (n=1408)
5.3
12.6
18
-
7/28/2019 01 Research Design 2009
19/23
% L
a b o u r v o
t e a
m o n g v o
t e r s
2 5
3 0
3 5
4 0
4 5
5 0
5 5
6 0
1992 1997
Read paper before 1997 that switched to Labour (n=185)
Did not read paper before 1997 that switched to Labour (n=1408)
5.3
12.6
19
-
7/28/2019 01 Research Design 2009
20/23
How to Establish Causality(i.e., how to rule out alternatives)
If you cant run an experiment or find anatural experiment
Control for confounding variablesDifference-in-differences (DD)MatchingControlling for variables with parametricmodels, e.g., regression
Eliminate reverse causationExploit time with panel data, i.e., measure the
outcome before and after some treatment
Much of 17.871is about this
20
-
7/28/2019 01 Research Design 2009
21/23
SummaryClassical experimentation unlikely, but always preferred
Always keep a classical experiment in mind when designingobservational studies
Strive for natural or quasi -experiments
Alternating years of standardized testingTiming of Senate electionsImposition of new voting machines9/11 terrorist attacksUse Regression-discontinuity designs
Geographical boundaries (e.g., minimum wage study)
Use Difference-in-differences designsGather as much cross-time data as possible (panelstudies)If you only have cross-sectional data, be humble!
21
-
7/28/2019 01 Research Design 2009
22/23
22
-
7/28/2019 01 Research Design 2009
23/23
(Angrist and Lavy, 1999) 23