research design: causal inference and counterfactuals › ... › uploads ›...

IntroductionCounterfactuals

ConfoundingCriteria

Research Design:

Causal inference and counterfactuals

Johan A. Elkink

University College Dublin

8 March 2013

Johan A. Elkink counterfactual causal inference


ConfoundingCriteria

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria

Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria

Inference

In regression analysis we look at the relationship between (a set of)independent variable(s) and a dependent variable.

Statistical inference is concerned with the question how likely it is toobserve this relationship given the null hypothesis of no relationship(frequentist)

A different question is whether or not we can deduce that the

independent variable is a cause of the dependent one.

(Note: references in these slides can be found in the associated handout.)



ConfoundingCriteria

Inference

In regression analysis we look at the relationship between (a set of)independent variable(s) and a dependent variable.

Statistical inference is concerned with the question how likely it is toobserve this relationship given the null hypothesis of no relationship(frequentist) or how much we should update our beliefs concerning thisrelationship given our new evidence (Bayesian).

A different question is whether or not we can deduce that the

independent variable is a cause of the dependent one.

(Note: references in these slides can be found in the associated handout.)



ConfoundingCriteria

Association

association 6= causation



ConfoundingCriteria

Association

Given that, say, X and Y are correlation (associated), there arestill many possible causal patterns at play.



ConfoundingCriteria

Many possible patterns



ConfoundingCriteria

Inference

Generally, to make causal inferences from your analysis, additionalassumptions need to be made in addition to the ones already madefor associational or predictive inference.



ConfoundingCriteria

Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria

Fundamental problem

Imagine, there are two kinds of people, one group, T = 1, that has acollege degree, and another group, T = 0, that does not. We want tomeasure where a college degree leads to a higher salary, Y .



ConfoundingCriteria

Fundamental problem

Imagine, there are two kinds of people, one group, T = 1, that has acollege degree, and another group, T = 0, that does not. We want tomeasure where a college degree leads to a higher salary, Y .

What we would like to know is the difference for any individual i whether

they have a college degree or not: Y Ti=1i

− Y Ti=0i

. However, for every

individual i , we either observe Y Ti=1i

, or we observe Y Ti=0i

– they either

have the degree or they don’t.



ConfoundingCriteria

We wish ...

Respondent Degree Y Ti=0i

Y Ti=1i

effect

1 Yes 121 133 +122 Yes 100 109 +93 No 90 92 +24 No 87 88 +15 Yes 143 146 +36 Yes 111 124 +137 No 92 92 08 Yes 95 109 +14



ConfoundingCriteria

We wish ... we have ...

Respondent Degree Y Ti=0i

Y Ti=1i

effect

1 Yes 1332 Yes 1093 No 904 No 875 Yes 1466 Yes 1247 No 928 Yes 109



ConfoundingCriteria

Potential outcomes

Potential outcome =

{

Y1i if Ti = 1Y0i if Ti = 0

E.g., Y1i is the salary of individual i had (s)he a college degree,irrespective of whether (s)he actually does.

(Angrist & Pischke 2009, 13-14)



ConfoundingCriteria

Potential outcomes

Potential outcome =

{

Y1i if Ti = 1Y0i if Ti = 0

E.g., Y1i is the salary of individual i had (s)he a college degree,irrespective of whether (s)he actually does.

Yi = Y0i + (Y1i − Y0i )Ti = Y0i + δTi ,

where δ = Y1i − Y0i is the causal effect.

(Angrist & Pischke 2009, 13-14)



ConfoundingCriteria

Average treatment effect

Because it is impossible to observe individual treatment effect, we usuallyturn to average treatment effect:

E [δ] = E [Y1i − Y0i ] = E [Y1i ]− E [Y0i ],

which we could naively estimate with

δ̂ = E [Y1i |Ti = 1]− E [Y0i |Ti = 0].



ConfoundingCriteria

Average treatment effect

Because it is impossible to observe individual treatment effect, we usuallyturn to average treatment effect:

E [δ] = E [Y1i − Y0i ] = E [Y1i ]− E [Y0i ],

which we could naively estimate with

δ̂ = E [Y1i |Ti = 1]− E [Y0i |Ti = 0].

This assumes that E [Y1i ] reflects the salary for people with a college

degree, irrespective of whether they got one or not, and that E [Y0i ]

reflects the salary without a college degree, irrespective of whether they

got one or not.



ConfoundingCriteria

Counterfactual causality

By making such assumptions – by looking at the ATE – we aremaking a counterfactual argument. We are making assumptions ofwhat Y1i would have been, had i had a college degree.



ConfoundingCriteria



This implies that we cannot measure a causal effect, only estimateit.



ConfoundingCriteria



This implies that we cannot measure a causal effect, only estimateit.

To understand when the ATE assumptions are reasonable, we needto look at the effect of covariates – other variables that relate toY , which we will denote by X.



ConfoundingCriteria

Treatment effect: abbreviations

ATE Average Treatment Effect E [δ] = E [Y1i − Y0i ]ATT ATE for the Treated E [δT ] = E [Y1i − Y0i |Ti = 1]ATC ATE for the Control (untreated) E [δC ] = E [Y1i − Y0i |Ti = 0]

PATE Population ATE E [Y1i − Y0i ]SATE Sample ATE En[Y1i − Y0i ]LATE Local ATE E [Y1i − Y0i |Xi = x ]CATE Conditional ATE E [Y1i − Y0i |Xi = x ]

and analogously we have LATT, PATT, SATC, etceta. Note that ATE,ATT, ATC implicitly refer to population values.

En[·] is the sample mean, i.e. En[x ] = x̄ = 1n

∑

n

i=1, where the En allowsfor the formulation of conditional and counterfactual means.



ConfoundingCriteria

Bias in causal inference

Using shorthand E01 = E [Y0i |Ti = 1], etc., and taking π as thepopulation proportion that received the treatment,

E [δ] = πE [δ|Ti = 1] + (1− π)E [δ|Ti = 0]

= π(E11 − E01) + (1− π)(E10 − E00)



ConfoundingCriteria



E [δ] = πE [δ|Ti = 1] + (1− π)E [δ|Ti = 0]

= π(E11 − E01) + (1− π)(E10 − E00)

can be decomposed into

(E11 − E00) = E [δ] + (E01 − E00) + (1− π){(E11 − E01)− (E10 − E00)}.



ConfoundingCriteria



E [δ] = πE [δ|Ti = 1] + (1− π)E [δ|Ti = 0]

= π(E11 − E01) + (1− π)(E10 − E00)

can be decomposed into

(E11 − E00) = E [δ] + (E01 − E00) + (1− π){(E11 − E01)− (E10 − E00)}.

(E11 − E00) observed difference in effectE [δ] average treatment effect

(E01 − E00) selection bias(1− π){(E11 − E01)− (E10 − E00)} differential treatment effect bias

(Morgan & Winship 2007



ConfoundingCriteria

SUTVA

The stable unit treatment value assumption “SUTVA is simply thea priori assumption that the value of Y for unit i when exposed totreatment t will be the same no matter what mechanism is used toassign treatment t to unit i and no matter what treatments theother units receive.”

(Rubin (1986: 961), as cited in Morgan & Winship (2007: 37))



ConfoundingCriteria

When to control?How to control?

Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria


Confounding

When studying effect of, say, T on Y , by examining the statisticalassociation between the two variables, we need to ascertain that theobserved effect is not caused by a third variable, say, X.

(Pearl 2000: 182-183)



ConfoundingCriteria


Confounding

When studying effect of, say, T on Y , by examining the statisticalassociation between the two variables, we need to ascertain that theobserved effect is not caused by a third variable, say, X.

“We can say that T and Y are confounded when there is a third variable

X that influences both T and Y ; such a variable is then called a

confounder of T and Y .”

(Pearl 2000: 182-183)



ConfoundingCriteria


Confounding

Another way of saying this is that if

E (Y |T ,X ) 6= E (Y |T )

andE (T |X ) 6= E (T ),

X is a confounder of the effect of T on Y .

(Lee 2005: 44)



ConfoundingCriteria


Confounding

If healthier patients take a drug and sicker patients do not, wecan find an association between drug and recovery even whenthe drug does not work.

If sicker patients take a drug and healthier patients do not, wemight not find an association between drug and recovery evenwhen the drug works.

association 6= causation

The first example is also called a spurious effect (not to beconfused with spurious regression).



ConfoundingCriteria


Endogeneity

The situation where cor(X, ε) 6= 0 is called endogeneity.

Endogeneity has three main causes:

Measurement error in X

Simultaneity or reverse causation

Omitted variables



ConfoundingCriteria


Confounding

Note that confounding is a causal concept, not an associationalone!

(Pearl 2000)



ConfoundingCriteria


Confounding

Note that confounding is a causal concept, not an associationalone!

X has to have a causal effect on T and X has to have a causaleffect on Y for there to be an issue.

(Pearl 2000)



ConfoundingCriteria


Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria


When to control?

X affects both T and Y =⇒ control

(Lee 2005: 43-48)



ConfoundingCriteria


Do control

This is the typical case of a confounding factor, and hence shouldbe eliminated through controlling.



ConfoundingCriteria


Do control



ConfoundingCriteria


When to control?


T affects Y , which in turn affects X =⇒ do not control

(Lee 2005: 43-48)



ConfoundingCriteria


Don’t control



ConfoundingCriteria


Don’t control

In this case, X is an effect of Y . By controlling for X , you canseverily underestimate the effect of T on Y .



ConfoundingCriteria


Don’t control

In this case, X is an effect of Y . By controlling for X , you canseverily underestimate the effect of T on Y .

Imagine that a college degree leads to a better income leads to anicer car. Controlling for the price of the car in estimating theeffect of having a college degree on income might cancel the effect.



ConfoundingCriteria


When to control?



T affects X , which in turn affects Y =⇒ do not control ...

(Lee 2005: 43-48)



ConfoundingCriteria


Don’t control



ConfoundingCriteria


Don’t control

To get the overall effect of T on Y , you want to include the effectthrough X .



ConfoundingCriteria


Don’t control

To get the overall effect of T on Y , you want to include the effectthrough X .

E.g. if you want to know the effect of changing the policyregarding smoking in pubs on the amount of smoking in general,you do not care through what mechanism this happened (throughpeer pressure, laziness, etc.), but only about the overall effect.



ConfoundingCriteria


When to control?




... unless you explicitly want only the direct effect

(Lee 2005: 43-48)



ConfoundingCriteria


Maybe control



ConfoundingCriteria


Maybe control

Remember the following equation:

β∗ = β + φγ



ConfoundingCriteria


Maybe control

Remember the following equation:

β∗ = β + φγ

Sometimes you are interested in β (so control), sometimes in β∗

(so don’t control).



ConfoundingCriteria


Maybe control

Example: A scholarship for poorer students might help them to get acollege degree, which in turn might help them to earn more money laterin life.



ConfoundingCriteria


Maybe control

Example: A scholarship for poorer students might help them to get acollege degree, which in turn might help them to earn more money laterin life. Having a scholarship on your CV, however, might also further yourcareer, independent of the effect of having a college degree.



ConfoundingCriteria


Maybe control


To see the overall effect of the scholarship, don’t control on having acollege degree.



ConfoundingCriteria


Maybe control


To see the overall effect of the scholarship, don’t control on having acollege degree.

To see the effect of having a scholarship, independent of the effect of

getting a college degree, do control for college degree.



ConfoundingCriteria


When to control?





X affects Y , but not T , nor the effect of T on Y

(Lee 2005: 43-48)



ConfoundingCriteria


Maybe control



ConfoundingCriteria


Maybe control

When X affects Y , but not T , there is no confounding issue andthe estimates for the effect of T on Y should not be affected byinclusion of X . However, including X in the model can still help forefficiency.

(Gelman & Hill 2007: 177)



ConfoundingCriteria


When to control?





X affects Y , but not T , nor the effect of T on Y

X affects Y , not T , but it does affect of effect of T on Y (interaction)

(Lee 2005: 43-48)



ConfoundingCriteria


Maybe control

Here including the interaction in your model can highlight how theeffect is different for different groups.



ConfoundingCriteria


Maybe control

Here including the interaction in your model can highlight how theeffect is different for different groups.

Note that it affects the interpretation, but that the estimation ofthe overall ATE is not affected by controlling for X .



ConfoundingCriteria


Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria


The ideal experiment

To avoid any effect of covariates the ideal is to randomly selectparticipants for your research from the overal population (enablesinference to the population) and to randomly assign the treatmentto these participants (enables causal inference).



ConfoundingCriteria


How to control?

Experiment

Field experiment

Natural experiment

Blocking

Matching

Regression

etc.



ConfoundingCriteria


Kitchen sink

A typical approach in the quantitative social sciences is to collect anumber of different theories / hypotheses, add them all asvariables to a regression, and see “who wins”. This is the kitchensink approach (or garbage can approach).

If anything, the above discussion should have made clear that todraw causal inferences, a clear distinction of treatment fromcovariates is crucial. In other words: focus your research!



ConfoundingCriteria


Kitchen sink

A typical approach in the quantitative social sciences is to collect anumber of different theories / hypotheses, add them all asvariables to a regression, and see “who wins”. This is the kitchensink approach (or garbage can approach).

If anything, the above discussion should have made clear that todraw causal inferences, a clear distinction of treatment fromcovariates is crucial. In other words: focus your research!

(Note that the “garbage can” phrase has also been used to argue against

ignoring nonlinearities (Achen 2005), as opposed to careless specification of the

causal effect.)



ConfoundingCriteria


Kitchen sink

Another way of putting the issue is that the above is all abouttrying to study the effect of a cause (treatment), rather than thecause of an effect. The latter is perhaps ill-defined and runs intothe “infinite regress of causation”.

(See Gerring (2001, 2012) for an extensive discussion of Y -centered and

X -centered research.)

(Gelman & Hill 2007: 187)



ConfoundingCriteria


Causal diagrams

The preceding examples underline how it is important to alwaysdraw out the causal diagram and consider carefully how you selectcases and select controls when making causal inferences.



ConfoundingCriteria


Equifinality

Equifinality refers to the situation where a particular outcomemight come about through different causal paths. No particularpath might show a strong association between independent anddependent variable.



ConfoundingCriteria


Equifinality

Equifinality refers to the situation where a particular outcomemight come about through different causal paths. No particularpath might show a strong association between independent anddependent variable.

Should this be seen as problematic for the counterfactual causalinference framework?



ConfoundingCriteria

Outline

1 Introduction

2 Counterfactuals

3 Confounding

When to control?

How to control?

4 Criteria



ConfoundingCriteria

KKV’s 5 rules

1 Construct falsifiable theories

2 Build theories that are internally consistent3 Select dependent variables carefully

No endogenous relationshipEnsure variation in dependent variable

4 Maximize concreteness

5 State theories as encompassing as feasible

(King, Keohane & Verba 1994: 100-114)



ConfoundingCriteria

Gerring’s criteria

Clarity

Manipulability

Separation

Independence (priority)

Impact

Mechanism



ConfoundingCriteria

Discussion points

How do causal inference and prediction relate? (“The proof ofthe pudding is in the eating?”)



ConfoundingCriteria

Discussion points


How does the theory thus far translate to qualitative research?Does it?



ConfoundingCriteria

Discussion points



Are all causal inferences counterfactual?



ConfoundingCriteria

Discussion points



Are all causal inferences counterfactual?

How does this all relate to causal mechanisms in the sense ofHedstrom & Swedberg?


research design: causal inference and counterfactuals › ... › uploads ›...

Documents