natural experiments in economics: an introduction 1...

ECONOMICS 481* (Fall 2004) M.G. Abbott

ECON 481* -- NOTE 10

Natural Experiments in Economics: An Introduction

1. Objectives of Experiments What are experiments? Experiments are empirical methodologies for estimating the causal effect of specific treatments, interventions or events on observed outcomes or behaviors. What is the objective of experiments? The fundamental objective of experiments is to estimate accurately and precisely the causal effect of a treatment, intervention or event on observed outcomes. The fundamental question addressed by an experiment can be expressed in several different ways. • Does a particular treatment or intervention cause a change in outcomes or

behavior?

Examples:

Does participation in a job training program cause an increase the employment earnings of program participants?

Does an increase in the minimum wage cause a decrease the employment of low-skilled, low-wage workers?

Does more generous workers’ compensation cause an increase the length of time that injured workers remain out of work?

File: 481note10.doc – Natural Experiments in Economics of 26


• Is some outcome or behavior different in the presence of a particular treatment or intervention than it otherwise would have been in the absence of that treatment or intervention?

Examples:

Does participation in a job training program increase the employment earnings of trainees relative to what they would have been in the absence of training?

Does an increase in the minimum wage decrease the employment levels of low-skilled workers relative to what those employment levels would have been in the absence of the minimum wage increase? Does an increase in workers’ compensation benefits increase the length of time that injured workers remain out of work relative to the length of time they would have remained out of work in the absence of the increase in benefits?

2. Controlled Experiments versus Natural Experiments

There are two broad types of experimental research designs: 1. Controlled or Randomized Experiments

Common in medical research and in the physical and natural sciences Much less common in economics

2. Natural Experiments or Quasi-Experiments

More common in economics



Controlled or Randomized Experiments • A controlled experiment, or randomized experiment, is one in which the

assignment of subjects to treatment and control groups is done randomly, such as by tossing a coin.

• The distinguishing characteristic of a controlled or randomized experiment is the

random assignment of subjects to treatment and control groups. • The major advantage of random assignment is that differences in mean

outcomes between treatment group members and control group members can be confidently attributed to the causal effect of the treatment.

Because the treatment and control groups are randomly selected, there is no reason to think that, in the absence of the treatment, the average outcomes of the control group would differ from the average outcomes of the treatment group.

Random selection of treatment group and control group members therefore guarantees that the experiences of the control group provide a valid counterfactual to those of the treatment group – i.e., that the observed outcomes of the control group are identical on average to the outcomes that the treatment group would have experienced in the absence of the treatment.

Natural Experiments or Quasi-Experiments

• A natural experiment, or quasi-experiment, is one in which some exogenous

event – such as a change in government policy or program – changes the environment in which economic agents (individuals, families, or firms) operate so as to partition the population of agents into two groups: a treatment group which is possibly affected by the event, and a control group which is not affected by the event.

• In a natural experiment, the control and treatment groups are not determined by

random assignment, but instead arise “naturally” or “incidentally” as a result of the exogenous event in question.



Randomized and Natural Experiments: Similarities • A common characteristic of both randomized experiments and natural experiments

is the creation of two groups of entities, agents or subjects:

• a treatment group of comprised of subjects that receive the treatment or experience the event in question;

• a control group comprised of subjects that do not receive the treatment or

experience the event in question. • The control group serves as the counterfactual to the treatment group: it is

used to estimate what would have happened to the treatment group in the absence of the treatment or intervention.

Randomized and Natural Experiments: Differences • The critical difference between randomized experiments and natural experiments

is the mechanism by which entities or agents are partitioned into a treatment group and a control group.

In randomized experiments, subjects are assigned to treatment and control groups by some sort of randomizing procedure. In natural experiments, entities are partitioned into treatment and control groups by some exogenous event or policy intervention.

• Exogeneity of the event or intervention means that assignment to treatment and

control groups is beyond the control of both economic agents and investigators or program administrators.

Economic agents cannot self-select themselves into the treatment group or the control group. Program administrators cannot systematically assign agents to the treatment group or the control group.



3. Elements of a Natural Experiment

Objective: To estimate the causal effect of some “treatment” – i.e., of some exogenous event or policy intervention – on an outcome variable Y.

•

•

Examples of treatments – of program or policy interventions The participation of some workers in a job training program The adoption of (or a mandated increase of) a minimum wage in one jurisdiction An increase in workers’ compensation benefits

Key Elements of a Natural Experiment

1. An exogenous treatment, event, or policy intervention that may affect the

observed outcomes or behavior of those agents that receive or experience the treatment.

2. The existence of two comparison groups of economic agents:

1. A treatment group consisting of those economic entities or agents that receive or experience the treatment, event, or policy intervention.

2. A control group consisting of those economic entities or agents that do not

receive or experience the treatment, event, or policy intervention.



• The Sample Data • The sample data consist of two random cross sections of observations on some

outcome variable Y (and possibly some other variables) corresponding to two different time periods.

Yit = the value of outcome variable Y for individual entity i in period t. 1. The first (earlier) cross-sectional sample – the pre-intervention sample –

consists of observations on Yit taken in period t = 1 before the event or policy intervention.

2. The second (later) cross-sectional sample – the post-intervention sample – consists of observations on Yit taken in period t = 2 after the event or policy intervention.

• Each cross-sectional sample – the pre-intervention sample and the post-

intervention sample – is partitioned into two subsets of observations: 1. Treatment group observations: those entities that were in the treatment group

before and after the event or policy intervention. The treatment group consists of those subjects or entities that received the treatment in question, i.e., that were subjected to or experienced the event, intervention, or policy change in question.

2. Control group observations: those entities that were in the control group

before and after the event or policy intervention. The control group consists of those subjects or entities that did not receive the treatment in question, i.e., that were not subjected to, or did not experience, the event, intervention, or policy change in question, but that experiences some or all of the other influences that affect the treatment group.



4. Example of a Natural Experiment The Card and Krueger (1994, 1995) analysis of the effects of the 1992 increase in the New Jersey state minimum wage rate on the employment of low-wage workers in fast-food restaurants. • The Policy Intervention

In April 1992, the State of New Jersey increased the state minimum wage rate by $0.80 per hour, from $4.25 per hour to $5.05 per hour. The increase went into effect on April 1, 1992. The state minimum wage rate in the adjacent state of Pennsylvania remained unchanged at $4.25 per hour.

• The Empirical Question Addressed

Do binding minimum wage rates reduce the employment of low-wage workers? Do increases in statutory minimum wage rates reduce the employment of low-wage workers relative to what it would have been in the absence of the minimum wage increase?

• Card and Krueger’s Sample Data

Early in 1992, before the increase in the state minimum wage in New Jersey, Card and Krueger conducted a telephone survey of 473 fast-food restaurants in New Jersey and eastern Pennsylvania belonging to four fast-food restaurant chains (Burger King, KFC, Wendy’s, and Roy Rogers). The response rate to their survey was remarkably high: 87 percent. The usable sample consisted of 410 fast-food restaurants: 331 restaurants in New Jersey, and 79 restaurants in Pennsylvania.



In November and December of 1992, about eight months after the higher New Jersey state minimum wage became effective, a second follow-up survey was conducted of the 410 fast-food restaurants in New Jersey and Pennsylvania. Both the first and second waves of the survey collected data for each restaurant on the employment levels and wage rates of supervisory and non-supervisory employees and on other characteristics of the restaurants (price of full meal, hours of operation per week, whether the restaurant was company-owned or operator-owned, location within the state).

• The Outcome Variable

Full-time-equivalent (FTE) employment in each restaurant, where each full-time worker was counted as 1, and each part-time worker was counted as 0.5.

• Comparison Groups: Treatment and Control Groups

The sample of New Jersey fast-food restaurants was classified into three subgroups according to the starting wage rate paid to non-supervisory workers in wave 1, the pre-intervention sample: (1) those with a starting wage rate of exactly $4.25 per hour; (2) those with a starting wage rate between $4.26 and $4.99 per hour; (3) those with a starting wage rate equal to $5.00 or more per hour. Card and Krueger’s analysis of the employment effects of the increase in the New Jersey state minimum wage rate employed several different pairs of comparison groups – i.e., several different pairs of treatment and control groups. 1. One comparison group configuration compared

a treatment group consisting of all fast-food restaurants in New Jersey with a control group consisting of all fast-food restaurants in Pennsylvania



2. A second comparison group configuration compared

a treatment group consisting of low-wage fast-food restaurants in New Jersey (those with starting wage rates of exactly $4.25 per hour in wave 1) with a control group consisting of all fast-food restaurants in Pennsylvania

3. A third comparison group configuration compared

a treatment group consisting of mid-wage fast-food restaurants in New Jersey (those with starting wage rates between $4.26 and $4.99 per hour in wave 1) with a control group consisting of all fast-food restaurants in Pennsylvania

4. A fourth comparison group configuration compared

a treatment group consisting of low-wage fast-food restaurants in New Jersey (those with starting wage rates of exactly $4.25 per hour in wave 1) with a control group consisting of high-wage fast-food restaurants in New Jersey (those with starting wage rates of $5.00 or more per hour in wave 1)

5. A fifth comparison group configuration compared

a treatment group consisting of mid-wage fast-food restaurants in New Jersey (those with starting wage rates between $4.26 and $4.99 per hour in wave 1) with a control group consisting of high-wage fast-food restaurants in New Jersey (those with starting wage rates of $5.00 or more per hour in wave 1)



5. A Simple Difference-in-Differences Estimator of the Treatment Effect

The sample data are partitioned into four mutually exclusive groups (or subsets) of observations:

•

•

1. Treatment group observations before the event or policy intervention; 2. Treatment group observations after the event or policy intervention; 3. Control group observations before the event or policy intervention; 4. Control group observations after the event or policy intervention.

For each of these four subsets of observations, we can compute the sample mean of the outcome variable Yit.

TBY = the sample mean of Yit for treatment group observations before the event or policy change

TAY = the sample mean of Yit for treatment group observations after the event

or policy change

CBY = the sample mean of Yit for control group observations before the event or policy change

CAY = the sample mean of Yit for control group observations after the event or

policy change



• Formulation 1 of the difference-in-differences estimator of the treatment effect • For treatment group members, the change in mean Y associated with the event

or policy intervention is:

TBTATreatment YYY −=∆

• For control group members, the change in mean Y associated with the event or policy intervention is:

CBCAControl YYY −=∆

• The difference-in-differences estimator of the treatment effect can be calculated as the difference between (1) the change in average Y for those in the treatment group ( TreatmentY∆ ) and (2) the change in average Y for those in the control group ( ControlY∆ ):

ControlTreatmentdd YYˆ ∆−∆=∆

= )YY()YY( CBCATBTA −−− (1)

= the treatment group change in the average value of Y from the pre- intervention (before) period to the post-intervention (after) period

)YY( TBTA −

minus the control group change in the average value of Y from the pre- intervention (before) period to the post-intervention (after) period

)YY( CBCA −



• Formulation 2 of the difference-in-differences estimator of the treatment effect • The pre-treatment difference in mean Y between treatment and control groups

is:

CBTBBefore YYY −=∆ • The post-treatment difference in mean Y between treatment and control groups

is:

CATAAfter YYY −=∆ • The difference-in-differences estimator of the treatment effect can also be

calculated as the difference between (1) the post-treatment difference in average Y between the treatment and control groups ( AfterY∆ ) and (2) the pre-treatment difference in average Y between the treatment and control groups ( ControlY∆ ):

BeforeAfterdd YYˆ ∆−∆=∆

= )YY()YY( CBTBCATA −−− (2)

= the post-treatment (after treatment) difference between treatment and control groups in the average value of Y )YY( CATA −

minus the pre-treatment (before treatment) difference between treatment and control groups in the average value of Y )YY( CBTB −



• Equivalence of Formulations 1 and 2 of the difference-in-differences estimator of the treatment effect

Formulation 1 of the simple difference-in-differences estimator of the treatment effect is given by equation (1):

ControlTreatmentdd YYˆ ∆−∆=∆ = )YY()YY( CBCATBTA −−− (1)

Formulation 2 of the simple difference-in-differences estimator of the treatment effect is given by equation (2):

BeforeAfterdd YYˆ ∆−∆=∆ = )YY()YY( CBTBCATA −−− (2)

The equivalence of these two formulations is a matter of simple arithmetic. Start with formulation 1 given by equation (1), and then re-arrange terms:

)YY()YY(ˆCBCATBTAdd −−−=∆ (1)

CBCATBTA YYYY +−−=

CBTBCATA YYYY +−−=

)YY()YY( CBTBCATA −−−= (2)



• Tabular representation of the difference-in-differences estimator of the treatment effect

1 After (At = 1)

2 Before (At = 0) Col. 1 – Col. 2

1. Treatment Group (Ti = 1) TAY TBY TBTA YY −

2. Control Group (Ti = 0) CAY CBY CBCA YY −

Row 1 – Row 2 CATA YY − CBTB YY − dd∆̂

Formulation 1 of the unadjusted difference-in-differences estimator of the treatment effect is:

ControlTreatmentdd YYˆ ∆−∆=∆ = )YY()YY( CBCATBTA −−− (1)

= treatment-group change in average Y − control-group change in average Y Formulation 2 of the unadjusted difference-in-differences estimator of the treatment effect is:

BeforeAfterdd YYˆ ∆−∆=∆ = )YY()YY( CBTBCATA −−− (2)

= treatment-control group difference in average Y after the intervention − treatment-control group difference in average Y before the intervention



• A Regression Model for Computing the Unadjusted Difference-in-Differences Estimate of the Treatment Effect

• The unadjusted (unconditional) difference-in-differences estimate of the effect

of the treatment on outcome variable Yit can be computed by OLS estimation of the following linear regression model:

itit3i2t10it uTATAY +β+β+β+β= i = 1, …, N; t = 1, 2 (3) where:

itY = the value of outcome variable Y for entity i in time period t; tA = the “after” indicator variable, defined to equal 1 if t = 2 and 0 if t = 1;

iT = the “treatment group” indicator variable, defined to equal 1 if entity i is in the treatment group and 0 if entity i is in the control group;

= a random error term for entity i in time period t. itu • The conditional mean value of outcome variable Yit for any given values of the

indicator variables At and Ti is:

( ) it3i2t10itit TATAT,AYE β+β+β+β= i = 1, …, N; t = 1, 2 (4) • Tabulate the conditional mean values of outcome variable Yit for the four groups

of observations identified in the sample data by the indicator variables At and Ti.

1. Control group members before the change, for which = 0 and = 0 tA iT2. Treatment group members before the change, for which = 0 and = 1 tA iT3. Control group members after the change, for which = 1 and = 0 tA iT4. Treatment group members after the change, for which = 1 and = 1 tA iT



• The conditional mean values of outcome variable Yit for the four groups of observations identified in the sample data by the indicator variables At and Ti are:

( ) it3i2t10itit TATAT,AYE β+β+β+β= (4)

1. Control group members before the intervention, for which = 0 and

= 0 in equation (4): tA

iT

( ) 0itit 0T,0AYE β=== 2. Treatment group members before the intervention, for which = 0 and

= 1 in equation (4): tA

iT

( ) 20itit 1T,0AYE β+β=== 3. Control group members after the intervention, for which = 1 and = 0

in equation (4): tA iT

( ) 10itit 0T,1AYE β+β===

4. Treatment group members after the intervention, for which = 1 and

= 1 in equation (4): tA iT

( ) 3210itit 1T,1AYE β+β+β+β===



Table 1: Unadjusted Difference-in-Differences Estimate of the Treatment Effect

1 After (At = 1)


1. Treatment (Ti = 1) 3210 β+β+β+β 20 β+β 31 β+β

2. Control (Ti = 0) 10 β+β 0β 1β

Row 1 – Row 2 32 β+β 2β 3β

31 β+β ( ) ( )1T,0AYE1T,1AYE itititit ==−=== = change in conditional mean Y of treatment group between

pre-intervention and post-intervention periods

1β ( ) ( )0T,0AYE0T,1AYE itititit ==−=== = change in conditional mean Y of treatment group between


= 3β ( ) ( )[ ]1T,0AYE1T,1AYE itititit ==−==

( ) ( )[ ]0T,0AYE0T,1AYE itititit ==−==−

32 β+β ( ) ( )0T,1AYE1T,1AYE itititit ==−=== = treatment group-control group difference in conditional mean Y

in the post-intervention period

2β ( ) ( )0T,0AYE1T,0AYE itititit ==−=== = treatment group-control group difference in conditional mean Y

in the pre-intervention period

= 3β ( ) ( )[ ]0T,1AYE1T,1AYE itititit ==−==




Result: The estimate of the coefficient 3β̂ 3β in regression equation (3) is the unadjusted difference-in-differences estimate of the treatment effect on the outcome variable Yit.

3β = ( ) ( )[ ]1T,0AYE1T,1AYE itititit ==−==


= change in conditional mean Y of treatment group between pre-intervention and post-intervention periods minus change in conditional mean Y of control group between pre-intervention and post-intervention periods

3β = ( ) ( )[ ]0T,1AYE1T,1AYE itititit ==−==


= treatment group-control group difference in conditional mean Y in the post-intervention period minus treatment group-control group difference in conditional mean Y in the pre-intervention period



6. An Adjusted Difference-in-Differences Estimator of the Treatment Effect

Objective: To compute a regression-adjusted difference-in-differences

estimate of the treatment effect on the outcome variable Y that adjusts for, or controls for, the systematic effects on Y of other observed characteristics of the units under study.

Suppose there are k observed characteristics of the individual observations, denoted as Z1, Z2, …, Zk. Include these k additional explanatory variables in the row vector

[ ]it,kit,2it,1it ZZZZ L= where Zj,it denotes the value of the control variable Zj for observation i in period t.

Regression Model for Computing an Adjusted Difference-in-Differences Estimate of the Treatment Effect

To control for the effects on the outcome variable Y of individual observed characteristics of treatment and control group members, a vector of observed control variables is included as additional explanatory variables in the linear regression model for the outcome variable Yit:

ititit3i2t10it uZTATAY +γ+β+β+β+β= i = 1, …, N; t = 1, 2 (5) where:

itY = the value of outcome variable Y for entity i in time period t; tA = the “after” indicator variable, defined to equal 1 if t = 2 and 0 if t = 1;

iT = the “treatment group” indicator variable, defined to equal 1 if entity i is in the treatment group and 0 if entity i is in the control group;

itZ = a vector of observed explanatory variables for entity i in time period t;

= a random error term for entity i in time period t. itu



Note: The vector product is simply a compact way of writing the following function of the observed control variables Z

γitZj,it (j = 1, 2, …, k):

it,kkit,22it,11it ZZZZ γ++γ+γ=γ L

• The conditional mean value of outcome variable Yit for any given values of the

indicator variables At and Ti and the control variables Zit is:

( ) γ+β+β+β+β= itit3i2t10ititit ZTATAZ,T,AYE (6)

• Tabulate the conditional mean values of outcome variable Yit, denoted as ( )ititit Z,T,AYE , for the four groups of observations identified in the sample

data:

1. Control group members before the change, for which = 0 and = 0 tA iT2. Treatment group members before the change, for which = 0 and = 1 tA iT3. Control group members after the change, for which = 1 and = 0 tA iT4. Treatment group members after the change, for which = 1 and = 1 tA iT



• The conditional mean values of outcome variable Yit for the four groups of observations identified in the sample data by the indicator variables At and Ti are:

( ) γ+β+β+β+β= itit3i2t10ititit ZTATAZ,T,AYE (6)

1. Control group members before the change, for which = 0 and = 0 in

equation (6) tA iT

( ) γ+β=== it0ititit ZZ,0T,0AYE

2. Treatment group members before the change, for which = 0 and = 1

in equation (6) tA iT

( ) γ+β+β=== it20ititit ZZ,1T,0AYE

3. Control group members after the change, for which = 1 and = 0 in

equation (6) tA iT

( ) γ+β+β=== it10ititit ZZ,0T,1AYE

4. Treatment group members after the change, for which = 1 and = 1 in

equation (6) tA iT

( ) γ+β+β+β+β=== it3210ititit ZZ,1T,1AYE



Table 2: Adjusted Difference-in-Differences Estimate of the Treatment Effect

1 After (At = 1)


1. Treatment (Ti = 1)

γ+β+β+β+β it3210 Z γ+β+β it20 Z 31 β+β

2. Control (Ti = 0)

γ+β+β it10 Z γ+β it0 Z 1β

Row 1 – Row 2 32 β+β 2β 3β

31 β+β ( ) ( )itititititit Z,1T,0AYEZ,1T,1AYE ==−=== = change in conditional mean Y of treatment group between


1β ( ) ( )itititititit Z,0T,0AYEZ,0T,1AYE ==−=== = change in conditional mean Y of control group between


= 3β ( ) ( )[ ]itititititit Z,1T,0AYEZ,1T,1AYE ==−==

( ) ( )[ ]itititititit Z,0T,0AYEZ,0T,1AYE ==−==−

32 β+β ( ) ( )itititititit Z,0T,1AYEZ,1T,1AYE ==−=== = treatment group-control group difference in conditional mean Y

in the post-intervention period

2β ( ) ( )itititititit Z,0T,0AYEZ,1T,0AYE ==−=== = treatment group-control group difference in conditional mean Y

in the pre-intervention period

= 3β ( ) ( )[ ]itititititit Z,0T,1AYEZ,1T,1AYE ==−==




Result: The estimate of the coefficient 3β̂ 3β in regression equation (5) is the adjusted difference-in-differences estimate of the treatment effect on the outcome variable Yit.

ititit3i2t10it uZTATAY +γ+β+β+β+β= i = 1, …, N; t = 1, 2 (5)

3β = ( ) ( )[ ]itititititit Z,1T,0AYEZ,1T,1AYE ==−==


= change in conditional mean Y of treatment group between pre-intervention and post-intervention periods minus change in conditional mean Y of control group between pre-intervention and post-intervention periods

3β = ( ) ( )[ ]itititititit Z,0T,1AYEZ,1T,1AYE ==−==


= treatment group-control group difference in conditional mean Y in the post-intervention period minus treatment group-control group difference in conditional mean Y in the pre-intervention period



7. Threats to the Validity of Natural Experiments Key Fact • The validity of a natural experiment depends crucially on the validity of the

control group. • In the absence of random assignment of treatment status, there is no guarantee that

treatment group members and control group members would be identical in the absence of the treatment.

• Card and Krueger (1995, pp. 23-24) suggest that the validity of a particular control

group can be assessed by asking the following questions.

1. Are the observed characteristics of the treatment and control groups essentially similar prior to the event or intervention under study?

2. Have the observed outcomes or behaviors of the treatment and control groups

tended to move together in the past (prior to the intervention or event under study)?

3. Can the intervention or event plausibly be viewed as exogenous to the

determination of treatment status, i.e., to the assignment of entities to the treatment and control groups?

4. Are there other control groups that can serve as plausible alternatives to the

particular control group in question?

Affirmative answers to these four questions enhance confidence in the validity of the control group.



Some Specific Threats to Validity of Natural Experiments (see Meyer 1995, pp. 152-153) 1. Omitted variables: Events other than the treatment or intervention that occur

between the pre-intervention and post-intervention observations and that could provide alternative explanations for the difference in measured outcomes between treatment and control groups.

2. Mismeasurement of variables: Changes in variable definitions or survey

methods between pre-intervention and post-intervention periods that produce changes in measured variables.

3. Endogeneity of policy interventions that occurs when governments respond to

variables associated either with past observed outcomes or future expected outcomes.

4. Simultaneity that occurs when one or more explanatory variables are jointly

determined with the outcome variable. 5. Selection problems that occur when the assignment of observations to treatment

and control groups results in correlation between the treatment variable and the outcome variable in the absence of treatment.

Example: Entities may be assigned to the treatment group based on past values of the outcome variable or on the values of other variables that are correlated with the outcome variable. In job training programs, program managers may assign to the treatment group of trainees individuals who have experienced recent decreases in earnings; this is the phenomenon known as the Ashenfelter dip.

6. Sample attrition: Different rates of attrition from treatment and control groups between the pre-intervention and post-intervention periods.

7. Omitted interactions: Different trends in outcomes between treatment and

control groups, or omitted explanatory variables that change in different ways for treatment group and control group members.



References Card, David, and Krueger, Alan B. (1994). “Minimum Wages and Employment: A

Case Study of the Fast Food Industry in New Jersey and Pennsylvania,” American Economic Review 84 (September), 772-793.

Card, David, and Krueger, Alan B. (1995). Myth and Measurement: The New

Economics of the Minimum Wage. Princeton, NJ: Princeton University Press. Meyer, Bruce D. (1995). “Natural and Quasi-Experiments in Economics,” Journal of

Business and Economic Statistics 13:2 (April), 151-161.


natural experiments in economics: an introduction 1...

Documents