copyright © 2006 pearson addison-wesley. all rights reserved. lecture 23: experiments (chapter...

Copyright © 2006 Pearson Addison-Wesley. All rights reserved.

Lecture 23:Experiments

(Chapter 15.1–15.5)

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 23-2

Agenda

• Review

• Negative Income Tax Experiments

• Estimating Means (Chapter 15.1)

• Randomized Experiments (Chapter 15.2, 15.5)

• Natural Experiments (Chapter 15.3)

• Differences-in-Differences (Chapter 15.4)


Review

• Under the Gauss–Markov assumptions, OLS is consistent, unbiased, and efficient.

• Under heteroskedasticity or serial correlation, OLS is still consistent and unbiased, but inefficient (and the OLS formula for estimated standard errors is incorrect)

• When X is correlated with , then OLS is inconsistent and biased.


Review (cont.)

• There are many possible reasons why X could be correlated with :–Omitted Variables

– Simultaneity

–Measurement Error


Review (cont.)

• Solutions to contemporaneous correlation must break the link between X and .

• Instrumental Variables breaks this link ex post, isolating part of the variation observed in X that is known to be uncorrelated with .

• Experiments break the link ex ante.


Example: Negative Income Tax

• In the 1970s, the government conducted extensive research on a guaranteed minimum income.

• Individuals with incomes lower than the minimum would have their incomes supplemented by the tax system (a negative income tax).


Example: Negative Income Tax (cont.)

• What would be the effects of an NIT?

• Would the guaranteed income lead to mass emigration from the labor force to public assistance, as low-wage workers lost the incentive to work?

• Policy makers wanted a clear answer, free of biases, and were willing to pay for a massive research program.



• In four separate NIT experiments, over eight thousand households were randomly assigned to receive guaranteed incomes.

• Because the assistance rate was assigned randomly, researchers knew it was uncorrelated with .



• The NIT experiment showed that a guaranteed minimum income had moderate effects on labor effort.

• Heads of households who faced the highest effective tax rate (because if they earned more money they would reduce their subsidy) decreased hours worked by about 11%-15%

• Second earners cut their hours of work more dramatically, by 20-30%



• Randomized social policy experiments have also been used to study:

–Class size (the Tennessee STAR experiment)

–Housing subsidies (HUD’s Moving to Opportunity study)

– Job training programs (the Department of Labor’s Job Training and Partnership Act assessments)



• Randomized social policies are very expensive, and so are conducted infrequently.

• Laboratory studies are becoming increasingly common (using undergraduate subjects in simulated settings, almost always playing for real but small amounts of money). Laboratory studies test economic theory but do not estimate the effects of social programs.



• The basic methodological principle of the NIT research is quite simple: if we assign X directly, then we can eliminate its relationship to (at least at the point of assignment).


Estimating Means (Chapter 15.1)

• In principle, experimental economists can assign a wide range of X values and use OLS methods to estimate coefficients.

• Small-scale laboratory experiments can easily assign a range of X values.

• However, for larger studies, it is often practical to work with only a few X values. The NIT studies are an exception.


Estimating Means (cont.)

• When X takes on only a few values, the parameter of interest shifts from estimating a slope to estimating a mean.

• We have already seen that dummy variables can be used to estimate different means for different groups. Now we will look more closely at estimating means.



• Typically we are interested in knowing the effect of one or more treatments.

• We want to know what outcomes are caused by the treatment/s.

• For example, what effect does the Head Start program have on participants’ grades?

• What effect would moving to a better neighborhood have on residents of impoverished neighborhoods?



• We really want to know the causal effect of the treatment T.

• We want to know what would happen on average to a person randomly chosen from the population if we gave him/her the treatment, as opposed to NOT giving him/her the treatment.

E(Yi| T

i1) - E(Y

i| T

i0)



• Unfortunately, we do not get to observe the same individuals in each state.

• A naïve analyst might simply compare the outcomes for those observed in the treatment state to those not observed in the treatment state.

• In an observational study, we cannot DIRECTLY compare groups that receive the treatment to groups that do not.



• Unfortunately, we do not get to observe the same individuals in each state.

• In an observational study, we cannot DIRECTLY compare groups that receive the treatment to groups that do not.

• Families that choose to join programs like Head Start differ systematically from families that do not join such programs.



• Whenever selection into a treatment is non-random, researchers must worry about unobserved heterogeneity among subjects.

• Some subjects have greater ability, motivation, resources, etc., that make them more likely to seek out and gain access to helpful treatments (and to avoid unhelpful ones).



• Treatments also tend to attract individuals who derive the most benefit from them.

• Public housing is highly unattractive. People in public housing tend to be those with the worst access to alternatives.



• Selection Bias: individuals are sorted into the treatment/non-treatment group on the basis of some underlying characteristic, such as motivation. This underlying characteristic has its own effect on outcomes.

• More motivated parents enroll their children in Head Start.

• Children of motivated parents do better academically.



• A naïve comparison would not estimate

but would come closer to estimating

E(Y | T 1) - E(Y | T 0)

( | 1 & 1) – ( | 0 & 0) E Y T M E Y T M



• The selection bias is a special case of omitted variables bias.

• The goal of experiments is to break the selection bias.



• “Breaking” the selection bias requires the experimenter to intervene in the agent’s world in some way.

• The greater the intervention, the greater the control the economist possesses, and the more certain the economist is to eliminate selection biases.

• The greater the intervention, the greater the danger that the results will not generalize to a more authentic situation.


Randomized Experiments (Chapter 15.2, 15.5)

• The experimenter randomly divides the subjects into two groups, a treatment group and a control group.

• The treatment group receives the treatment (T = 1).

• The control group does not receive the treatment (T = 0).


Randomized Experiments (cont.)

• Individual subjects will vary in myriad unobservable ways.

• By randomly assigning the treatment, the experiment ensures that on average these differences will “cancel out.”

• More motivated subjects are as likely to be in the control group as in the treatment group.



• Because the treatment has been randomly assigned, the treatment and control groups are on average the same (within the bounds of sampling error).

• The only systematic difference between the two groups is the treatment.

• The effect of the treatment can be estimated by comparing the two groups’ means.



• Example: the Moving to Opportunity Program

–HUD randomly assigns a sample of low-income participants to receive a special housing voucher (the treatment group) or to receive no voucher (the control group)



• The vouchers are only valid for housing in “good” neighborhoods.

• Receipt of the voucher is not correlated with any other traits of the participants.

• Actual use of the voucher will be correlated with such traits as motivation, income, and existing connections in a qualifying “good” neighborhood.



• Researchers must be very careful about which effects have been successfully isolated from , and which have not.

• The Moving to Opportunity experiment DOES provide an unbiased estimate of the effect of giving low income families a voucher to move to a “good” neighborhood.

• Such estimates are valuable for assessing proposals for such a voucher program.



• The experiment does NOT provide an unbiased estimate of the effects of living in a good neighborhood vs. a poor neighborhood.

• Not every subject assigned to the treatment group will take advantage of the treatment.

• Treatment subjects will self-select into those who use the treatment, and those who do not.



• Suppose we define the “treatment” as moving from a poor neighborhood to a good neighborhood.

• If the participants in the program are representative of the population of interest, and if everyone who receives a voucher uses it, then we can estimate the average effect of the treatment.




• If the participants in the program are NOT representative of the population of interest, and if everyone who receives a voucher uses it, then we can estimate the average effect of the treatment on the treated.




• If NOT everyone who receives a voucher uses it, then we can estimate the average effect of the treatment on those we intend to treat.


Randomized Experiment (cont.)

• Biases in Randomized Experiments:

–Non-Response Bias: participants do not provide their data to the researchers

– Attrition Bias: participants drop out of the study

– Sample Selection Bias: individuals who agree to participate in a randomized study differ from the population of interest



• General Equilibrium Effects: the experiment shows the results of a small-scale program.

• Implementing the program on a larger scale might change the environment in ways that a smaller scale study does not.



• Impoverished neighborhoods are not greatly affected by the emigration of a few of their most talented residents from a small study.

• If HUD decides to implement the program on a wide scale, poor neighborhoods could be greatly affected.



• Another example of General Equilibrium Effects: training programs

• If a few individuals randomly receive extra job training, their wages increase because (i) they are more productive and (ii) they have a competitive edge.

• If everyone receives the extra training, no one gains a competitive edge.


Natural Experiments (Chapter 15.3)

• Researchers must choose how much control to exert over their experiment.

• Greater control decreases any possible correlations between the treatment and .

• For example, HUD could have (at great expense) moved everyone in the treatment group to a good neighborhood, instead of relying on participants to choose how to use vouchers.


Natural Experiments (cont.)

• In a laboratory experiment, experimental economists can exert great control over every aspect of the subjects’ environment.

• Greater control increases the artificiality of the experiment.

• At some point, economists wish to generalize from the subjects and environments studied to a real program in the population of interest.



• Internal Validity: the ability of the economist to attribute differences between the treatment and control groups to the treatment itself (X and are uncorrelated).

• External Validity: the ability of the economist to generalize from the experiment to the setting and population of interest.



• Often economists face a trade-off between internal and external validity.

• The more they break the natural connections between X and , the more danger arises that the results will not generalize.

• Natural experiments often offer greater external validity (and are much cheaper!)



• A natural experiment is an observational study of a natural setting that appears to assign a treatment in a reasonably random manner.

• Natural experiments have great external validity, at least to the particular setting and population affected.

• Natural experiments have weaker internal validity; the experiment does not control the randomization process directly.



• Example: medical residency clearinghouses

• The American Medical Association runs a centralized clearinghouse to assign medical school graduates to hospitals for residency training programs.

• Hospitals and applicants each submit ranked preference lists, and the AMA assigns matches.



• The AMA hired economist Alvin Roth to redesign their matching procedures.

• Roth knew from game theory that a matching algorithm would work better if it were “stable,” i.e. if no residency/hospital pair would be better off by abandoning their assigned matches to match with each other.

• But is this property really important?



• Roth observed a natural experiment in the United Kingdom.

• In the UK, each region conducts its own regional clearinghouse.

• Some regions adopted matching procedures that were “stable” (i.e. that had desirable game theoretic properties); others did not.



• The Treatment: using a “stable” matching procedure

• The Treatment Group: regions in the UK that had adopted stable procedures

• The Control Group: regions in the UK that had NOT adopted stable procedures



• The Outcome: regions in the Treatment group were still using their matching procedures several years later. Regions in the Control group had abandoned their clearinghouses.



• However, Roth could only HOPE that the two groups differed only in their choice of procedures. Perhaps the Treatment group regions were simply better at running clearinghouses, or had cultural differences?

• The experiment’s internal validity was suspect.

• Roth’s solution: a laboratory experiment.



• Roth designed a simple “matching market” that could be run in a few minutes by undergraduate subjects on computers.

• Some undergraduates pretended to be hospitals; others pretended to be applicants.

• Undergraduates were paid based on their ability to make a “good” match.



• In some randomly chosen laboratory markets, the clearinghouse used the stable rules adopted by the Treatment group in the natural experiment.

• Other laboratory markets used the bad rules adopted by the Control group in the natural experiment.

• The laboratory subjects reached the same outcome as the real UK regions.



• In Roth’s randomized laboratory experiment, the Treatment and Control groups differed ONLY in the rules used.

• However, the laboratory study used a very abstract computer game and undergraduate subjects. External validity is suspect.



• The laboratory experiments, with their strong internal validity, suggest that good game theoretic rules are a major factor in the success of clearinghouses.

• The natural experiments, with their strong external validity, suggest that the rules are important in real markets similar to the one the AMA wished to design.

• The two studies reinforce each other.



• Another example: the Minimum Wage

– Policy makers have long argued over the effects of minimum wage laws

– Benefits: some workers receive higher wages

–Costs: businesses must pay higher wages; some workers become unemployed when employers decide not to pay the higher wage


Hit Figure 15.1 The Nominal and Real Minimum Wage


Natural Experiments

• Card and Krueger uncovered a natural experiment to study the effect of a minimum wage increase on the employment of low-skill teenagers in the fast food industry.

• In 1992, New Jersey rose its state-mandated minimum wage from $4.25 to $5.05. Fast food workers in NJ naturally became a treatment group.



• Card and Krueger needed a control group.

• They chose to focus on fast food workers in neighboring Pennsylvania, which did not increase its minimum wage law.

• NJ and PA are very similar states. Teenagers’ families decisions to live in one or the other are very unlikely to be correlated with NJ’s decision to raise its minimum wage in 1992.



• After the change in the minimum wage, fast food restaurants in NJ employed on average 30.0 workers.

• In our control state of PA, restaurants employed 30.9 workers.

• At first glance, it appears that the higher minimum wage state has slightly smaller employment in fast food restaurants.



• One problem: NJ and PA are similar, but not quite identical.

• Before NJ’s law changed, fast food restaurants in NJ employed on average 29.8 workers.

• PA restaurants employed 33.1 workers.

• We need to take these baseline differences into account.



• PA is not quite identical to NJ

• Perhaps we should look for a different control group.

• We could use NJ restaurants before the law changed.

• These are the exact same restaurants as in the treatment group, so they should be identical.



• Problem: there are other differences between early 1992 and late 1992 than just the change in the law.

• Perhaps the macroeconomy improved.

• Perhaps restaurants increased employment because of seasonal variation in the consumption of fast food.



• The Solution: look at the change in employment in both NJ and PA when the NJ law was enacted.

• Employment per restaurant in NJ increased from 29.8 to 30.0

• Employment per restaurant in PA fell from 33.1 to 30.9


HIT TABLE 15.1 Diff-in-Diffs Estimate of the Employment Effect of New Jersey’s Minimum Wage Increase


Natural Experiments

• NJ and PA do have some differences, but we are HOPING those differences do not change from early 1992 to late 1992.

• Conditions for fast food restaurants do change from early 1992 to late 1992, but we are HOPING those differences are the same in both NJ and PA (except for the NJ minimum wage increase).



• In effect, to interpret our natural experiment, we have used TWO control groups.

• Neither control group is perfect, but using the two together HOPEFULLY eliminates the imperfections.

• This procedure of using two control groups is called Differences-in-Differences (or “Diffs-in-Diffs”).


Differences-in-Differences (Chapter 15.4)

• Let us construct a more general DGP.

• The outcome, Yi , is the result of three factors:

1. Receiving the treatment (e.g. a minimum wage increase)

2. Being in the group that receives the treatment (e.g. NJ)

3. Being in the time after the treatment has occurred (e.g. late 1992)


Differences-in-Differences (cont.)

where DTreatment = 1 if the observation is of a subject assigned to the treatment group, either before or after the treatment is received

and DAfter = 1 if the observation is of a subject in either group after the treatment has been received by the treatment group

Y 0

1DTreatment

2D After

3D After ¥DTreatment



• 1 captures the underlying difference between the treatment and control groups.

• 2 captures the underlying difference between the two time periods.

• 3 captures the effect of the treatment.

0 1 2

3

( )

•

Treatment After

After Treatment

E Y D D

D D



0 1 2

3

( )

•

Treatment After

After Treatment

E Y D D

D D

0 1 2 3 0 2

0 1 0

-

-

Treatment Group Control Group

After

- Before

------------------------------------------------------------------

2 3 2 3-



• One method for estimating 3 is to calculate the means for each group at each time and subtract twice.

0 1 2 3 0 2

0 1 0

-

-


After

- Before

-------------------------------------------------------------------

2 3 2 3-



• However, we want to calculate an estimated standard error for the treatment group. This calculation is not transparent when subtracting group means.

• We also might want to add other explanators.

• Can we express Diffs-in-Diffs in a regression framework?



• We can estimate group means using dummy variables.


Checking Understanding

0 1 2 3

4

Question: which coefficient estimates

the treatment effect, controlling for ?

Assigned After Assigned Afteri i i i i

i i

Y D D D D

X

X


Checking Understanding (cont.)

0 1 2 3

4

0 1 2 3 0 2 1 3

( ) , Controlling for


After

Assigned After Assigned Afteri i i i i

i i

Y D D D D

X

E Y X

0 1 0 1

2 3 2 3

3ˆ

Before

estimates the treatment effect.



• The Great Weakness of Diffs-in-Diffs

– Some other difference may arise between the Treatment and Control groups at the same time that the Treatment occurs.

– Example: suppose at the same time NJ increased the minimum wage, PA introduced a new food labeling law that reduced the demand for fast food.


Review

• We are often interested in knowing the effect of one or more treatments.

• We want to know what outcomes are caused by the treatment/s.


Review (cont.)

• We really want to know the causal effect of the treatment T.

• We want to know what would happen on average to a person randomly chosen from the population if we gave him/her the treatment, as opposed to NOT giving him/her the treatment.

( | 1) - ( | 0)i i i iE Y T E Y T


Review (cont.)

• Whenever selection into a treatment is non-random, researchers must worry about unobserved heterogeneity among subjects.

• Some subjects have greater ability, motivation, resources, etc., that make them more likely to seek out and gain access to helpful treatments (and to avoid unhelpful ones).


Review (cont.)

• Treatments also tend to attract individuals who derive the most benefit from them.


Review (cont.)

• Selection Bias: individuals are sorted into the treatment/non-treatment group on the basis of some underlying characteristic, such as motivation. This underlying characteristic has its own effect on outcomes.


Review (cont.)

• The experimenter randomly divides the subjects into two groups, a treatment group and a control group.

• The treatment group receives the treatment (T = 1).

• The control group does not receive the treatment (T = 0).


Review (cont.)

• Individual subjects will vary in myriad unobservable ways.

• By randomly assigning the treatment, the experiment ensures that on average these differences will “cancel out.”

• More motivated subjects are as likely to be in the control group as in the treatment group.


Review (cont.)

• Because the treatment has been randomly assigned, the treatment and control groups are on average the same (within the bounds of sampling error).

• The only systematic difference between the two groups is the treatment.

• The effect of the treatment can be estimated by comparing the two groups’ means.


Review (cont.)

• If our subject pool is the same as the population of interest, and the entire Treatment group responds to the Treatment, we can estimate the average effect of the Treatment.

• If our subject pool differs from the population of interest, but the entire Treatment group is responsive, we can estimate the average effect of the Treatment on the Treated.


Review (cont.)

• If not all of our Treatment Group responds to the Treatment, then we estimate the average effect of the treatment on those we intend to treat.


Review (cont.)

• Biases in Randomized Experiments:

–Non-Response Bias: participants do not provide their data to the researchers

– Attrition Bias: participants drop out of the study

– Sample Selection Bias: individuals who agree to participate in a randomized study differ from the population of interest


Review (cont.)

• General Equilibrium Effects: the experiment shows the results of a small-scale program.

• Implementing the program on a larger scale might change the environment in ways that a smaller scale study does not.


Review (cont.)

• Internal Validity: the ability of the economist to attribute differences between the treatment and control groups to the treatment itself (X and are uncorrelated).

• External Validity: the ability of the economist to generalize from the experiment to the setting and population of interest.


Review (cont.)

• Often economists face a trade-off between internal and external validity.

• The more they break the natural connections between X and , the more danger arises that the results will not generalize.

• Natural experiments often offer greater external validity.


Review (cont.)

• A natural experiment is an observational study of a natural setting that appears to assign a treatment in a reasonably random manner.

• Natural experiments have great external validity, at least to the particular setting and population affected.

• Natural experiments have weaker internal validity; the experiment does not control the randomization process directly.


Review (cont.)

• Suppose the outcome, Yi , is the result of three factors:

1. Receiving the treatment (e.g. a minimum wage increase)

2. Being in the group that receives the treatment (e.g. NJ)

3. Being in the time after the treatment has occurred (e.g. late 1992)


Review (cont.)

where DTreatment = 1 if the observation is of a subject assigned to the treatment group, either before or after the treatment is received

and DAfter = 1 if the observation is of a subject in either group after the treatment has been received by the treatment group

0 1 2

3 •

Treatment After

After Treatment

Y D D

D D


Review (cont.)

• 1 captures the underlying difference between the treatment and control groups.

• 2 captures the underlying difference between the two time periods.

• 3 captures the effect of the treatment.

0 1 2

3

( )

•

Treatment After

After Treatment

E Y D D

D D


Review: Diffs-in-Diffs

0 1 2

3

( )

•

Treatment After

After Treatment

E Y D D

D D

0 1 2 3 0 2

0 1 0

-

-


After

- Before

------------------------------------------------------------------

2 3 2 3-


Review

• The Great Weakness of Diffs-in-Diffs

– Some other difference may arise between the Treatment and Control groups at the same time that the Treatment occurs.

– Example: suppose at the same time NJ increased the minimum wage, PA introduced a new food labeling law that reduced the demand for fast food.

copyright © 2006 pearson addison-wesley. all rights reserved. lecture 23: experiments (chapter...

Documents

pearson addisonwesley

guaranteed minimum income

estimating means chapter

tax system

randomized experiments

guaranteed income lead

natural experiments

separate nit experiments