Treatment Evaluation

Identification

• Graduate and professional economics mainly concerned with identification in empirical work.

• Concept of understanding what is the causal relationship behind empirical results.

Selection Bias

Example 1:• Do hospitals make people healthier?

Selection Bias

National Health Interview Survey (NHIS)• “During the past 12 months, was the

respondent a patient in a hospital overnight?”• “Would you say your health in general is

excellent, very good, good, fair, poor?”(1 is excellent; 5 is poor)

Selection Bias

Selection Bias

• Going to the hospital makes people sicker• It’s not impossible: hospitals are full of other

sick people who might infect us, and dangerous machines and chemicals that might hurt us.

Selection Bias

• People who go to the hospital are probably less healthy to begin with.

• Even after hospitalization, people who have sought medical care are not as healthy, on average, as those who never get hospitalized.

• Though they may well be better after hospitalization than they otherwise would have been.

Selection Bias

Example 2:• Does college education increase wage?

Selection Bias

• College graduates earn 84% more than high school graduates.

Selection Bias

• Selection into college: higher ability, smarter, work harder, etc.

• College graduates would have earned more even without college education.

• Simple comparison can not identify the causal impact of college education on wage.

Solution 1: Randomization

• Random assignment makes the treatment independent of potential outcomes.

• It eliminates selection bias and reveals true treatment effect.

• Treatment effect: compare post-treatment outcome between those who get the treatment and those who don’t.

Solution 1: Randomization

• Example 1: hormone replacement therapy (HRT)• Recommended for middle-aged women to reduce

menopausal symptoms.

Solution 1: Randomization

• Nurses Health Study (non-experimental survey of nurses): better health among the HRT users.

• Randomized trial: few benefits; serious side effects(see, e.g., Women’s Health Initiative [WHI], Hsia, et al., 2006).

Solution 1: Randomization

• Example 2: government-subsidized training programs.

• Provide a combination of classroom instruction and on-the-job training for groups of disadvantaged workers such as the long-term unemployed, drug addicts, and ex-offenders.

• Aim: increase employment and earning.

Solution 1: Randomization

• Non-experimental studies: trainees earn less than comparison groups (see, e.g., Ashenfelter, 1978; Ashenfelter and Card, 1985; Lalonde 1995).

• Evidence from randomized evaluations of training programs generate mostly positive effects (see, e.g., Lalonde, 1986; Orr, et al, 1996).

Solution 1: Randomization

Problems of randomization:• Randomly offers, but people don’t want to be

part of the game• High costs• Small sample size

Solution 2: Difference-in-difference

• Panel data available

Solution 2: Difference-in-difference

• New problem: time trend• Compare change in outcomes between

treatment group and control group• Impact is the difference in the change in

outcome

Impact = (Yt1-Yt0

) - (Yc1-Yc0

)

Solution 2: Difference-in-difference

Pre Post

Solution 2: Difference-in-difference

Effect of program using only pre- & post- data from T group (ignoring general time trend).

Pre Post

Solution 2: Difference-in-difference

Effect of program using only T & C comparison from post-intervention (ignoring pre-existing differences between T & C groups).

Pre Post

Solution 2: Difference-in-difference

• Whatever happened to the control group over time is what would have happened to the treatment group in the absence of the program.

Pre Post

Effect of program difference-in-difference (taking into account pre-existing differences between T & C and general time trend).

Solution 2: Difference-in-difference

• Example:

Schooling and labor market consequences of school construction

in Indonesia: evidence from an unusual policy experiment

Esther Duflo, MITAmerican Economic Review, Sept 2001

Solution 2: Difference-in-difference

School infrastructure

Educational achievement

Educational achievement?

Salary level?

Solution 2: Difference-in-difference• 1973-1978: The Indonesian government built

61,000 schools equivalent to one school per 500 children between 5 and 14 years old

• The enrollment rate increased from 69% to 85% between 1973 and 1978

• The number of schools built in each region depended on the number of children out of school in those regions in 1972, before the start of the program.

Solution 2: Difference-in-difference• 2 sources of variations in the intensity of the

program for a given individual• By region:

simplify the intensity of the program: high or low

• By age:Young cohort of children who benefittedOlder cohort of children who did not benefit

Solution 2: Difference-in-difference

Intensity of the Building Program

Age in 1974 High Low

2-6 (young cohort)

8.49 9.76

12-17 (older cohort)

8.02 9.4

Difference 0.47 0.36 0.12 DD(0.089)

Solution 2: Difference-in-difference

• Fundamental assumption that trends (slopes) are the same in treatments and controls (sometimes true, sometimes not)

TimeTreatment

Outcome

EstimatedAverage Treatment Effect

Average Treatment Effect

Treatment Group

Control Group

Solution 2: Difference-in-difference

• Need a minimum of three points in time (age of cohort in the example) to verify this and estimate treatment (two pre-intervention)

TimeTreatment

Outcome

Treatment Group

Control Group

Average Treatment Effect

First

observation

Second

observation

Third

observation

Solution 2: Difference-in-difference

Intensity of the Building Program

Age in 1974 High Low

12-17 8.02 9.40

18-24 7.70 9.12

Difference 0.32 0.28 0.034 DD(0.098)

Solution 3: Matching

• Panel data NOT available• Controls: non-participants with same

characteristics as participants • The matches are selected on the basis of

similarities in observed characteristics

Solution 3: Matching

• Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation

Solution 3: Matching

• For each participant find a sample of non-participants that have similar propensity scores (prob. of treatment)

• Compare the outcome

Solution 3: Matching

• Common support0

.1.2

.3.4

-5 0 5 10x

kdensity treatment kdensity control

Solution 3: Matching

• Assumes no selection bias based on unobserved characteristics