gary_henry_causal attribution for environmental program evaluation

8/14/2019 Gary_Henry_Causal Attribution for Environmental Program Evaluation

1/23

Gary T. HenryMacRae Professor of Public Policy

University of North Carolina at Chapel Hill

2008 Environmental EvaluatorsNetworking Forum

June 12, 2008


2/23

Not just RCT vs. everythingelse

The Received Theory ofCausality has Changed Campbell & Stanley (1966);

Cook and Campbell (1979);Shadish, Cook and Campbell(2002)

Design-based logic of inquiryapproach

Objective: to establish that a

causal relationship exists andcan reasonably begeneralized

Method: Making alternativeexplanations implausible


3/23

Units, individuals or plots of land, havealternative potential outcomes, for example,recycling/not recycling or deforested/forested,

respectively.


4/23

Each unit has alternative outcomesEvaluation Question: Does an environmental

program alter the potential outcomes in thedesired direction for a unit?

For a particular unit, we would like to observethe outcome after the intervention occurredfor two conditions:1. If the unit was included in the intervention;

2. If the unit was not included in the intervention.The objective is an unbiased estimate of the

effect of the program


5/23

Fortunately, someone (Donald Rubin) hasdone the math for us

(but unfortunately the fine print says we have to collectthe data)

The Objective: Unbiased estimate of theeffect of treatment

Possible assignments (X, treatment orcontrol)

Potential outcomes (Y)YiT= outcome for individual i after exposure to

treatment

YiC= outcome for individual i after exposure to control


6/23

Unit PotentialOutcomewithoutProgram

PotentialOutcomewithoutProgram(YiC)

PotentialOutcomewithProgram

PotentialOutcomewithProgram(YiT)

Label

1 deforested 0 forested 1 Programsuccess

2 forested 1 forested 1 Nodifferenc

e3 deforested 0 deforested 0

Nodifferenc

e4 forested 1 deforested 0

Programfailure

These four units exhaust all of the logicalpossibilities


7/23

( )Ti Cii Y Y

The fundamental problem with causal inference:It is impossible to observe the idealcomparison

All designs including RCTs are approximations of the idealCausal inference requires assumptions: RCTs require the fewestExtrapolation of treatment effects to target population requiresadditional assumptions


8/23

Strata

Units inStudy

Population

Potential Outcomes Label

YT YC

1 40 1 0Programsuccess

2 20 1 1No

Difference

3 20 0 0No

difference

4 20 0 1Programfailure

Program produced 60 forested parcelsNo program produces 40 forested parcelsThe program effect was 20 forested parcels or a 1.5increase in forested parcels


9/23

The average treatment effect (ATE)

( )

( ) ( )

( ) ( )

T C

T C

T C

E Y Y

E Y E Y

Y Y


10/23

StrataPercentage

of StudyPopulation

Possible Outcomes

YT YC

ControlGroup

1 20 1 0

2 10 1 1

3 10 0 0

4 10 0 1

TreatmentG

roup

1 20 1 0

210 1 1

3 10 0 0

4 10 0 1

Independence =Equivalence of theStudy PopulationPercentages forEach Strata in

Control andTreatment


11/23

StrataPercentage

of StudyPopulation

Possible Outcomes

YT YC

ControlGroup

1 20 ? 0

2 10 ? 1

3 10 ? 0

4 10 ? 1

TreatmentGroup

1 20 1 ?

2 10 1 ?

3 10 0 ?

4 10 0 ?

Independence =Equivalence of theStudy PopulationPercentages forEach Strata in

Control andTreatment


12/23

To complete the ingredients needed forcausal attribution (unbiased effect sizeestimate) we need a switch to assign unitsto treatment and control

We need a switch that meets theindependence assumption: createsequivalent groups

The Switch (S)The switch assigns each individual to treatment (S =

1)or control (S = 0)( | 1) ( | 0)i i i i

E Y S E Y S


13/23


14/23


15/23

Strata

Percentageof Study

Population

Potential Outcomes Observed outcomes

YT YC X YT YC

Contro

lGroup

1 20 1 0 0 * 0

2 10 1 1 0 * 1

3 10 0 0 0 * 0

4 10 0 1 0 * 1

TreatmentGroup

1 20 1 0 1 1 *

2 10 1 1 1 1 *

3 10 0 0 1 0 *

4 10 0 1 1 0 *


16/23


17/23

1. Random assignment to treatment & controlIf independence produces equivalence, extraneous sources of variation

(aka influence of disturbing variables) are equally distributed acrosstreatment and control

Simplifies analysis

2. Matched sampling3. Matched sampling using propensity scores

Propensity scores are each individuals probability of being assigned totreatment

Matches based on finding individual in control similar to each individual intreatment based on propensity scores

4. Cutoff on assignment variable assigns individuals totreatment and control (regression discontinuity)If model correctly specified, produces unbiased estimate of average

treatment effect5. Instrumental variable6. Fixed effects (within individual estimates for panel data)Or using regression to adjust estimates


18/23

Several important studies about differences in effect sizesbetween experimental and observational studies

Lipsey and Wilson (1992)Weisburd, Lum & Petrosino (2001)Glazerman, Levy & Myer (2003) matched sample labor

force interventions; assumed randomized experimentunbiased

5. Matching works well (better w/ one-on-one matchingextensive covariates;

6. Regression works well (better with specification tests,numerous controls, especially pretests);

7. Large sample studies less biased8. Controls selected from similar sitesLarge consensus that regression discontinuity is second

best switch after randomized control trials (van derKlaauw 2003; Trochim, Cappelleri, Reichhardt 1991)


19/23


20/23

LEANAME ADM

Students

Proficient

Teacher

Retention

eac er%

greater

than 5

years

%

Above

Poverty Combined RankWELDON CITY 1,078 51.10 81.72 55.56 73.18 261.56 116 YVANCE COUNTY 8,157 62.70 78.96 58.53 74.06 274.25 115 YHERTFORD COUNTY 3,606 61.30 81.25 65.17 70.78 278.50 114 YHOKE COUNTY 6,593 66.40 72.41 62.47 77.75 279.03 113 YWARREN COUNTY 3,110 67.30 82.79 62.56 70.07 282.72 112 YLEXINGTON CITY 3,162 64.30 86.75 54.88 76.92 282.85 111 Y

NORTHAMPTON COUNTY 3,254 63.80 83.22 67.58 70.52 285.12 110 YHALIFAX COUNTY 5,428 67.10 90.43 62.56 66.19 286.28 109 YTHOMASVILLE CITY 2,666 64.10 78.86 63.27 81.10 287.33 108 YWASHINGTON COUNTY 2,155 57.40 88.36 71.20 73.13 290.09 107 YEDGECOMBE COUNTY 7,591 66.40 81.67 66.98 76.89 291.94 106 YFRANKLIN COUNTY 7,877 72.80 78.47 59.45 81.99 292.71 105 YMONTGOMERY COUNTY 4,502 68.20 81.95 64.26 78.65 293.06 104 YROBESON COUNTY 24,134 67.80 86.03 68.04 74.01 295.88 103 YHYDE COUNTY 670 73.50 85.53 69.01 69.22 297.26 102 YELIZABETH CITY/PASQUOTA 5,902 71.90 81.43 70.55 75.52 299.40 101 YKANNAPOLIS CITY 4,673 72.10 87.43 56.10 83.90 299.53 100

DURHAM COUNTY 30,810 71.20 81.24 64.72 82.70 299.86 99TYRRELL COUNTY 644 82.80 75.44 73.13 68.52 299.89 98BERTIE COUNTY 3,404 65.50 92.31 70.52 71.99 300.32 97DUPLIN COUNTY 8,802 75.60 79.62 67.97 78.25 301.44 96HARNETT COUNTY 16,917 75.50 81.66 63.17 81.22 301.55 95ANSON COUNTY 4,403 63.50 89.86 71.61 78.02 302.99 94GREENE COUNTY 3,222 72.30 86.70 64.84 79.93 303.77 93LENOIR COUNTY 10,211 77.70 79.87 70.21 76.25 304.03 92CHARLOTTE/MECKLENBURG 117,773 75.60 83.27 59.17 86.58 304.62 91


21/23

p-value Coefficient Std. Err

Intercept 0.000 -0.097667 (0.021596 )

Asian mean 0.466 -0.001846 (0.002530 )

Black mean 0.067 0.000669 (0.000365 )Hispanic mean 0.266 0.001985 (0.001781 )

Multiracial mean 0.039 0.008692 (0.004204 )

American Indian mean 0.168 -0.001173 (0.000849 )

Free lunch mean 0.070 -0.001265 (0.000698 )

Reduced lunch mean 0.569 0.000825 (0.001450 )

School size 0.236 0.000016 (0.000014 )

DSSF Dummy 0.000 0.164639 (0.036082 )

Year 2006 0.001 -0.037745 (0.010466 )

Combined 0.000 0.471141 (0.123720 )

Combined Squared 0.020 0.524394 (0.224090 )

Combined Cubed 0.001 -2.669102 (0.731942 )

Regular classroom instruction ---- ---- ----

(Net Effects Model) -All Students

1. Estimates with individual level controls


22/23

StrataPercentage

of TargetPopulation

Potential Outcomes Observed outcomes

YT YC X YT YC

ControlGroup

1 10 1 0 0 * 0

2 5 1 1 0 * 1

3 5 0 0 0 * 0

4 5 0 1 0 * 1

TreatmentGroup

1 10 1 0 1 1 *

2 5 1 1 1 1 *

3 5 0 0 1 0 *

4 5 0 1 1 0 *

UnobservedSample

1 20 1 0 * * *

2 10 1 1 * * *

3 10 0 0 * * *

4 10 0 1 * * *


23/23

1. What kind of evidence isneeded to influenceenvironmental policy andprogram decisions?

2. Is there a program on thehorizon for which it wouldbe helpful to have this

information? likely to have large benefits? highly controversial?

Can you find the resourcesto invest in obtainingtrustworthy informationabout program effects?

Consider the extrapolationproblem how to estimateeffects on target populationbased on study population.

gary_henry_causal attribution for environmental program evaluation

Documents