Econometric Evaluation of Social Programs

Part I: Definitions of Treatment Effects

James J. Heckman and Edward J. Vytlacil

Econ 312, Spring 2019

Heckman and Vytlacil Definitions of Treatment Effects

: More General Criteria

• One might compare outcomes in different sets that are ordered.

• Thus if Y (s, ω) is scalar income and we compare outcomes fors ∈ SA with outcomes for s ′ ∈ SB , where SA ∩ SB = ∅, thenone might compare YsA to YsB , where

sA = argmaxs∈SA(Y (s, ω)) ,

sB = argmaxs∈SB(Y (s, ω)) ,

and we suppress the dependence of sA and sB on ω.

Heckman and Vytlacil Definitions of Treatment Effects

• This compares the best in one choice set with the best in theother.

• Another contrast compares the best choice with the next bestchoice.

• To do so, define s ′ = argmaxs∈S (Y (s, ω)) and SB = S r s ′,and define the treatment effect as Ys′ − YsB .

Heckman and Vytlacil Definitions of Treatment Effects

• Social welfare theory constructs aggregates over Ω ornonempty, nonsingleton subsets of Ω (see Sen, 1999).

• Let sp (ω) denote the s ∈ Sp that ω receives under policy p.

• This is a shorthand notation for the element in Sτ determinedby the map p = (a, τ) assigned to agent ω under policy p.

Heckman and Vytlacil Definitions of Treatment Effects

• A comparison of two policy outcomes sp(ω)ω∈Ω andsp′(ω)ω∈Ω, where p 6= p′ for some ω ∈ Ω, can be

expressed as

RG

(Y (sp (ω) , ω)ω∈Ω

)− RG

(Y (sp′(ω), ω)ω∈Ω

)using the social welfare function defined over outcomesRG

(Y (s, ω) , ωω∈Ω

).

Heckman and Vytlacil Definitions of Treatment Effects

• The cost-benefit comparison of two policies p and p′ is

CBp,p′ =

∫ΩW (Y (sp(ω), ω)) dµ(ω)−

∫ΩW(Y(sp′(ω), ω

))dµ(ω),

where p, p′ are two different policies and p′ may correspond toa benchmark of no policy and µ(ω) is the distribution of ω.

• The Benthamite criterion replaces W (Y (s (ω) , ω)) withR(Y (s(ω), ω)) in the preceding expressions and integratesutilities across agents:

Bp,p′ =

∫ΩR(Y (sp(ω), ω))dµ(ω)−

∫ΩR(Y (sp′(ω), ω))dµ(ω).

Heckman and Vytlacil Definitions of Treatment Effects

: The Evaluation Problem

• Assume a well-defined set of individuals ω ∈ Ω and a universe ofcounterfactuals or hypotheticals for each agent Y (s, ω), s ∈ S.

• Different policies p ∈ P give different incentives by assignmentmechanism a to agents who are allocated to treatment by arule τ ∈ T .

Heckman and Vytlacil Definitions of Treatment Effects

Begin material from previous slide presentations(2).

Heckman and Vytlacil Definitions of Treatment Effects

How To Construct Counterfactuals?

• Central problem in the evaluation literature is the absence ofinformation on outcomes for person ω other than the outcomethat is observed.

• Even a perfectly implemented social experiment does not solvethis problem.

• Randomization with full compliance identifies only onecomponent of Y (s, ω)s∈S for any person.

• In addition, some of the s ∈ S may never be observed.

Heckman and Vytlacil Definitions of Treatment Effects

• For each policy regime, at any point in time we observe personω in some state but not in any of the other states.

• Do not observe Y (s ′, ω) for person ω if we observe Y (s, ω),s 6= s ′.

• Let D (s, ω) = 1 if we observe person ω in state s under policyregime p.

• Observed objective outcome

Y (ω) =∑s∈S

D (s, ω)Y (s, ω) . (1)

Heckman and Vytlacil Definitions of Treatment Effects

• The evaluation problem in this model is that we only observeeach individual in one of S possible states.

• We do not know the outcome of the individual in other statesand hence cannot directly form individual level treatmenteffects.

• The selection problem arises because we only observe certainpersons in any state.

• We observe Y (s, ω) only for persons for whom D (s, ω) = 1.

• In general, the outcomes of persons found in S = s are notrepresentative of what the outcomes of people would be if theywere randomly assigned to s.

Heckman and Vytlacil Definitions of Treatment Effects

• The Roy model (1951): Two possible treatment outcomes(S = 0, 1) and a scalar outcome measure and a particularassignment mechanism D (1, ω) = 1 [Y (1, ω) > Y (0, ω)](reveals R(1, ω)− R(0, ω) ≥ 0).

• The economist’s use of choice data distinguishes theeconometric approach from the statistical approach.

Heckman and Vytlacil Definitions of Treatment Effects

How To Construct Counterfactuals?

• Two main avenues of escape from this problem.

• The first avenue, featured in explicitly formulated econometricmodels and often called “structural econometric analysis ”,derives from the Cowles tradition.

• Models Y (s, ω) explicitly in terms of its determinants asspecified by theory.

• This entails describing the random variables characterizing ωand carefully distinguishing what agents know and what theanalyst knows.

Heckman and Vytlacil Definitions of Treatment Effects

How To Construct Counterfactuals?

• This approach also models D(s, ω) and the dependencebetween Y (s, ω) and D(s, ω) produced from variables commonto Y (s, ω) and D (s, ω).

• Specifies a full model and attempts to addressproblems (P1)–(P3).

Heckman and Vytlacil Definitions of Treatment Effects

How To Construct Counterfactuals?

• A second avenue, pursued in the recent treatment effectliterature, redirects attention away from estimating thedeterminants of Y (s, ω) toward estimating some populationversion of individual “causal effects,” without modeling whatfactors give rise to the outcome or the relationship between theoutcomes and the mechanism selecting outcomes.

• Agent valuations of outcomes are typically ignored.

• The treatment effect literature focuses largely on policyproblem (P-1) for the subset of outcomes that is observed.

• Seeks to answer a narrower problem.

Heckman and Vytlacil Definitions of Treatment Effects

• For program (state, treatment) j compared to program (state,treatment) k ,

ATE(j , k) = E (Y (j , ω)− Y (k , ω)) .

TT(j , k) =E (Y (j , ω)− Y (k , ω) | D(j , ω) = 1) . (2)

• These are the traditional parameters for average returns.

• But for economic analysis, marginal returns are more important.

Heckman and Vytlacil Definitions of Treatment Effects

• The distinction between the marginal and average return is acentral concept in economics.

• The Effect Of Treatment for People at the Margin ofIndifference (EOTM) between j and k , given that these arethe best two choices available is, with respect to personalpreferences, and with respect to choice-specific costs C (j , ω).

Heckman and Vytlacil Definitions of Treatment Effects

EOTMR(j , k) = (3)

E

Y (j , ω)−Y (k, ω)

∣∣∣∣∣∣R (Y (j , ω) ,C (j , ω) , ω) = R (Y (k , ω) ,C (k, ω) , ω) ;R (Y (j , ω) ,C (j , ω) , ω)R (Y (k , ω) ,C (k , ω) , ω)

≥ R (Y (`, ω) ,C (`, ω) , ω)

,

` 6= j , k.

Heckman and Vytlacil Definitions of Treatment Effects

• A generalization of this parameter called the MarginalTreatment Effect, introduced into the evaluation literature byBjorklund and Moffitt (1987).

• Return to people at the margin of choice.

• Will discuss methods for identifying this return.

Heckman and Vytlacil Definitions of Treatment Effects

Policy relevant treatment effect

• Effect on aggregate outcomes of one policy regime p ∈ Pcompared to the effect of another policy regime p′ ∈ P :

PRTE: E (Y (sp(ω), ω)− Y (sp′(ω), ω)),where p, p′ ∈ P .

sp(ω) is treatment allocated under policy p.

• Corresponding to this objective outcome is the subjectivecounterpart:

Subjective PRTE: E (R(sp(ω), ω))− E (R(sp′(ω), ω)),where p, p′ ∈ P .

Heckman and Vytlacil Definitions of Treatment Effects

• Modern political economy seeks to know the proportion ofpeople who benefit from policy regime p compared with p′.Voting Criterion:

Pr (Y (sp(ω), ω) > Y (sp′(ω), ω)) .

• For particular treatments within a policy regime p, it is also ofinterest to determine the proportion who benefit from jcompared to k as

Pr (Y (j , ω) > Y (k , ω)) .

• Option values also interesting: option of having access to aprogram.

• Uncertainty and regret.

Heckman and Vytlacil Definitions of Treatment Effects