[Topic 10-Censoring and Truncation] 1/25

10. Censoring, Tobit and Two Part Models

[Topic 10-Censoring and Truncation] 2/25

Censoring and Corner Solution Models• Censoring model: y = T(y*) = 0 if y* < 0 y = T(y*) = y* if y* > 0. • Corner solution: y = 0 if some exogenous condition is met; y = g(x)+e if the condition is not met. We then model P(y=0) and E[y|x,y>0]. • Hurdle Model: y = 0 with P(y=0|z) Model E[y|x,y > 0]

[Topic 10-Censoring and Truncation] 3/25

The Tobit Modely* = x’+ε, y = Max(0,y*), ε ~ N[0,2]

Variation: Nonzero lower limit Upper limit Both tails censored

Easy to accommodate. (Already done in major software.)

Log likelihood and Estimation. See Appendix.

(Tobin: “Estimation of Relationships for Limited Dependent Variables,” Econometrica, 1958. Tobin’s probit?)

[Topic 10-Censoring and Truncation] 4/25

Conditional Mean Functions

2y* , ~N[0, ], y Max(0,y*)E[y* | ]E[y| ]=Prob[y=0| ]×0+Prob[y>0| ]E[y|y>0, ] =Prob[y*>0| ] E[y*|y*>0, ]

( ) = ( ) = ( )

x βx x

x x x xx x

x β x β/x β+ x β/x β/ x

( )( ) "Inverse Mills ratio"( )

( )E[y| ,y>0]= ( )

β + x β/x β/x β/

x β/x x β+ x β/

[Topic 10-Censoring and Truncation] 5/25

Conditional Means

[Topic 10-Censoring and Truncation] 6/25

Predictions and Residuals• What variable do we want to predict?

• y*? Probably not – not relevant• y? Randomly drawn observation from the population• y | y>0? Maybe. Depends on the desired function

• What is the residual?• y – prediction? Probably not. What do you do with

the zeros?• Anything - x? Probably not. x is not the mean.

• What are the partial effects? Which conditional mean?

[Topic 10-Censoring and Truncation] 7/25

OLS is Inconsistent - Attenuation E[y| ] = ( / ) ( / )

Nonlinear function of . What is estimated by OLS regression of y on ?Slopes of the linear projection are approximatelyequal to the derivatives of the conditional me

x x β x β + x βx

x

anevaluated at the means of the data.E[y| ] ( / )

Note the ; 0 < ( / ) < 1attenuation

x x β βxx β

[Topic 10-Censoring and Truncation] 8/25

Partial Effects in Censored Regressions

For the latent regression:E[y*| ]E[y*| ]= ;

E[y| ]E[y| ]= ;

E[y| 0]= /

(a)

xx x'β βx

x'β x'β x x'βx x'β+ βx

x'β x'βx,y> x'β+

x'βx'β+ x'β+

E[y| 0] [1- (a)(a (a))]x,y> βx

[Topic 10-Censoring and Truncation] 9/25

Application: Fair’s DataFair’s (1977) Extramarital Affairs Data, 601 observations. Psychology Today. Source: Fair (1977) and http://fairmodel.econ.yale.edu/rayfair/pdf/1978ADAT.ZIP. Several variables not used are denoted X1, ..., X5.y = Number of affairs in the past year, (0,1,2,3,4-10=7, more=12, mean = 1.46. (Frequencies 451, 34, 17, 19, 42, 38)z1 = Sex, 0=female; mean=.476z2 = Age, mean=32.5z3 = Number of years married, mean=8.18z4 = Children, 0=no; mean=.715z5 = Religiousness, 1=anti, …,5=very. Mean=3.12z6 = Education, years, 9, 12, 16, 17, 18, 20; mean=16.2z7 = Occupation, Hollingshead scale, 1,…,7; mean=4.19z8 = Self rating of marriage. 1=very unhappy; 5=very happy

Fair, R., “A Theory of Extramarital Affairs,” Journal of Political Economy, 1978.Fair, R., “A Note on Estimation of the Tobit Model,” Econometrica, 1977.

[Topic 10-Censoring and Truncation] 10/25

Fair’s Study• Corner solution model• Discovered the EM method in the Econometrica

paper• Used the tobit instead of the Poisson (or some

other) count model• Did not account for the censoring at the high

end of the data

[Topic 10-Censoring and Truncation] 11/25

Estimated Tobit Model

[Topic 10-Censoring and Truncation] 12/25

Discarding the Limit Data

2

22

2

22

y*= +y=y* if y* > 0, y is unobserved if y* 0.f(y| )=f(y*| ,y*>0)= Truncated (at zero) normal

1 (y ) 1f(y*| ,y*>0)= exp Prob[y*>0|x]221 (y ) 1 = exp (22

x β

x xx βx

x βx β / )

ML is not OLS

[Topic 10-Censoring and Truncation] 13/25

Regression with the Truncated Distribution

i i

i

1n

ii 1

( / )E[y| ,y>0]= + ( / ) = +OLS will be inconsistent:

1Plim = plim plimn nA left out variable problem.Approximately: plim plim(1 - a

ii i

i

i

i

x βx xβ xβx β

X'Xb β+ x

b β

2

ni 1

)1a= , (a)n

General result: Attenuationix β/

[Topic 10-Censoring and Truncation] 14/25

Estimated Tobit Model

[Topic 10-Censoring and Truncation] 15/25

Two Part Specifications

ni=1

ni=1

Truncated Regression Probit

Tobit Log LikelihoodlogL= dlogf(y| y 0) (1 d)logP(y 0) = dlogf(y| y 0) dlogP(y 0) d logP(y>0) + (1 d)logP(y 0) logL + logL = logL for the model for y when y > 0 + logL for the model for whether y is = 0 or > 0.A two part model: Treat the model for quantity differentlyfrom the model for whether quantity is positive or not.==> A probit model and a separate truncated regression.

[Topic 10-Censoring and Truncation] 16/25

Doctor Visits (Censored at 10)

[Topic 10-Censoring and Truncation] 17/25

Two Part Hurdle Model

Critical chi squared [7] = 14.1. The tobit model is rejected.

[Topic 10-Censoring and Truncation] 18/25

Panel Data Application• Pooling: Standard results, incuding

“cluster” estimator(s) for asymptotic covariance matrices

• Random effects• Butler and Moffitt – same as for probit• Mundlak/Wooldridge extension – group means• Extension to random parameters and latent

class models• Fixed effects: Some surprises (Greene,

Econometric Reviews, 2005)

[Topic 10-Censoring and Truncation] 19/25

Neglected Heterogeneity

it it i it i it

itit it 2 2

c2 2

c

2 2 itc 2 2

c

y * (c ) assuming c

Prob[y * 0| ]

MLE estimates / ( ) Attenuated for

Partial effects are / ( )

Consistently estimated by ML eve

x β xx βx

β βx ββ

n though is not(The standard result.)

β .

[Topic 10-Censoring and Truncation] 20/25

Fixed Effects MLE for Tobit

No bias in slopes. Large bias in estimator of

[Topic 10-Censoring and Truncation] 21/25

APPENDIX: TOBIT MATH

[Topic 10-Censoring and Truncation] 22/25

Estimating the Tobit Model

n ii ii=1

i i i

Log likelihood for the tobit model for estimation of and :y1logL= (1-d)log d log

d 1 if y 0, 0 if y = 0. Derivatives are very complicated,Hessian is

i i

βx β x β

ni i ii=1

2i i ii=1

nightmarish. Consider the Olsen transformation*:=1/ , =- / . (One to one; =1/ ,

logL= (1-d)log d log y

(1-d)log d(log (1/ 2)log2 (1/ 2) y )i i

i i

β β=-x x

x x

n

ni i ii 1

ni i ii 1

logL (1-d) de

logL 1d ey

*Note on the Uniqueness of the MLE in the Tobit Model," Econometrica, 1978.

ii

i

x xx

[Topic 10-Censoring and Truncation] 23/25

Hessian for Tobit Model

n 2i i ii=1

n ni i i i i ii 1 i 1

2 nii 1

logL= log (1-d)log d(log (1/ 2)log2 (1/ 2) y )

logL logL 1(1-d) de d ey

logL (1-d) (

i i

ii

i

i ii

i i

x x

x xx

x xx x x

2

i

2 nii 1

2 ni i2i 1

d

logL d y

logL 1d d y y

i i

i i

i i

x x

x

[Topic 10-Censoring and Truncation] 24/25

Simplified Hessian

22 ni ii 1

2 nii 1

2 ni 2i 1

logL (1-d) ( d

logL d y

logL 1d

i ii i i

i i

i i

x xx x xx x

x

i

2 n i i i i2 2i 1

i i i

i i i i i i i i

d y y

((1 d) d) d ylogLdy d(1/ y )

a (a) / (a), (a )

i i

i i i i

i i

i

x x xx

x

[Topic 10-Censoring and Truncation] 25/25

Recovering Structural Parameters

2

2

2

1( , )( , ) 1

ˆ ˆ( , )ˆUse the delta method to estimate Asy.Var ˆ( , )ˆˆ

( , ) 1 1( , ) 1 ( , )1 1

ˆ ˆ( ,ˆEst.Asy.Var

β

β

βI I G

0'0'

β

ˆ) ˆ ˆ( , ) Est.Asy.Var ( , )ˆ ˆˆˆ( , )ˆˆ

G G '