16. censoring, tobit and two part models. censoring and corner solution models censoring model: y =...

16. Censoring, Tobit and Two Part Models

Censoring and Corner Solution Models

• Censoring model:

y = T(y*) = 0 if y* < 0

y = T(y*) = y* if y* > 0. • Corner solution:

y = 0 if some exogenous condition is met;

y = g(x)+e if the condition is not met.

We then model P(y=0) and E[y|x,y>0]. • Hurdle Model:

y = 0 with P(y=0|z)

Model E[y|x,y > 0]

The Tobit Modely* = x’+ε, y = Max(0,y*), ε ~ N[0,2]

Variation: Nonzero lower limit Upper limit Both tails censored

Easy to accommodate. (Already done in major software.)

Log likelihood and Estimation. See Appendix.

(Tobin: “Estimation of Relationships for Limited Dependent Variables,” Econometrica, 1958. Tobin’s probit?)

Conditional Mean Functions

2y* , ~N[0, ], y Max(0,y*)

E[y* | ]

E[y| ]=Prob[y=0| ]×0+Prob[y>0| ]E[y|y>0, ]

=Prob[y*>0| ] E[y*|y*>0, ]

( ) =

( )

= ( )

x β

x x

x x x x

x x

x β x β/x β+

x β/

x β/ x

( )

( ) "Inverse Mills ratio"

( )

( )E[y| ,y>0]=

( )

β + x β/

x β/x β/

x β/x x β+

x β/

Conditional Means

Predictions and Residuals• What variable do we want to predict?

• y*? Probably not – not relevant• y? Randomly drawn observation from the population• y | y>0? Maybe. Depends on the desired function

• What is the residual?• y – prediction? Probably not. What do you do with

the zeros?• Anything - x? Probably not. x is not the mean.

• What are the partial effects? Which conditional mean?

OLS is Inconsistent - Attenuation

E[y| ] = ( / ) ( / )

Nonlinear function of .

What is estimated by OLS regression of y on ?

Slopes of the linear projection are approximately

equal to the derivatives of the conditional me

x x β x β + x β

x

x

an

evaluated at the means of the data.

E[y| ]( / )

Note the ; 0 < ( / ) < 1attenuation

xx β β

xx β

Partial Effects in Censored Regressions

For the latent regression:

E[y*| ]E[y*| ]= ;

E[y| ]E[y| ]= ;

E[y| 0]= /

(a)

xx x'β β

x

x'β x'β x x'βx x'β+ β

x

x'β x'βx,y> x'β+

x'βx'β+ x'β+

E[y| 0]

[1- (a)(a (a))]x,y>

βx

Application: Fair’s Data

Fair’s (1977) Extramarital Affairs Data, 601 observations. Psychology Today. Source: Fair (1977) and http://fairmodel.econ.yale.edu/rayfair/pdf/1978ADAT.ZIP. Several variables not used are denoted X1, ..., X5.y = Number of affairs in the past year, (0,1,2,3,4-10=7, more=12, mean = 1.46. (Frequencies 451, 34, 17, 19, 42, 38)z1 = Sex, 0=female; mean=.476z2 = Age, mean=32.5z3 = Number of years married, mean=8.18z4 = Children, 0=no; mean=.715z5 = Religiousness, 1=anti, …,5=very. Mean=3.12z6 = Education, years, 9, 12, 16, 17, 18, 20; mean=16.2z7 = Occupation, Hollingshead scale, 1,…,7; mean=4.19z8 = Self rating of marriage. 1=very unhappy; 5=very happy

Fair, R., “A Theory of Extramarital Affairs,” Journal of Political Economy, 1978.Fair, R., “A Note on Estimation of the Tobit Model,” Econometrica, 1977.

Fair’s Study

• Corner solution model• Discovered the EM method in the Econometrica

paper• Used the tobit instead of the Poisson (or some

other) count model• Did not account for the censoring at the high

end of the data

Estimated Tobit Model

Discarding the Limit Data

2

22

2

22

y*= +

y=y* if y* > 0, y is unobserved if y* 0.

f(y| )=f(y*| ,y*>0)= Truncated (at zero) normal

1 (y ) 1f(y*| ,y*>0)= exp

Prob[y*>0|x]22

1 (y ) 1 = exp

(22

x β

x x

x βx

x βx β / )

ML is not OLS

Regression with the Truncated Distribution

i i

i

1n

ii 1

( / )E[y| ,y>0]= +

( / )

= +

OLS will be inconsistent:

1Plim = plim plim

n n

A left out variable problem.

Approximately: plim plim(1 - a

ii i

i

i

i

x βx x β

x β

x β

X'Xb β+ x

b β

2

n

i 1

)

1a= , (a)

nGeneral result: Attenuation

ix β/

Estimated Tobit Model

Two Part Specifications

n

i=1

n

i=1

Truncated Regression Probit

Tobit Log Likelihood

logL= dlogf(y| y 0) (1 d)logP(y 0)

= dlogf(y| y 0) dlogP(y 0)

d logP(y>0) + (1 d)logP(y 0)

logL + logL

= logL for the model for y when y > 0 +

logL for the model for whether y is = 0 or > 0.

A two part model: Treat the model for quantity differently

from the model for whether quantity is positive or not.

==> A probit model and a separate truncated regression.

Doctor Visits (Censored at 10)

Two Part Hurdle Model

Critical chi squared [7] = 14.1. The tobit model is rejected.

Panel Data Application

• Pooling: Standard results, incuding “cluster” estimator(s) for asymptotic covariance matrices

• Random effects• Butler and Moffitt – same as for probit• Mundlak/Wooldridge extension – group means• Extension to random parameters and latent

class models• Fixed effects: Some surprises (Greene,

Econometric Reviews, 2005)

Neglected Heterogeneity

it it i it i it

itit it 2 2

c

2 2c

2 2 itc 2 2

c

y * (c ) assuming c

Prob[y * 0| ]

MLE estimates / ( ) Attenuated for

Partial effects are / ( )

Consistently estimated by ML eve

x β x

x βx

β β

x ββ

n though is not

(The standard result.)

β .

Fixed Effects MLE for Tobit

No bias in slopes. Large bias in estimator of

APPENDIX: TOBIT MATH

Estimating the Tobit Model

n ii ii=1

i i i

Log likelihood for the tobit model for estimation of and :

y1logL= (1-d)log d log

d 1 if y 0, 0 if y = 0. Derivatives are very complicated,

Hessian is

i i

β

x β x β

n

i i ii=1

2

i i ii=1

nightmarish. Consider the Olsen transformation*:

=1/ , =- / . (One to one; =1/ ,

logL= (1-d)log d log y

(1-d)log d(log (1/ 2) log2 (1/ 2) y )

i i

i i

β β=-

x x

x x

n

n

i i ii 1

n

i i ii 1

logL(1-d) de

logL 1d ey

*Note on the Uniqueness of the MLE in the Tobit Model," Econometrica, 1978.

ii

i

xx

x

Hessian for Tobit Model

n 2

i i ii=1

n n

i i i i i ii 1 i 1

2n

ii 1

logL= log (1-d)log d(log (1/ 2)log2 (1/ 2) y )

logL logL 1(1-d) de d ey

logL(1-d) (

i i

ii

i

i ii

i i

x x

xx

x

x xx

x x

2

i

2n

ii 1

2n

i i2i 1

d

logL d y

logL 1d d y y

i i

i i

i i

x x

x

Simplified Hessian

22

n

i ii 1

2n

ii 1

2n

i 2i 1

logL(1-d) ( d

logL d y

logL 1d

i ii i i

i i

i i

x xx x x

x x

x

i

2n i i i i

2 2i 1i i i

i i i i i i i i

d y y

((1 d) d) d ylogLdy d(1/ y )

a (a) / (a), (a )

i i

i i i i

i i

i

x x x

x

x

Recovering Structural Parameters

2

2

2

1( , )

( , ) 1

ˆ ˆ( , )ˆUse the delta method to estimate Asy.Var

ˆ( , )ˆˆ

( , ) 1 1( , ) 1

( , )1 1

ˆ ˆ( ,ˆEst.Asy.Var

β

β

βI

IG

0'0'

β

ˆ) ˆ ˆ( , ) Est.Asy.Var ( , )ˆ ˆˆˆ( , )ˆˆ

G G '