16. censoring, tobit and two part models. censoring and corner solution models censoring model: y =...
TRANSCRIPT
16. Censoring, Tobit and Two Part Models
Censoring and Corner Solution Models
• Censoring model:
y = T(y*) = 0 if y* < 0
y = T(y*) = y* if y* > 0. • Corner solution:
y = 0 if some exogenous condition is met;
y = g(x)+e if the condition is not met.
We then model P(y=0) and E[y|x,y>0]. • Hurdle Model:
y = 0 with P(y=0|z)
Model E[y|x,y > 0]
The Tobit Modely* = x’+ε, y = Max(0,y*), ε ~ N[0,2]
Variation: Nonzero lower limit Upper limit Both tails censored
Easy to accommodate. (Already done in major software.)
Log likelihood and Estimation. See Appendix.
(Tobin: “Estimation of Relationships for Limited Dependent Variables,” Econometrica, 1958. Tobin’s probit?)
Conditional Mean Functions
2y* , ~N[0, ], y Max(0,y*)
E[y* | ]
E[y| ]=Prob[y=0| ]×0+Prob[y>0| ]E[y|y>0, ]
=Prob[y*>0| ] E[y*|y*>0, ]
( ) =
( )
= ( )
x β
x x
x x x x
x x
x β x β/x β+
x β/
x β/ x
( )
( ) "Inverse Mills ratio"
( )
( )E[y| ,y>0]=
( )
β + x β/
x β/x β/
x β/x x β+
x β/
Conditional Means
Predictions and Residuals• What variable do we want to predict?
• y*? Probably not – not relevant• y? Randomly drawn observation from the population• y | y>0? Maybe. Depends on the desired function
• What is the residual?• y – prediction? Probably not. What do you do with
the zeros?• Anything - x? Probably not. x is not the mean.
• What are the partial effects? Which conditional mean?
OLS is Inconsistent - Attenuation
E[y| ] = ( / ) ( / )
Nonlinear function of .
What is estimated by OLS regression of y on ?
Slopes of the linear projection are approximately
equal to the derivatives of the conditional me
x x β x β + x β
x
x
an
evaluated at the means of the data.
E[y| ]( / )
Note the ; 0 < ( / ) < 1attenuation
xx β β
xx β
Partial Effects in Censored Regressions
For the latent regression:
E[y*| ]E[y*| ]= ;
E[y| ]E[y| ]= ;
E[y| 0]= /
(a)
xx x'β β
x
x'β x'β x x'βx x'β+ β
x
x'β x'βx,y> x'β+
x'βx'β+ x'β+
E[y| 0]
[1- (a)(a (a))]x,y>
βx
Application: Fair’s Data
Fair’s (1977) Extramarital Affairs Data, 601 observations. Psychology Today. Source: Fair (1977) and http://fairmodel.econ.yale.edu/rayfair/pdf/1978ADAT.ZIP. Several variables not used are denoted X1, ..., X5.y = Number of affairs in the past year, (0,1,2,3,4-10=7, more=12, mean = 1.46. (Frequencies 451, 34, 17, 19, 42, 38)z1 = Sex, 0=female; mean=.476z2 = Age, mean=32.5z3 = Number of years married, mean=8.18z4 = Children, 0=no; mean=.715z5 = Religiousness, 1=anti, …,5=very. Mean=3.12z6 = Education, years, 9, 12, 16, 17, 18, 20; mean=16.2z7 = Occupation, Hollingshead scale, 1,…,7; mean=4.19z8 = Self rating of marriage. 1=very unhappy; 5=very happy
Fair, R., “A Theory of Extramarital Affairs,” Journal of Political Economy, 1978.Fair, R., “A Note on Estimation of the Tobit Model,” Econometrica, 1977.
Fair’s Study
• Corner solution model• Discovered the EM method in the Econometrica
paper• Used the tobit instead of the Poisson (or some
other) count model• Did not account for the censoring at the high
end of the data
Estimated Tobit Model
Discarding the Limit Data
2
22
2
22
y*= +
y=y* if y* > 0, y is unobserved if y* 0.
f(y| )=f(y*| ,y*>0)= Truncated (at zero) normal
1 (y ) 1f(y*| ,y*>0)= exp
Prob[y*>0|x]22
1 (y ) 1 = exp
(22
x β
x x
x βx
x βx β / )
ML is not OLS
Regression with the Truncated Distribution
i i
i
1n
ii 1
( / )E[y| ,y>0]= +
( / )
= +
OLS will be inconsistent:
1Plim = plim plim
n n
A left out variable problem.
Approximately: plim plim(1 - a
ii i
i
i
i
x βx x β
x β
x β
X'Xb β+ x
b β
2
n
i 1
)
1a= , (a)
nGeneral result: Attenuation
ix β/
Estimated Tobit Model
Two Part Specifications
n
i=1
n
i=1
Truncated Regression Probit
Tobit Log Likelihood
logL= dlogf(y| y 0) (1 d)logP(y 0)
= dlogf(y| y 0) dlogP(y 0)
d logP(y>0) + (1 d)logP(y 0)
logL + logL
= logL for the model for y when y > 0 +
logL for the model for whether y is = 0 or > 0.
A two part model: Treat the model for quantity differently
from the model for whether quantity is positive or not.
==> A probit model and a separate truncated regression.
Doctor Visits (Censored at 10)
Two Part Hurdle Model
Critical chi squared [7] = 14.1. The tobit model is rejected.
Panel Data Application
• Pooling: Standard results, incuding “cluster” estimator(s) for asymptotic covariance matrices
• Random effects• Butler and Moffitt – same as for probit• Mundlak/Wooldridge extension – group means• Extension to random parameters and latent
class models• Fixed effects: Some surprises (Greene,
Econometric Reviews, 2005)
Neglected Heterogeneity
it it i it i it
itit it 2 2
c
2 2c
2 2 itc 2 2
c
y * (c ) assuming c
Prob[y * 0| ]
MLE estimates / ( ) Attenuated for
Partial effects are / ( )
Consistently estimated by ML eve
x β x
x βx
β β
x ββ
n though is not
(The standard result.)
β .
Fixed Effects MLE for Tobit
No bias in slopes. Large bias in estimator of
APPENDIX: TOBIT MATH
Estimating the Tobit Model
n ii ii=1
i i i
Log likelihood for the tobit model for estimation of and :
y1logL= (1-d)log d log
d 1 if y 0, 0 if y = 0. Derivatives are very complicated,
Hessian is
i i
β
x β x β
n
i i ii=1
2
i i ii=1
nightmarish. Consider the Olsen transformation*:
=1/ , =- / . (One to one; =1/ ,
logL= (1-d)log d log y
(1-d)log d(log (1/ 2) log2 (1/ 2) y )
i i
i i
β β=-
x x
x x
n
n
i i ii 1
n
i i ii 1
logL(1-d) de
logL 1d ey
*Note on the Uniqueness of the MLE in the Tobit Model," Econometrica, 1978.
ii
i
xx
x
Hessian for Tobit Model
n 2
i i ii=1
n n
i i i i i ii 1 i 1
2n
ii 1
logL= log (1-d)log d(log (1/ 2)log2 (1/ 2) y )
logL logL 1(1-d) de d ey
logL(1-d) (
i i
ii
i
i ii
i i
x x
xx
x
x xx
x x
2
i
2n
ii 1
2n
i i2i 1
d
logL d y
logL 1d d y y
i i
i i
i i
x x
x
Simplified Hessian
22
n
i ii 1
2n
ii 1
2n
i 2i 1
logL(1-d) ( d
logL d y
logL 1d
i ii i i
i i
i i
x xx x x
x x
x
i
2n i i i i
2 2i 1i i i
i i i i i i i i
d y y
((1 d) d) d ylogLdy d(1/ y )
a (a) / (a), (a )
i i
i i i i
i i
i
x x x
x
x
Recovering Structural Parameters
2
2
2
1( , )
( , ) 1
ˆ ˆ( , )ˆUse the delta method to estimate Asy.Var
ˆ( , )ˆˆ
( , ) 1 1( , ) 1
( , )1 1
ˆ ˆ( ,ˆEst.Asy.Var
β
β
βI
IG
0'0'
β
ˆ) ˆ ˆ( , ) Est.Asy.Var ( , )ˆ ˆˆˆ( , )ˆˆ
G G '