multiple regression copyright © 2006 pearson addison-wesley. all rights reserved. 7-1

•Multiple Regression

Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 7-1


Multiple Regression

• We know how to regress Y on a constant and a single X variable

• 1 is the change in Y from a 1-unit change in X

Y 0 1·X


Multiple Regression (cont.)

• Usually we will want to include more than one independent variable.

• How can we extend our procedures to permit multiple X variables?


Gauss–Markov DGP with Multiple X ’s

Y 0

1X

1i

2X

2i

kX

ki

i

E(i) 0

Var(i) 2

Cov(i,

j) 0, for i j

X1X

k fixed across samples (so we can

treat them like constants).


BLUE Estimators

• Ordinary Least Squares is still BLUE

• The OLS formula for multiple X ’s requires matrix algebra, but is very similar to the formula for a single X


BLUE Estimators (cont.)

• Intuitions from the single variable formulas tend to generalize to multiple variables.

• We’ll trust the computer to get the formulas right.

• Let’s focus on interpretation.


Single Variable Regression


Multiple Regression

• 1 is the change in Y from a 1-unit change in X1

Y 0 1X1



• How can we interpret 1 now?

• 1 is the change in Y from a 1-unit change in X1 , holding X2…Xk FIXED

Y 0 1X1 2 X2 k Xk



• How do we implement multiple regression with our software?


Example: Growth

• Regress GDP growth from 1960–1985 on– GDP per capita in 1960 (GDP60)

– Primary school enrollment in 1960 (PRIM60)

– Secondary school enrollment in 1960 (SEC60)

– Government spending as a share of GDP (G/Y)

– Number of coups per year (REV)

– Number of assassinations per year (ASSASSIN)

– Measure of Investment Price Distortions (PPI60DEV)


Hit Table Ext.1.1 A Multiple Regression Model of per Capita GDP Growth.


Example: Growth (cont.)

• A 1-unit increase in GDP in 1960 predicts a 0.008 unit decrease in GDP growth, holding fixed the level of PRIM60, SEC60, G/Y, REV, ASSASSIN, and PPI60DEV.

3.02 – 0.008· 60

0.025· 60 0.031· 60

- 0.119· / –1.950·

- 3.330· – 0.014· 60

GDP Growth GDP

PRIM SEC

G Y REV

ASSASSIN PPI DEV



• Before we controlled for other variables, we found a POSITIVE relationship between growth and GDP per capita in 1960.

• After controlling for measures of human capital and political stability, the relationship is negative, in accordance with “catch up” theory.



• Countries with high values of GDP per capita in 1960 ALSO had high values of schooling and a low number of coups/assassinations.

• Part of the relationship between growth and GDP per capita is actually reflecting the influence of schooling and political stability.

• Holding those other variables constant lets us isolate the effect of just GDP per capita.


Example: Growth

• The Growth of GDP from 1960–1985 was higher:

1. The lower starting GDP, and

2. The higher the initial level of human capital.

• Poor countries tended to “catch up” to richer countries as long as the poor country began with a comparable level of human capital, but not otherwise.



• Bigger government consumption is correlated with lower growth; bigger government investment is only weakly correlated with growth.

• Politically unstable countries tended to have weaker growth.

• Price distortions are negatively related to growth.



• The analysis leaves largely unexplained the very slow growth of Sub-Saharan African countries and Latin American countries.

20

Omitted Variable Bias

The error ε arises because of factors that influence Y but are not

included in the regression function; so, there are always omitted

variables.

Sometimes, the omission of those variables can lead to bias in

the OLS estimator.

21

Omitted variable bias, ctd.

The bias in the OLS estimator that occurs as a result of an

omitted factor is called omitted variable bias. For omitted

variable bias to occur, the omitted factor “Z” must be:

1. A determinant of Y (i.e. Z is part of ε); and

2. Correlated with the regressor X (i.e. corr(Z,X) 0)

Both conditions must hold for the omission of Z to result in

omitted variable bias.

22

Omitted variable bias, ctd.

In the test score example:

1. English language ability (whether the student has English as

a second language) plausibly affects standardized test

scores: Z is a determinant of Y.

2. Immigrant communities tend to be less affluent and thus

have smaller school budgets – and higher STR: Z is

correlated with X.

Accordingly, 1 is biased. What is the direction of this bias?

What does common sense suggest?

If common sense fails you, there is a formula…

23

The omitted variable bias formula:

1 p

1 + uXu

X

If an omitted factor Z is both:

(1) a determinant of Y (that is, it is contained in u); and

(2) correlated with X,

then Xu 0 and the OLS estimator 1 is biased (and is not

consistent). The math makes precise the idea that districts with few ESL students (1) do better on standardized tests and (2) have smaller classes (bigger budgets), so ignoring the ESL factor results in overstating the class size effect.

Is this is actually going on in the CA data?

24

Measures of Fit for Multiple Regression

Actual = predicted + residual: Yi = iY + ie

Se = std. deviation of ie (with d.f. correction)

RMSE = std. deviation of ie (without d.f. correction)

R2 = fraction of variance of Y explained by X

2R = “adjusted R2” = R2 with a degrees-of-freedom correction

that adjusts for estimation uncertainty; 2R < R2

25

Se and RMSE

As in regression with a single regressor, the Se and the RMSE are

measures of the spread of the Y’s around the regression line:

2

2

1

2

1

i

ie

en

RMSE

en

S

26

R2 and 2R

The R2 is the fraction of the variance explained – same definition

as in regression with a single regressor:

R2 = explained SS/Total SS= = TSS

residualSS1 ,

The R2 always increases when you add another regressor

(why?) – a bit of a problem for a measure of “fit”

27

R2 and , ctd.

The 2R (the “adjusted R2”) corrects this problem by “penalizing”

you for including another regressor – the 2R does not necessarily

increase when you add another regressor.

Adjusted R2: ]

1

1)1[(1 22

kn

nRR

Note that 2R < R2, however if n is large the two will be very

close.

2R

28

Measures of fit, ctd.

Test score example:

(1) ·TestScore = 698.9 – 2.28STR,

R2 = .05, Se = 18.6

(2) ·TestScore = 686.0 – 1.10STR – 0.65PctEL,

R2 = .426, 2R = .424, Se = 14.5

What – precisely – does this tell you about the fit of regression (2) compared with regression (1)?

Why are the R2 and the 2R so close in (2)?

29

The Least Squares Assumptions for Multiple Regression

Yi = 0 + 1X1i + 2X2i + … + kXki + ui, i = 1,…,n

1. The conditional distribution of u given the X’s has mean

zero, that is, E(u|X1 = x1,…, Xk = xk) = 0.

2. (X1i,…,Xki,Yi), i =1,…,n, are i.i.d.

3. Large outliers are rare: X1,…, Xk, and Y have four moments:

E( 41iX ) < ,…, E( 4

kiX ) < , E( 4iY ) < .

4. There is no perfect multicollinearity.

Testing Hypotheses


t- test – individual testF-test – joint test

•Functional Form, Scaling and Use of Dummy Variables


Scaling the Data

• Y hat = 40.76 + 0.1283X

• Y = Consumption in $

• X = Income in $

Interpret the equation

• Suppose we change the units of measurement of income (X) to $100 increases

We have scaled the data

Choice of scale does not affect measurement of underlying relationship but affects interpretation of coefficients.

• Now the equation becomes

Yhat = 40.77 + 12.83x

What we did was divided income by 100 so the coefficient of income becomes 100 times larger.

Yhat = 40.77 + (100 * 0.1283)(x/100)

• Scaling X alone changes the slope coefficient

• Changes the standard error of the coefficient by the same factor

• T ratio is unaffected.

• All other regression statistics are unchanged

• Suppose we change the measurement of Y but not X

• All coefficients must change in order for equation to remain valid

• E.g. If Consumption is measured in cents instead of $

• 100 y hat = (100*40.77) + (100*.1283)x

• Yhat* = 4077 + 12.83 X

• Changing the scale of Y alone • All coefficients must change

• Scales standard errors of the coefficients accordingly

• T-ratios and R sq is unchanged

• If X and Y are changed by the same factor

• No change in regression results for slope but estimated intercept will change

• T and Rsq. are unaffected

• Consider the following regressions

• yi = 0 + 1xi + i• Yi = 0* + 1* Xi + i

• yi is measured in inches

• Yi is measured in ft. (12 inches)

• xi is measured in cm.

• Xi is measured in inches. (2.54 cm)

• If estimated 0 = 10 , what is the estimated 0* =

• If estimated. 1* = 22, what is the estimated 1=

Dummy Variables

• Used to capture qualitative explanatory variables

• Used to capture any event that has only two possible outcomes

e.g. race, gender , geographic region of residence etc.

Use of Intercept Dummy

• Most common use of dummy variables.

• Modifies the regression model intercept parameter

e.g. Let test the “location”, “location” “location” model of real estate

Suppose we take into account location near say a university or golf course

• Pt = βo + β1 St +β2 Dt + εt

• St = square footage

• D = dummy variable to represent if the characteristic is present or not

• D = 1 if property is in a desirable neighborhood

• 0 if not in a desirable neighborhood

• Effect of the dummy variable is best seen by examining the E(Pt).

• If model is specified correctly, E(εt )

• = 0

• E(Pt ) = ( βo + β2 ) + β1 St when D=1

βo + β1 St when D = 0

• B2 is the location premium in this case.

• It is the difference between the Price of a house in a desirable are and one in a not so desirable area, all things held constant

• The dummy variable is to capture the shift in the intercept as a result of some qualitative variable

• Dt is an intercept dummy variable

• Dt is treated as any explanatory variable.

• You can construct a confidence interval for B2

• You can test if B2 is significantly different from zero.

• In such a test, if you accept Ho, then there is no difference between the two categories.

• Application of Intercept Dummy Variable

• Wages = B0 + B1EXP + B2RACE +B3SEX + Et

• Race = 1 if white

0 if non white

Sex = 1 if male

0 if female

• WAGES = 40,000 + 1487EXP + 1102RACE +1082SEX

• Mean salary for black female

40,000 + 1487 EXP

Mean salary for white female

41,102 + 1487EXP +1102

• Mean salary for Asian male

• Mean salary for white male

• What sucks more, being female or non white?

• Determining the # of dummies to use

• If h categories, then use h-1 dummies

• Category left out defines reference group

• If you use h dummies you’d fall into the dummy trap

Slope Dummy Variables

• Allows for different slope in the relationship

• Use an interaction variable between the actual variable and a dummy variable

e.g.

Pt = Bo + B1Sqfootage+B2(Sqfootage*D)+et

D= 1 desirable area, 0 otherwise

• Captures the effect of location and size on the price of a house

• E(Pt) = B0 + (B1+B2)Sqfoot if D=1

= BO + B1Sqfoot if D = 0

in the desirable area, price per square foot is b1+b2, and it is b1 in other areas

If we believe that a house location affects both the intercept and the slope then the model is

Pt = B0 +B1sqfoot +B2(sqfoot*D) + B3D +et

54

Dummies for Multiple Categories

• We can use dummy variables to control for something with multiple categories

• Suppose everyone in your data is either a HS dropout, HS grad only, or college grad

• To compare HS and college grads to HS dropouts, include 2 dummy variables

• hsgrad = 1 if HS grad only, 0 otherwise; and colgrad = 1 if college grad, 0 otherwise

55

Multiple Categories (cont)

• Any categorical variable can be turned into a set of dummy variables

• Because the base group is represented by the intercept, if there are n categories there should be n – 1 dummy variables

• If there are a lot of categories, it may make sense to group some together

• Example: top 10 ranking, 11 – 25, etc.

56

Interactions Among Dummies

• Interacting dummy variables is like subdividing the group

• Example: have dummies for male, as well as hsgrad and colgrad

• Add male*hsgrad and male*colgrad, for a total of 5 dummy variables –> 6 categories

• Base group is female HS dropouts

• hsgrad is for female HS grads, colgrad is for female college grads

• The interactions reflect male HS grads and male college grads

57

More on Dummy Interactions

• Formally, the model is y = 0 + 1male + 2hsgrad + 3colgrad + 4male*hsgrad + 5male*colgrad + 1x + u, then, for example:

• If male = 0 and hsgrad = 0 and colgrad = 0

• y = 0 + 1x + u


• y = 0 + 2hsgrad + 1x + u


• y = 0 + 1male + 3colgrad + 5male*colgrad + 1x + u

58

Other Interactions with Dummies

• Can also consider interacting a dummy variable, d, with a continuous variable, x

• y = 0 + 1d + 1x + 2d*x + u

• If d = 0, then y = 0 + 1x + u

• If d = 1, then y = (0 + 1) + (1+ 2) x + u

• This is interpreted as a change in the slope

59

y

x

y = 0 + 1x

y = (0 + 0) + (1 + 1) x

Example of 0 > 0 and 1 < 0

d = 1

d = 0

Multicollinearity

• Omitted Variables Bias is a problem when the omitted variable is an explanator of Y and correlated with X1

• Including the omitted variable in a multiple regression solves the problem.

• The multiple regression finds the coefficient on X1, holding X2 fixed.

0 1 1 2 2

1

1 1

1 2

0

1

0

i i i i

i

i i

i i

Y X X

w

w X

w X

Multicollinearity (cont.)

• Multivariate Regression finds the coefficient on X1, holding X2 fixed.

• To estimate 1, OLS requires:

• Are these conditions always possible?

w1i X2i 0


• To strip out the bias caused by the correlation between X1 and X2 , OLS has to impose the restriction

• This restriction in essence removes those parts of X1 that are correlated with X2

• If X1 is very correlated with X2, OLS doesn’t have much left-over variation to work with.

• If X1 is perfectly correlated with X2, OLS has nothing left.


• Suppose X2 is simply a function of X1

• For some silly reason, we want to estimate the returns to an extra year of education AND the returns to an extra month of education.

• So we stick in two variables, one recording the number of years of education and one recording the number of months of education.

X1 12·X2


1 2

0 1 1 2 2

0 1 2 2 2

0 1 2 2

1 2 1 2

12

(12 )

(12 )

, 12 .

Suppose the marginal contribution of another

month of schooling is .

We can pick any so long as

We cannot uniquely id

X X

Y X X

Y X X

Y X

entify our coefficients.

1

1 1 1 2

1 2

1 2 1 2

1 0

12

12 ( ) 1 0

We need such that

AND

Substituting in ...

AND

i

i i i i

i i

i i i i

w

w X w X

X X

w X w X


• Let’s look at this problem in terms of our unbiasedness conditions.

• No weights can do both these jobs!

X1 aX2 bX3 cX4


• Bottom Line: you CANNOT add variables that are perfectly correlated with each other (and nearly perfect correlation isn’t good).

• You CANNOT include a group of variables that are a linear combination of each other:

• You CANNOT include a group of variables that sum to 1 and also include a constant.


• Multicollinearity is easy to fix. Simply omit one of the troublesome variables.

• Maybe you can find more data for which your variables are not multicollinear. This isn’t possible if your variables are weighted sums of each other by definition.

Checking Understanding

• You have a cross-section of workers from 1999. Which of the following variables would lead to multicollinearity?

1. A Constant, Year of birth, Age

2. A Constant, Year of birth, Years since they finished high school

3. A Constant, Year of birth, Years since they started working for their current employer

Checking Understanding (cont.)

1. A Constant, Year of Birth, and Age will be a problem.

• These variables will be multicollinear (or nearly multicollinear, which is almost as bad).

1999 -

1999·1 -1·

(except for some

slight slippage from month of birth)

Age Birthyear

Age Birthyear


2. A Constant, Year of Birth, and Years Since High School PROBABLY suffers from ALMOST perfect multicollinearity.

• Most Americans graduate from high school around age 18. If this is true in your data, then

1999 - Birthyear 18 Years Since Graduation

Birthyear 1·(1999 18) -1·(Years since H .S.)


3. A Constant, Birthyear, Years with Current Employer is very unlikely to be a problem.

• There is usually ample variation in the ages at which different workers begin their employment with a particular firm.

• Multicollinearity

• When two or more of the explanatory variables are highly related (correlated)

• Collinearity exists so the question is how much before it becomes a problem.

• Perfect multicollinearity

• Imperfect Multicollinearity

• Using the Ballantine

• Detecting Multicollinearity

1. Check simple correlation coefficients (r)

If |r| > 0.8, then multicollinearity may be a problem

2. Perform a t-test at on the correlation coefficient

221

2

r

nrtn

3. Check Variance Inflation Factors (VIF) or the Tolerance (TOL)

• Run a regression of each X on the other Xs

• Calculate the VIF for each Bhati

)1(

1)ˆ(

2i

i RVIF

• The higher VIF, the severity of the problem of multicollinearity

• If VIF is greater than 5, then there might be a problem (arbitrarily chosen)

)ˆ(1 ivif

• Tolerance (TOR) = (1 – Rsq)

0 < TOR < 1

If TOR is close to zero then multicollinearity is severe.

You could use VIF or TOR.

• EFFECTS OF MULTICOLLINEARITY

1. OLS estimates are still unbiased

2. Standard error of the estimated coefficients will be inflated

3. t- statistics will be small

4. Estimates will be sensitive to small changes, either from dropping a variable or adding a few more observations

• With multicollinearity, you may accept Ho for all your t-test but reject Ho for you F-test

Dealing with Multicollinearity

1. Ignore It.

Do this if multicollinearity is not causing any problems.

i.e. if the t-statistics are insignificant and unreliable then do something. If not, do nothing

2. Drop a variable.

If two variables are significantly related, drop one of them (redundant)

3. Increase the sample size

The larger the sample size the more accurate the estimates


X1 aX2 bX3

Review

• Perfect multicollinearity occurs when 2 or more of your explanators are jointly perfectly correlated.

• That is, you can write one of your explanators as a linear function of other explanators:


Review (cont.)

• OLS breaks down with perfect (or even near perfect) multicollinearity.

• Multicollinearity most frequently occurs when you want to include:

– Time, age, and birthyear effects

– A dummy variable for each category, plus a constant


Review (cont.)

• Dummy variables (also called binary variables) take on only the values 0 or 1.

• Dummy variables let you estimate separate intercepts and slopes for different groups.

• To avoid multicollinearity while including a constant, you need to omit the dummy variable for one group (e.g. males or non-Hispanic whites). You want to pick one of the larger groups to omit.

multiple regression copyright © 2006 pearson addison-wesley. all rights reserved. 7-1

Documents

pearson addisonwesley

multiple regression

multiple x s slide

single x slide

fixed slide

multiple x variables

multiple regression

multiple variables