12a - 1 © 2000 prentice-hall, inc. statistics multiple regression and model building chapter 12...

12a - 12a - 11

© 2000 Prentice-Hall, Inc.© 2000 Prentice-Hall, Inc.

StatisticsStatistics

Multiple Regression and Model Multiple Regression and Model BuildingBuilding

Chapter 12 part IChapter 12 part I

12a - 12a - 22


Learning ObjectivesLearning Objectives

1.1. Explain the Linear Multiple Regression Explain the Linear Multiple Regression ModelModel

2.2. Explain Residual AnalysisExplain Residual Analysis

3.3. Test Overall SignificanceTest Overall Significance

4.4. Explain MulticollinearityExplain Multicollinearity

5.5. Interpret Linear Multiple Regression Interpret Linear Multiple Regression Computer OutputComputer Output

12a - 12a - 33


Types of Types of Regression ModelsRegression Models

RegressionModels

LinearNon-

Linear

2+ ExplanatoryVariables

Simple

Non-Linear

Multiple

Linear

1 ExplanatoryVariable

RegressionModels

LinearNon-

Linear

2+ ExplanatoryVariables

Simple

Non-Linear

Multiple

Linear

1 ExplanatoryVariable

12a - 12a - 44


Regression Modeling Regression Modeling Steps Steps

1.1. Hypothesize Deterministic ComponentHypothesize Deterministic Component

2.2. Estimate Unknown Model ParametersEstimate Unknown Model Parameters

3.3. Specify Probability Distribution of Specify Probability Distribution of Random Error TermRandom Error Term Estimate Standard Deviation of ErrorEstimate Standard Deviation of Error

4.4. Evaluate ModelEvaluate Model

5.5. Use Model for Prediction & Estimation Use Model for Prediction & Estimation

12a - 12a - 55








Expanded in

Multiple

RegressionExpanded in

Multiple

Regression

12a - 12a - 66


Linear Multiple Linear Multiple Regression ModelRegression Model

Hypothesizing the Hypothesizing the Deterministic ComponentDeterministic Component

Expanded in

Multiple


Multiple

Regression

12a - 12a - 77





3.3. Specify Probability Distribution of Random Specify Probability Distribution of Random Error TermError Term Estimate Standard Deviation of ErrorEstimate Standard Deviation of Error



12a - 12a - 88


Linear Multiple Linear Multiple Regression ModelRegression Model

1.1. Relationship between 1 dependent & Relationship between 1 dependent & 2 or more independent variables is a 2 or more independent variables is a linear functionlinear function

Y X X Xi i i k ki i 0 1 1 2 2 Y X X Xi i i k ki i 0 1 1 2 2

Dependent Dependent (response) (response) variablevariable

Independent Independent (explanatory) (explanatory) variablesvariables

Population Population slopesslopes

Population Population Y-interceptY-intercept

Random Random errorerror

12a - 12a - 99


X2

Y

X1E(Y) = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(Observed Y )

i

X2

Y

X1E(Y) = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(Observed Y )

i

PopulationPopulation Multiple Multiple Regression ModelRegression Model

Bivariate modelBivariate model

12a - 12a - 1010


Sample Multiple Sample Multiple Regression ModelRegression Model

X2

Y

X1

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(Observed Y)

^

i

Y i = 0 + 1X 1i + 2X 2i

^^^

^^

^^^

^

X2

Y

X1

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(Observed Y)

^

i

Y i = 0 + 1X 1i + 2X 2i

^^^

^^

^^^

^Bivariate modelBivariate model

12a - 12a - 1111


Parameter EstimationParameter Estimation

Expanded in

Multiple


Multiple

Regression

12a - 12a - 1212








12a - 12a - 1313


Multiple Linear Multiple Linear Regression Regression EquationsEquations

Too complicated

by hand! Ouch!

12a - 12a - 1414


Interpretation of Interpretation of Estimated Estimated

CoefficientsCoefficients

12a - 12a - 1515



CoefficientsCoefficients1.1. Slope (Slope (kk))

Estimated Estimated YY Changes by Changes by kk for Each 1 for Each 1

Unit Increase in Unit Increase in XXkk Holding All Other Holding All Other

Variables ConstantVariables Constant Example: If Example: If 11 = 2, then Sales ( = 2, then Sales (YY) Is Expected ) Is Expected

to Increase by 2 for Each 1 Unit Increase in to Increase by 2 for Each 1 Unit Increase in Advertising (Advertising (XX11) Given the Number of Sales ) Given the Number of Sales

Rep’s (Rep’s (XX22) )

^̂

^̂

^̂

12a - 12a - 1616



CoefficientsCoefficients1.1. Slope (Slope (kk))

Estimated Estimated YY Changes by Changes by kk for Each 1 Unit for Each 1 Unit

Increase in Increase in XXkk Holding All Other Variables Holding All Other Variables

ConstantConstant Example: If Example: If 11 = 2, then Sales ( = 2, then Sales (YY) Is Expected to ) Is Expected to

Increase by 2 for Each 1 Unit Increase in Increase by 2 for Each 1 Unit Increase in Advertising (Advertising (XX11) Given the Number of Sales Rep’s ) Given the Number of Sales Rep’s

((XX22) )

2.2. Y-Intercept (Y-Intercept (00)) Average Value of Average Value of YY When When XXkk = 0 = 0

^̂

^̂

^̂

^̂

12a - 12a - 1717


Parameter Parameter Estimation ExampleEstimation Example

You work in advertising for You work in advertising for the New York Times. You the New York Times. You want to find the effect of want to find the effect of ad sizead size (sq. in.) & (sq. in.) & newspaper newspaper circulationcirculation (000) on the number of (000) on the number of ad ad responsesresponses (00). (00).

You’ve collected the You’ve collected the following data:following data:

RespResp SizeSize CircCirc

11 11 2244 88 8811 33 1133 55 7722 66 4444 1010 66

12a - 12a - 1818


ParameterParameter Estimation Estimation

Computer OutputComputer Output Parameter EstimatesParameter Estimates

ParameterParameter Standard T for H0: Standard T for H0:Variable Variable DF DF Estimate Estimate Error Param=0 Prob>|T|Error Param=0 Prob>|T|

INTERCEP INTERCEP 1 1 0.0640 0.0640 0.2599 0.246 0.82140.2599 0.246 0.8214

ADSIZE ADSIZE 1 1 0.2049 0.2049 0.0588 3.656 0.03990.0588 3.656 0.0399

CIRC CIRC 1 1 0.2805 0.2805 0.0686 4.089 0.02640.0686 4.089 0.0264

P

2

0

1^̂

^̂

^̂

^̂

12a - 12a - 1919


Interpretation of Interpretation of Coefficients SolutionCoefficients Solution

12a - 12a - 2020



1.1. Slope (Slope (11)) # Responses to Ad Is Expected to Increase # Responses to Ad Is Expected to Increase

by .2049 (20.49) for Each 1 Sq. In. Increase by .2049 (20.49) for Each 1 Sq. In. Increase in Ad Size in Ad Size Holding Circulation ConstantHolding Circulation Constant

^̂

12a - 12a - 2121




by .2049 (20.49) for Each 1 Sq. In. Increase in by .2049 (20.49) for Each 1 Sq. In. Increase in Ad Size Ad Size Holding Circulation ConstantHolding Circulation Constant


by .2805 (28.05) for Each 1 Unit (1,000) by .2805 (28.05) for Each 1 Unit (1,000) Increase in CirculationIncrease in Circulation Holding Ad Holding Ad Size Size ConstantConstant

^̂

^̂

12a - 12a - 2222


Evaluating the ModelEvaluating the Model

Expanded in

Multiple


Multiple

Regression

12a - 12a - 2323








12a - 12a - 2424


Evaluating Multiple Evaluating Multiple Regression Model Regression Model

StepsSteps

1.1. Examine Variation MeasuresExamine Variation Measures

2.2. Do Residual AnalysisDo Residual Analysis

3.3. Test Parameter SignificanceTest Parameter Significance Overall ModelOverall Model Individual CoefficientsIndividual Coefficients

4.4. Test for MulticollinearityTest for Multicollinearity

12a - 12a - 2525



StepsSteps





NewNew

!!NewNew

!!

ExpanExpan

ded!ded!NewNew

!!

12a - 12a - 2626


Variation MeasuresVariation Measures

12a - 12a - 2727



StepsSteps





NewNew

!!NewNew

!!

ExpanExpan

ded!ded!NewNew

!!

12a - 12a - 2828


Coefficient of Coefficient of MultipleMultiple

DeterminationDetermination1.1. Proportion of Variation in Proportion of Variation in YY ‘Explained’ by ‘Explained’ by

All All XX Variables Variables Taken TogetherTaken Together

RR22 = = Explained VariationExplained Variation = = SSRSSR

Total VariationTotal Variation SS SSyyyy

2.2. Never Decreases When New Never Decreases When New XX Variable Is Variable Is Added to ModelAdded to Model Only Only YY Values Determine SS Values Determine SSyyyy

Disadvantage When Comparing ModelsDisadvantage When Comparing Models

12a - 12a - 2929


Residual AnalysisResidual Analysis

12a - 12a - 3030



StepsSteps





NewNew

!!NewNew

!!

ExpanExpan

ded!ded!NewNew

!!

12a - 12a - 3131


Residual AnalysisResidual Analysis

1.1. Graphical Analysis of ResidualsGraphical Analysis of Residuals Plot Estimated Errors vs. Plot Estimated Errors vs. XXii Values Values

Difference Between Actual Difference Between Actual YYii & Predicted & Predicted YYii

Estimated Errors Are Called ResidualsEstimated Errors Are Called Residuals Plot Histogram or Stem-&-Leaf of ResidualsPlot Histogram or Stem-&-Leaf of Residuals

2.2. PurposesPurposes Examine Functional Form (Linear vs. Examine Functional Form (Linear vs.

Non-Linear Model)Non-Linear Model) Evaluate Violations of AssumptionsEvaluate Violations of Assumptions

12a - 12a - 3232


Linear Regression Linear Regression Assumptions Assumptions

1.1. Mean of Probability Distribution of Error Mean of Probability Distribution of Error Is 0Is 0

2.2. Probability Distribution of Error Has Probability Distribution of Error Has Constant VarianceConstant Variance

3.3. Probability Distribution of Error is Probability Distribution of Error is NormalNormal

4.4. Errors Are IndependentErrors Are Independent

12a - 12a - 3333


X

e

Residual Plot Residual Plot for Functional Formfor Functional Form

X

e

Add XAdd X22 Term Term Correct SpecificationCorrect Specification

^ ^

12a - 12a - 3434


Residual Plot Residual Plot for Equal Variancefor Equal Variance

X

SR

Unequal VarianceUnequal Variance

X

SR

Correct SpecificationCorrect Specification

Fan-shaped.Fan-shaped.Standardized residuals used typically. Standardized residuals used typically.

12a - 12a - 3535


Residual Plot Residual Plot for Independencefor Independence

X

SR

Not IndependentNot Independent

X

SR

Correct SpecificationCorrect Specification

Plots reflect sequence data were collected. Plots reflect sequence data were collected.

12a - 12a - 3636


Residual Analysis Residual Analysis Computer OutputComputer Output

Dep Var Predict Dep Var Predict StudentStudent

Obs SALES Value Obs SALES Value Residual Residual -2-1-0 1 2 Residual Residual -2-1-0 1 2

1 1.0000 0.6000 1 1.0000 0.6000 0.4000 1.044 | |** | 0.4000 1.044 | |** |

2 1.0000 1.3000 2 1.0000 1.3000 -0.3000 -0.592 | *| | -0.3000 -0.592 | *| |

3 2.0000 2.00003 2.0000 2.0000 0 0.000 | | | 0 0.000 | | |

4 2.0000 2.70004 2.0000 2.7000 -0.7000 -1.382 | **| | -0.7000 -1.382 | **| |

5 4.0000 3.4000 5 4.0000 3.4000 0.6000 1.567 | |*** | 0.6000 1.567 | |*** |

Plot of standardized (student) residuals

12a - 12a - 3737


Testing ParametersTesting Parameters

12a - 12a - 3838



StepsSteps





NewNew

!!

NewNew

!!

ExpanExpan

ded!ded!NewNew

!!

12a - 12a - 3939


Testing Overall Testing Overall SignificanceSignificance

1.1. Shows If There Is a Linear Relationship Shows If There Is a Linear Relationship Between Between AllAll XX Variables Variables TogetherTogether & & YY

2.2. Uses F Test StatisticUses F Test Statistic

3.3. HypothesesHypotheses HH00: : 1 1 = = 22 = ... = = ... = kk = 0 = 0

No Linear RelationshipNo Linear Relationship

HHaa: At Least One Coefficient Is Not 0 : At Least One Coefficient Is Not 0 At Least One At Least One XX Variable Affects Variable Affects YY

12a - 12a - 4040


Testing Overall Testing Overall SignificanceSignificance

Computer OutputComputer Output

Analysis of VarianceAnalysis of Variance

Sum of Mean Sum of Mean Source DF Squares Square F Value Prob>FSource DF Squares Square F Value Prob>F

Model 2 9.2497 4.6249 55.440 0.0043Model 2 9.2497 4.6249 55.440 0.0043

Error 3 0.2503 0.0834Error 3 0.2503 0.0834

C Total 5 9.5000C Total 5 9.5000

k n - k -1

n - 1 P-Value

MS(Model) MS(Error)

12a - 12a - 4141


MulticollinearityMulticollinearity

12a - 12a - 4242



StepsSteps





NewNew

!!NewNew

!!

ExpanExpan

ded!ded!NewNew

!!

12a - 12a - 4343


MulticollinearityMulticollinearity

1.1. High Correlation Between High Correlation Between XX Variables Variables

2.2. Coefficients Measure Combined EffectCoefficients Measure Combined Effect

3.3. Leads to Unstable Coefficients Leads to Unstable Coefficients Depending on Depending on XX Variables in Model Variables in Model

4.4. Always Exists -- Matter of DegreeAlways Exists -- Matter of Degree

5.5. Example: Using Both Age & Height as Example: Using Both Age & Height as Explanatory Variables in Same Model Explanatory Variables in Same Model

12a - 12a - 4444


Detecting Detecting MulticollinearityMulticollinearity

1.1. Examine Correlation MatrixExamine Correlation Matrix Correlations Between Pairs of Correlations Between Pairs of XX Variables Variables

Are More than With Are More than With YY Variable Variable

2.2. Examine Variance Inflation Factor (VIF)Examine Variance Inflation Factor (VIF) If VIFIf VIFjj > 5, Multicollinearity Exists > 5, Multicollinearity Exists

3.3. Few RemediesFew Remedies Obtain New Sample DataObtain New Sample Data Eliminate One Correlated Eliminate One Correlated X X VariableVariable

12a - 12a - 4545


Correlation Matrix Correlation Matrix Computer OutputComputer Output

Correlation AnalysisCorrelation Analysis

Pearson Corr Coeff /Prob>|R| under HO:Rho=0/ N=6Pearson Corr Coeff /Prob>|R| under HO:Rho=0/ N=6

RESPONSE ADSIZE CIRCRESPONSE ADSIZE CIRC

RESPONSE RESPONSE 1.00000 1.00000 0.90932 0.931170.90932 0.93117

0.0 0.0120 0.0069 0.0 0.0120 0.0069

ADSIZE ADSIZE 0.909320.90932 1.00000 1.00000 0.741180.74118

0.0120 0.0 0.0918 0.0120 0.0 0.0918

CIRC CIRC 0.931170.93117 0.741180.74118 1.000001.00000

0.0069 0.0918 0.0 0.0069 0.0918 0.0

rY1 rY2

All 1’sr12

12a - 12a - 4646


Variance Inflation Variance Inflation Factors Factors

Computer OutputComputer Output

Parameter Standard T for H0:Parameter Standard T for H0:Variable DF Estimate Error Param=0 Prob>|T|Variable DF Estimate Error Param=0 Prob>|T|

INTERCEP 1 0.0640 0.2599 0.246 0.8214INTERCEP 1 0.0640 0.2599 0.246 0.8214

ADSIZE 1 0.2049 0.0588 3.656 0.0399ADSIZE 1 0.2049 0.0588 3.656 0.0399

CIRC 1 0.2805 0.0686 4.089 0.0264CIRC 1 0.2805 0.0686 4.089 0.0264

VarianceVarianceVariable Variable DF DF InflationInflation

INTERCEP INTERCEP 1 1 0.00000.0000

ADSIZE ADSIZE 1 1 2.2190 2.2190

CIRC CIRC 1 1 2.2190 2.2190

VIF1 5

12a - 12a - 4747


Regression CautionsRegression Cautions

12a - 12a - 4848


Regression CautionsRegression Cautions

1.1. Violated AssumptionsViolated Assumptions

2.2. Relevancy of Relevancy of Historical DataHistorical Data

3.3. Level of SignificanceLevel of Significance

4.4. ExtrapolationExtrapolation

5.5. Cause & EffectCause & Effect

12a - 12a - 4949


YY

InterpolationInterpolation

XX

ExtrapolationExtrapolation ExtrapolationExtrapolation

Relevant RangeRelevant Range

ExtrapolationExtrapolation

12a - 12a - 5050


Cause & EffectCause & Effect

Liquor Liquor ConsumptionConsumption

# Teachers# Teachers

12a - 12a - 5151


ConclusionConclusion

1.1. Explained the Linear Multiple Regression Explained the Linear Multiple Regression ModelModel

2.2. Explained Residual AnalysisExplained Residual Analysis

3.3. Tested Overall SignificanceTested Overall Significance

4.4. Explained MulticollinearityExplained Multicollinearity

5.5. Interpreted Linear Multiple Regression Interpreted Linear Multiple Regression Computer OutputComputer Output

End of Chapter

Any blank slides that follow are blank intentionally.

12a - 1 © 2000 prentice-hall, inc. statistics multiple regression and model building chapter 12...

Documents