![Page 1: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/1.jpg)
Linear Regression ModelsAndy WangCIS 5930-03
Computer SystemsPerformance Analysis
![Page 2: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/2.jpg)
2
Linear Regression Models
• What is a (good) model?• Estimating model parameters• Allocating variation• Confidence intervals for regressions• Verifying assumptions visually
![Page 3: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/3.jpg)
3
What Is a (Good) Model?
• For correlated data, model predicts response given an input
• Model should be equation that fits data• Standard definition of “fits” is least-
squares– Minimize squared error– Keep mean error zero– Minimizes variance of errors
![Page 4: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/4.jpg)
4
Least-Squared Error• If then error in estimate for
xi is
• Minimize Sum of Squared Errors (SSE)
• Subject to the constraint
xbby 10ˆ iii yye ˆ
n
iii
n
ii xbbye
1
210
1
2
n
iii
n
ii xbbye
110
1
0
![Page 5: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/5.jpg)
5
Estimating Model Parameters
• Best regression parameters are
where
• Note error in book!
221 xnxyxnxyb
xbyb 10
22
11
iii
ii
xxyxxy
yn
yxn
x
![Page 6: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/6.jpg)
6
Parameter EstimationExample
• Execution time of a script for various loop counts:
• = 6.8, = 2.32, xy = 88.7, x2 = 264
•
• b0 = 2.32 (0.30)(6.8) = 0.28
Loops 3 5 7 9 10Time 1.2 1.7 2.5 2.9 3.3
x y
30.0
8.6526432.28.657.8821
b
![Page 7: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/7.jpg)
7
Graph of Parameter Estimation Example
0
1
2
3
0 2 4 6 8 10 12
![Page 8: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/8.jpg)
8
Allocating Variation• If no regression, best guess of y is• Observed values of y differ from ,
giving rise to errors (variance)• Regression gives better guess, but
there are still errors• We can evaluate quality of regression
by allocating sources of errors
yy
![Page 9: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/9.jpg)
9
The Total Sum of Squares
• Without regression, squared error is
0SSSSY
2
2
2SST
2
1
2
2
1
2
2
11
2
1
22
1
2
yny
ynynyy
ynyyy
yyyyyy
n
ii
n
ii
n
ii
n
ii
n
iii
n
ii
![Page 10: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/10.jpg)
10
The Sum of Squaresfrom Regression
• Recall that regression error is
• Error without regression is SST• So regression explains SSR = SST - SSE• Regression quality measured by
coefficient of determination
22 ˆSSE yye ii
SSTSSESST
SSTSSR2
R
![Page 11: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/11.jpg)
11
Evaluating Coefficientof Determination
• Compute• Compute
• Compute
• where R = R(x,y) = correlation(x,y)
22 )(SST yny xybyby 10
2SSE
SSTSSESST2
R
![Page 12: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/12.jpg)
12
Example of Coefficientof Determination
• For previous regression example
– y = 11.60, y2 = 29.88, xy = 88.7,
– SSE = 29.88-(0.28)(11.60)-(0.30)(88.7) = 0.028– SST = 29.88-26.9 = 2.97– SSR = 2.97-.03 = 2.94– R2 = (2.97-0.03)/2.97 = 0.99
3 5 7 9 101.2 1.7 2.5 2.9 3.3
9.2632.25 22 yn
![Page 13: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/13.jpg)
13
Standard Deviationof Errors
• Variance of errors is SSE divided by degrees of freedom– DOF is n2 because we’ve calculated 2
regression parameters from the data– So variance (mean squared error, MSE) is
SSE/(n2)
• Standard dev. of errors is square root:(minor error in book)
2SSE
nse
![Page 14: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/14.jpg)
14
Checking Degreesof Freedom
• Degrees of freedom always equate:– SS0 has 1 (computed from )– SST has n1 (computed from data and ,
which uses up 1)– SSE has n2 (needs 2 regression
parameters)– So
yy
)2(111SSESSR0SSSSYSST
nnn
![Page 15: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/15.jpg)
15
Example ofStandard Deviation of
Errors• For regression example, SSE was 0.03,
so MSE is 0.03/3 = 0.01 and se = 0.10• Note high quality of our regression:
– R2 = 0.99– se = 0.10
![Page 16: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/16.jpg)
16
Confidence Intervals
for Regressions• Regression is done from a single population sample (size n)– Different sample might give different
results– True model is y = 0 + 1x– Parameters b0 and b1 are really means
taken from a population sample
![Page 17: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/17.jpg)
17
Calculating Intervalsfor Regression
Parameters• Standard deviations of parameters:
• Confidence intervals arewhere t has n - 2 degrees of freedom– Not divided by sqrt(n)
22
22
2
1
0
1
xnxss
xnxx
nss
eb
eb
ibi tsb
![Page 18: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/18.jpg)
18
Example of Regression Confidence Intervals
• Recall se = 0.13, n = 5, x2 = 264, = 6.8• So
• Using 90% confidence level, t0.95;3 = 2.353
x
017.0)8.6(5264
10.0
12.0)8.6(5264
)8.6(5110.0
2
2
2
1
0
b
b
s
s
![Page 19: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/19.jpg)
19
Regression Confidence Example, cont’d
• Thus, b0 interval is
– Not significant at 90%
• And b1 is
– Significant at 90% (and would survive even 99.9% test)
)57.0,004.0()12.0(353.238.0
)34.0,26.0()016.0(353.230.0
![Page 20: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/20.jpg)
20
Confidence Intervalsfor Predictions
• Previous confidence intervals are for parameters– How certain can we be that the parameters
are correct?• Purpose of regression is prediction
– How accurate are the predictions?– Regression gives mean of predicted
response, based on sample we took
![Page 21: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/21.jpg)
21
Predicting m Samples• Standard deviation for mean of future
sample of m observations at xp is
• Note deviation drops as m • Variance minimal at x = • Use t-quantiles with n–2 DOF for interval
22
2
ˆ11
xnxxx
nmss p
eymp
x
![Page 22: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/22.jpg)
22
Example of Confidenceof Predictions
• Using previous equation, what is predicted time for a single run of 8 loops?
• Time = 0.28 + 0.30(8) = 2.68• Standard deviation of errors se = 0.10
• 90% interval is then
11.0)8.6(5264
8.6851110.0 2
2
ˆ1
py
s
)93.2,42.2()11.0(353.268.2
![Page 23: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/23.jpg)
23
Prediction Confidence
x
y
![Page 24: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/24.jpg)
24
Verifying Assumptions
Visually• Regressions are based on assumptions:– Linear relationship between response y and
predictor x• Or nonlinear relationship used in fitting
– Predictor x nonstochastic and error-free– Model errors statistically independent
• With distribution N(0,c) for constant c
• If assumptions violated, model misleading or invalid
![Page 25: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/25.jpg)
25
Testing Linearity• Scatter plot x vs. y to see basic curve type
Linear Piecewise Linear
Outlier Nonlinear (Power)
![Page 26: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/26.jpg)
26
TestingIndependence of Errors• Scatter-plot i versus• Should be no visible trend• Example from our curve fit:
iy
-0.1
0
0.1
0.2
0 1 2 3 4
![Page 27: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/27.jpg)
27
More on Testing Independence
• May be useful to plot error residuals versus experiment number– In previous example, this gives same plot
except for x scaling• No foolproof tests
– “Independence” test really disproves particular dependence
– Maybe next test will show different dependence
![Page 28: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/28.jpg)
28
Testing for Normal Errors
• Prepare quantile-quantile plot of errors• Example for our regression:
-0.2
-0.1
0
0.1
0.2
-1.5 -1 -0.5 0 0.5 1 1.5
![Page 29: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/29.jpg)
29
Testing for Constant Standard Deviation
• Tongue-twister: homoscedasticity• Return to independence plot• Look for trend in spread• Example:
-0.1
0
0.1
0.2
0 1 2 3 4
![Page 30: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/30.jpg)
30
Linear RegressionCan Be Misleading
• Regression throws away some information about the data– To allow more compact summarization
• Sometimes vital characteristics are thrown away– Often, looking at data plots can tell you
whether you will have a problem
![Page 31: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/31.jpg)
31
Example ofMisleading RegressionI II III IV
xy xy xy xy108.0410 9.1410 7.46 86.58
86.958 8.148 6.77 85.76137.5813 8.741312.74 87.71
98.819 8.779 7.11 88.84118.3311 9.2611 7.81 88.47149.9614 8.1014 8.84 87.04
67.246 6.136 6.08 85.2544.264 3.104 5.39 1912.50
1210.84 129.1312 8.158 5.5674.827 7.267 6.42 87.9155.685 4.745 5.73 86.89
![Page 32: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/32.jpg)
32
What Does Regression Tell Us About These
Data Sets?• Exactly the same thing for each!• N = 11• Mean of y = 7.5• Y = 3 + .5 X• Standard error of regression is 0.118• All the sums of squares are the same• Correlation coefficient = .82• R2 = .67
![Page 33: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/33.jpg)
33
Now Look at the Data Plots
0
2
4
6
8
10
12
0 5 10 15 200
2
4
6
8
10
12
0 5 10 15 20
0
2
4
6
8
10
12
0 5 10 15 200
2
4
6
8
10
12
0 5 10 15 20
I II
III IV
![Page 34: Linear Regression Models Andy Wang CIS 5930-03 Computer Systems Performance Analysis](https://reader033.vdocuments.us/reader033/viewer/2022050813/5a4d1b7b7f8b9ab0599b91c0/html5/thumbnails/34.jpg)
White Slide