least squares approximation: a linear algebra technique
DESCRIPTION
Least Squares Approximation: A Linear Algebra Technique. By Larry Wong and James Sfregola. Soo……You have a bunch of Data. What can we do with this? Use Least Squares Approximation (LSE) to estimate the relationship of the dependent variable and the independent variable(s). - PowerPoint PPT PresentationTRANSCRIPT
Least Squares Approximation: A Linear Algebra Technique
By Larry Wong and James Sfregola
Soo……You have a bunch of Data.
Attendence Big 4 Derby League Position per capita incomeadjusted
population
39,968 1 0 11 35000 1582564
21,852 1 0 10 36725 137471
24,409 1 0 7 36725 371847
26,770 1 0 19 43600 852013
41,917 1 1 2 43600 1351187
37,162 1 0 6 36725 623160
24,510 1 1 16 43600 724121
43,958 1 1 3 36725 738873
40,699 1 0 14 36725 630043
75,595 1 1 1 36725 1194415 What can we do with this?
Use Least Squares Approximation (LSE) to estimate the relationship of the dependent variable and the independent variable(s).
What is Least Squares?What is Least Squares?
• Approximates solutions for inconsistent Approximates solutions for inconsistent over determined systems over determined systems – What does inconsistent mean?What does inconsistent mean?
– Over determined, what’s that?Over determined, what’s that?
• In other words LSE allows you to describe In other words LSE allows you to describe a model as being represented by the line a model as being represented by the line of best fit.of best fit.
• Where the “best fit” line, curve or polynomial is Where the “best fit” line, curve or polynomial is created from the approximations.created from the approximations.
Least Squares it’s Actually Pretty
Useful Least Squares Approximations are
highly relevant in various fields Really, how so?
Still don’t believe us? Fine we’ll give examples…but not now.
So How Does it Estimate a Solution?
Least Squares Approximation estimates a best fit solution for a system by Minimizing the magnitude error
vector, e.
Estimating the coefficients of the best fit equation
How Do We Ensure Positive Error?
Least Squares– Eliminates negative terms– More susceptible to outlying data
Least Absolute Difference– Difficult to work with in linear algebra
`Petal L.`
`Petal
W.`
1 2 3 4 5 6 7
0.51.0
1.52.0
2.5
The Least Squares Theorem For an over determined inconsistent
system Ax=b: This is easily solved using inverse and
transpose multiplication
the resulting vector x can be determined in the form x=(AT A) -1 AT b
where x will correspond to the coefficients in your linear or polynomial expression.
How to Approximate the Error How to Approximate the Error Term?Term?
• ee = = bb-A-Axx , where , where e e is the error termis the error term
• Where e1= the Where e1= the distance from distance from our data point to our data point to the best fit the best fit approximation approximation given by the given by the orthogonal orthogonal projectionprojection
Application to Economics
Econometrics utilizes Ordinary Least Squares Approximations (OLS) Similar to LSE except it follows 7
classical assumptions Estimated Regression Equation
-30000
-20000
-10000
0
10000
20000
30000
0
20000
40000
60000
80000
50 100 150 200 250 300 350
Residual Actual Fitted
BibliographyBibliography
• Poole, David Poole, David Linear AlgebraLinear Algebra: A : A Modern Introduction. Canada: Modern Introduction. Canada: Thomson Brooks/Cole, 2006. Thomson Brooks/Cole, 2006.
• Studenmund, A.H. Using Studenmund, A.H. Using Econometrics: A Practical Guide. New Econometrics: A Practical Guide. New York: Pearson Education, 2006. York: Pearson Education, 2006.