![Page 1: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/1.jpg)
LINEAR ALGEBRASourav Sen Gupta
![Page 2: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/2.jpg)
![Page 3: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/3.jpg)
PLAN OF ACTION
Part I — Geometry of Multivariate Data
1. Vector space modelling of multivariate data
2. Understanding Matrices as linear operators
3. Subspaces and Singular Value Decomposition
Part II — Applications in Multivariate Analysis
1. SVD and Principal Component Analysis
2. Eigenvalues and Eigenvectors in PageRank
3. Factor Analysis and SVD in Recommenders
![Page 4: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/4.jpg)
MULTIVARIATE DATA
![Page 5: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/5.jpg)
MULTIVARIATE DATA
p variables/features
n samples/observations
X1 X2 X3 X4S1 5.1 3.5 1.4 0.2S2 4.9 3 1.4 0.2S3 4.7 3.2 1.3 0.2S4 4.6 3.1 1.5 0.2S5 5 3.6 1.4 0.2
S146 6.7 3 5.2 2.3
S147 6.3 2.5 5 1.9
S148 6.5 3 5.2 2
S149 6.2 3.4 5.4 2.3
S150 5.9 3 5.1 1.8
![Page 6: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/6.jpg)
MULTIVARIATE DATA — HISTOGRAMS
![Page 7: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/7.jpg)
MULTIVARIATE DATA — BOXPLOTS
![Page 8: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/8.jpg)
MULTIVARIATE DATA — CORRPLOT
![Page 9: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/9.jpg)
MULTIVARIATE DATA — SCATTERPLOT
![Page 10: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/10.jpg)
MULTIVARIATE DATA — SCATTERPLOT
![Page 11: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/11.jpg)
MULTIVARIATE DATA — SCATTERPLOT
![Page 12: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/12.jpg)
MULTIVARIATE DATA — SCATTERPLOT
![Page 13: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/13.jpg)
MULTIVARIATE DATA — SCATTERPLOTS
![Page 14: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/14.jpg)
MULTIVARIATE DATA — SCATTERPLOTS
![Page 15: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/15.jpg)
MULTIVARIATE DATA — 3D SCATTERPLOT
![Page 16: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/16.jpg)
MULTIVARIATE DATA — 3D SCATTERPLOT
![Page 17: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/17.jpg)
![Page 18: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/18.jpg)
LET’S START WITH 2D
![Page 19: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/19.jpg)
2D GEOMETRY
![Page 20: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/20.jpg)
2D GEOMETRY
(x,y)
x - units
y - units
X
Y
![Page 21: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/21.jpg)
SIZE
X
Y
(x,y)
![Page 22: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/22.jpg)
SIZE
X
Y
(x,y)
size of (x,y) = sqrt(x2 + y2)L2 Norm
![Page 23: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/23.jpg)
SIZE
X
Y
(x,y)
size of (x,y) = abs(x)+ abs(y)L1 Norm
![Page 24: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/24.jpg)
SIZE
X
Y
(x,y)
size of (x,y) = max(abs(x), abs(y))L∞ Norm
![Page 25: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/25.jpg)
UNIT CIRCLE
1
2
∞
![Page 26: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/26.jpg)
ANGLE
X
Y
(x,y)
(u,v)
![Page 27: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/27.jpg)
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = |(x,y)| |(u,v)| cos ADot Product
A
![Page 28: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/28.jpg)
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = |(x,y)| |(u,v)|Dot Product
![Page 29: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/29.jpg)
ANGLE
X
Y
(x,y)
(u,v)
(x,y).(u,v) = 0Dot Product
![Page 30: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/30.jpg)
SIZE AND ANGLE CHEATSHEET
Notation for dot product : or =
L2 norm of a vector :
Distance between two vectors :
Do you see any connection with Statistics?
v · w vTwnX
i=1
viwi
vT v =nX
i=1
v2i = ||v||22
||v � w||2 =
vuutnX
i=1
(vi � wi)2
![Page 31: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/31.jpg)
ANOTHER LOOK AT 2D
![Page 32: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/32.jpg)
2D GEOMETRY
(x,y)
x - units
y - units
X
Y
![Page 33: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/33.jpg)
2D GEOMETRY
(u,v)
u - units
v - units
U
V
![Page 34: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/34.jpg)
2D GEOMETRY
U
V
X
Y
U = aX + bY
V = cX + dY
![Page 35: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/35.jpg)
2D GEOMETRY
U
V
X
Y
UV
�=
a bc d
� XY
�
![Page 36: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/36.jpg)
TRANSFORMATIONS
![Page 37: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/37.jpg)
TRANSLATIONpq
�
![Page 38: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/38.jpg)
ROTATIONcos ✓ sin ✓� sin ✓ cos ✓
�
![Page 39: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/39.jpg)
SCALING↵ 00 �
�
![Page 40: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/40.jpg)
GENERICa bc d
�
![Page 41: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/41.jpg)
ACTION OF A MATRIX
![Page 42: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/42.jpg)
MATRIX-VECTOR MULTIPLICATION
x =
M v w
![Page 43: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/43.jpg)
LINEAR MAP
Mv w
![Page 44: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/44.jpg)
LINEAR MAP
Mv w
R5 R4
![Page 45: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/45.jpg)
Rn RmMm⇥n
Domain of the function? — Where does the input originate from?
Codomain of the function? — What are possible values of output?
Range of the function? — What are the actual values of output?
Kernel of the function? — Which inputs map to zero in the range?
![Page 46: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/46.jpg)
SUBSPACES
![Page 47: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/47.jpg)
RANGE
+ + + + =
Must be the linear combination of Columns
![Page 48: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/48.jpg)
KERNEL
=
Must be the orthogonal to all Rows
![Page 49: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/49.jpg)
Rn RmMm⇥n
Column Space of the Matrix = Linear span of all Columns = Range
Row Space of the Matrix = Linear span of Rows = Non-zero output
Null Space of the Matrix = Map to zero = Orthogonal to Row Space
![Page 50: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/50.jpg)
Rn RmMm⇥n
ColSpaceRowSpace
Null Space
![Page 51: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/51.jpg)
LINEAR EQUATIONS
![Page 52: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/52.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
x b
![Page 53: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/53.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
x bSolution always exists.
Solution is unique as Mx = b and My = b implies (x-y) = 0.
![Page 54: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/54.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
![Page 55: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/55.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
xbu
v
Infinitely many solutions.
If Mu = b, then M(u+v) = b, for any v from Null Space.
![Page 56: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/56.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
![Page 57: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/57.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
xb
y
e
If Mx = y, then Mx ~ b + e, with irreducible error e.
Solution does not exist!
![Page 58: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/58.jpg)
xb
y
e
Rn RmMm⇥n
T
![Page 59: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/59.jpg)
xb
y
e
Rn RmMm⇥n
T
MTMx = MTb = MTy provides solution x = (MTM)-1MTy
![Page 60: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/60.jpg)
SOLVABILITY OF LINEAR EQUATION Mx = b
Unique Infinite
Infinite None
![Page 61: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/61.jpg)
SINGULAR VALUES
![Page 62: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/62.jpg)
ACTION OF A MATRIX
M
![Page 63: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/63.jpg)
ACTION OF A MATRIX
M
![Page 64: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/64.jpg)
ACTION OF A MATRIX
M
u1v1
M v1 = d1 u1
![Page 65: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/65.jpg)
ACTION OF A MATRIX
M
u1v1
u2
v2
M v1 = d1 u1
M v2 = d2 u2
![Page 66: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/66.jpg)
SINGULAR VECTORS
M v1 u1v2 v3 v4 v5 u2 u3 u4 u5
d1
d2
d3
d4
d5
x x=
![Page 67: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/67.jpg)
SINGULAR VECTORS
M
v1
u1
v2
v3
v4
v5
u2 u3 u4 u5
d1
d2
d3
d4
d5
xx=
![Page 68: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/68.jpg)
SINGULAR VALUE DECOMPOSITION
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
M = U⌃V T
![Page 69: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/69.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
ColSpaceRowSpace
v1
v2
v3
v4
v5
u1
u2
u3
u4
M = U⌃V T
![Page 70: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/70.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
M
![Page 71: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/71.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT
rotation
M
v1
v2
e1
e2
![Page 72: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/72.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT ⌃
rotation scaling
M
e1
e2
d1e1
d2e2
![Page 73: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/73.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT U⌃
rotation rotationscaling
M
d1e1
d2e2
d1u1
d2u2
![Page 74: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/74.jpg)
M
v1
u1
v2
v3
v4
v5
u2 u3 u4
d1
d2 xx=
VT U⌃
rotation rotationscaling
M
![Page 75: LINEAR ALGEBRA - Indian Statistical Instituteacmsc/WMSFMLT2016/LinearAlgebra.pdf · Understanding Matrices as linear operators 3. Subspaces and Singular Value Decomposition Part II](https://reader031.vdocuments.us/reader031/viewer/2022021900/5b77239d7f8b9ad3338c8133/html5/thumbnails/75.jpg)
VT U⌃
rotation rotationscaling
M
REFERENCESGilbert Strang — Lectures in Linear Algebra — MIT OCW 18.06 (or YouTube)Gilbert Strang — The Fundamental Theorem of Linear Algebra — AMM 1993Lloyd Trefethen and David Bau — Numerical Linear Algebra — SIAM 1997