linear algebra and calculus!computer-learning.github.io/class/slides/linear_algebra.pdf · linear...
TRANSCRIPT
![Page 1: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/1.jpg)
Linear Algebra and Calculus!
Jeremy Irvin and Daniel Spokoyny
![Page 2: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/2.jpg)
Eigenvalues and Eigenvectors● Given a square matrix , we say
is an eigenvalue of A and is the corresponding eigenvector if
● Intuitively, this means multiplying A by x results in a new vector in the same direction as x but scaled by .
![Page 3: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/3.jpg)
Eigenvalues and Eigenvectors● Note that if x is an eigenvector, then cx is
an eigenvector for any complex c.
● We can rewrite the equation above as
● has a non-zero solution iff has a non-empty nullspace,
which only happens if is singular, ie,
![Page 4: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/4.jpg)
E-values and E-vectors Properties● The trace of A is equal to the sum of its
eigenvalues,
● The determinant of A is equal to the product of its eigenvalues
● The rank of A is equal to the number of non-zero eigenvalues of A.
● The eigenvalues of a diagonal matrix are just the diagonal entries.
![Page 5: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/5.jpg)
Diagonalization
● We can write all the eigenvector equations simultaneously as
.with the eigenvectors of A and a diagonal matrix whose entries are the eigenvalues A, ie,
![Page 6: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/6.jpg)
Diagonalization
● If the eigenvectors of A are linearly independent, then X will be invertible, so
We say that A is diagonalizable.
![Page 7: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/7.jpg)
Quadratic Forms
● Given any symmetric matrix and vector , the scalar value is called a quadratic form.
● Explicitly, we have
![Page 8: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/8.jpg)
Definite Matrices● A is positive definite if for all non-zero
vectors > 0.
● A is negative definite if for all non-zero vectors
< 0. ● Positive and negative definite matrices are
full rank and thus invertible.
● For any matrix , is positive semidefinite.
![Page 9: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/9.jpg)
E-values and E-vectors of Symmetric Matrices● Let be any symmetric matrix:
○ All eigenvalues of A are real.
○ The non-collinear eigenvectors of A are orthonormal.
○ Thus we can decompose A:
where U is an orthogonal matrix.
![Page 10: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/10.jpg)
E-values and E-vectors of Symmetric Matrices
● We can use this to show that definiteness only depends on sign of eigenvalues:
where .● For any quadratic form subject to
, its maximum value is the maximum eigenvalue of A, and its minimum value is the minimum eigenvalue of A.
![Page 11: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/11.jpg)
Singular Value Decomposition (SVD)
● Goal: Given any matrix , find orthogonal matrices U and V such that
● If A diagonalizable,
● If A positive semidefinite,
![Page 12: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/12.jpg)
Singular Value Decomposition (SVD)
● See blackboard pictures.
● We know we can find an orthonormal basis for the rowspace of A.
● Can we find one that is mapped into an orthonormal basis for the column space of A?
![Page 13: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/13.jpg)
Singular Value Decomposition (SVD)
● Goal: Find an orthonormal basis for the row space of A such that
where is an orthonormal basis for the column space of A.
![Page 14: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/14.jpg)
Singular Value Decomposition (SVD)
● In matrix notation,
where orthonormal basis for the null space of A, and an orthonormal basis for the null space of AT .● Or
![Page 15: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/15.jpg)
Singular Value Decomposition (SVD)
● But because and are orthogonal complements, V is orthogonal.
● Similarly, U is orthogonal.
● Therefore
![Page 16: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/16.jpg)
Singular Value Decomposition (SVD)
● How do we find U and V?● Trick:
● Since AT A is positive semidefinite, V is the orthogonal matrix of eigenvectors of AT A, and its eigenvalues are the squares of the diagonal entries of Σ.
![Page 17: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/17.jpg)
Singular Value Decomposition (SVD)
● Similarly,
● So to find U, simply find the eigenvectors of AAT .
![Page 18: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/18.jpg)
SVD Example
Find orthonormal in the row space of A ( ) and orthonormal in the column space of A ( ), and such that
![Page 19: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/19.jpg)
SVD Example
Therefore the eigenvectors of AT A are
with eigenvalues 32 and 18 respectively.
![Page 20: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/20.jpg)
SVD Example
Therefore the eigenvectors of AT A are
again with eigenvalues 32 and 18 respectively (is this surprising?)
![Page 21: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/21.jpg)
SVD Example
Therefore:
![Page 22: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/22.jpg)
SVD Example 2
See blackboard for geometric intuition.
![Page 23: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/23.jpg)
SVD Example 2
● Therefore the eigenvalues of AT A are 0 and 125. Hence:
![Page 24: Linear Algebra and Calculus!computer-learning.github.io/class/Slides/linear_algebra.pdf · Linear Algebra and Calculus! Jeremy Irvin and Daniel Spokoyny. Eigenvalues and Eigenvectors](https://reader035.vdocuments.us/reader035/viewer/2022062414/5f087a967e708231d4223662/html5/thumbnails/24.jpg)
So who cares?● Why is SVD even useful?
● SVD can be used for dimensionality reduction - given high dimensional data, one can use SVD to represent the data using less dimensions, while still capturing the most significant (largest eigenvalues) features.
● We will see important applications later.