cs 2750: machine learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · announcement • ta...
TRANSCRIPT
![Page 1: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/1.jpg)
CS 2750: Machine Learning
Linear Algebra and Matlab
Prof. Adriana KovashkaUniversity of Pittsburgh
January 10, 2017
![Page 2: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/2.jpg)
Announcement
• TA won’t be back until the last week of January
• Skype office hours:
– Tuesday/Wednesday 9:30am to 11:30am
– Username: vicl720
• Please do Doodle:
– http://doodle.com/poll/gtaxphum9mutd74x
![Page 3: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/3.jpg)
Linear Algebra Primer
Professor Fei-Fei Li
Stanford
31-Jan-17 3
Another, very in-depth linear algebra review from CS229 is available here:http://cs229.stanford.edu/section/cs229-linalg.pdfAnd a video discussion of linear algebra from EE263 is here (lectures 3 and 4):http://see.stanford.edu/see/lecturelist.aspx?coll=17005383-19c6-49ed-9497-2ba8bfcfe5f6
![Page 4: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/4.jpg)
Vectors and Matrices
• Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, word counts, movie ratings, pixel brightnesses, etc. We’ll define some common uses and standard operations on them.
Fei-Fei Li 3
![Page 5: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/5.jpg)
Vector
• A column vector where
• A row vector where
denotes the transpose operation
Fei-Fei Li 5
![Page 6: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/6.jpg)
Vector
• You’ll want to keep track of the orientation of your vectors when programming in MATLAB.
• You can transpose a vector V in MATLAB by writing V’.
Fei-Fei Li 6
![Page 7: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/7.jpg)
Vectors have two main uses
• Vectors can represent an offset in 2D or 3D space
• Points are just vectors from the origin
Fei-Fei Li 7
• Data can also be treated as a vector
• Such vectors don’t have a geometric interpretation, but calculations like “distance” still have value
![Page 8: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/8.jpg)
Matrix
• A matrix is an array of numbers with size 𝑚 ↓ by 𝑛 →, i.e. m rows and n columns.
• If , we say that is square.
Fei-Fei Li 8
![Page 9: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/9.jpg)
Matrix Operations• Addition
– Can only add a matrix with matching dimensions, or a scalar.
• Scaling
31-Jan-17 9
![Page 10: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/10.jpg)
Matrix Operations
• Inner product (dot product) of vectors
– Multiply corresponding entries of two vectors and add up the result
– x·y is also |x||y|Cos( the angle between x and y )
Fei-Fei Li 10
![Page 11: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/11.jpg)
Matrix Operations
• Inner product (dot product) of vectors
– If B is a unit vector, then A·B gives the length of A which lies in the direction of B (projection)
Adapted from Fei-Fei Li 11
(if B is unit-length hence norm is 1)
![Page 12: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/12.jpg)
• L1 norm
• L2 norm
• Lp norm (for real numbers p ≥ 1)
Norms
![Page 13: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/13.jpg)
Matrix Operations• Multiplication
• The product AB is:
• Each entry in the result is (that row of A) dot product with (that column of B)
Fei-Fei Li 13
![Page 14: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/14.jpg)
Matrix Operations• Multiplication example:
Fei-Fei Li 14
– Each entry of the matrix product is made by taking the dot product of the corresponding row in the left matrix, with the corresponding column in the right one.
![Page 15: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/15.jpg)
Matrix Operations
• Transpose – flip matrix, so row 1 becomes column 1
• A useful identity:
Fei-Fei Li 15
![Page 16: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/16.jpg)
Special Matrices
• Identity matrix I– Square matrix, 1’s along
diagonal, 0’s elsewhere– I ∙ [another matrix] = [that
matrix]
• Diagonal matrix– Square matrix with numbers
along diagonal, 0’s elsewhere– A diagonal ∙ [another matrix]
scales the rows of that matrix
Fei-Fei Li 16
![Page 17: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/17.jpg)
Special Matrices
• Symmetric matrix
Fei-Fei Li 17
![Page 18: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/18.jpg)
• Given a matrix A, its inverse A-1 is a matrix such that AA-1 = A-1A = I
• E.g.
• Inverse does not always exist. If A-1 exists, A is invertible or non-singular. Otherwise, it’s singular.
Fei-Fei Li 18
Inverse
![Page 19: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/19.jpg)
• MATLAB example:
Fei-Fei Li 19
Matrix Operations
>> x = A\B
x =
1.0000
-0.5000
![Page 20: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/20.jpg)
• Say you have the matrix equation AX=B, where A and B are known, and you want to solve for X
• You could use MATLAB to calculate the inverse and premultiply by it: A-1AX=A-1B → X=A-1B
• MATLAB command would be inv(A)*B• But calculating the inverse for large matrices often brings problems
with computer floating-point resolution, or your matrix might not even have an inverse. Fortunately, there are workarounds.
• Instead of taking an inverse, directly ask MATLAB to solve for X in AX=B, by typing A\B
• MATLAB will try several appropriate numerical methods (including the pseudoinverse if the inverse doesn’t exist)
• MATLAB will return the value of X which solves the equation– If there is no exact solution, it will return the closest one– If there are many solutions, it will return the smallest one
Fei-Fei Li 20
Pseudo-inverse
![Page 21: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/21.jpg)
Matrix Operation Properties
• Matrix addition is commutative and associative – A + B = B + A
– A + (B + C) = (A + B) + C
• Matrix multiplication is associative and distributive but not commutative – A(B*C) = (A*B)C
– A(B + C) = A*B + A*C
– A*B != B*A
![Page 22: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/22.jpg)
Linear independence
• Suppose we have a set of vectors v1, …, vn
• If we can express v1 as a linear combination of the other vectors v2…vn, then v1 is linearly dependent on the other vectors. – The direction v1 can be expressed as a combination of
the directions v2…vn. (E.g. v1 = .7 v2 -.7 v4)
• If no vector is linearly dependent on the rest of the set, the set is linearly independent.– Common case: a set of vectors v1, …, vn is always
linearly independent if each vector is perpendicular to every other vector (and non-zero)
Fei-Fei Li 22
![Page 23: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/23.jpg)
Linear independence
Not linearly independent
Fei-Fei Li 23
Linearly independent set
![Page 24: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/24.jpg)
Matrix rank
• Column/row rank
• Column rank always equals row rank
• Matrix rank
• If a matrix is not full rank, inverse doesn’t exist
– Inverse also doesn’t exist for non-square matrices
Fei-Fei Li 24
![Page 25: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/25.jpg)
Singular Value Decomposition (SVD)• There are several computer algorithms that
can “factor” a matrix, representing it as the product of some other matrices
• The most useful of these is the Singular Value Decomposition
• Represents any matrix A as a product of three matrices: UΣVT
• MATLAB command: [U,S,V] = svd(A);
Fei-Fei Li 25
![Page 26: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/26.jpg)
Singular Value Decomposition (SVD)
UΣVT = A• Where U and V are rotation matrices, and Σ is
a scaling matrix. For example:
Fei-Fei Li 26
![Page 27: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/27.jpg)
Singular Value Decomposition (SVD)
• In general, if A is m x n, then U will be m x m, Σwill be m x n, and VT will be n x n.
Fei-Fei Li 27
![Page 28: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/28.jpg)
Singular Value Decomposition (SVD)• U and V are always rotation matrices.
– Geometric rotation may not be an applicable concept, depending on the matrix. So we call them “unitary” matrices – each column is a unit vector.
• Σ is a diagonal matrix– The number of nonzero entries = rank of A
– The algorithm always sorts the entries high to low
Fei-Fei Li 28
![Page 29: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/29.jpg)
Singular Value Decomposition (SVD)
M = UΣVT
Illustration from Wikipedia
![Page 30: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/30.jpg)
SVD Applications• We’ve discussed SVD in terms of geometric
transformation matrices
• But SVD of a data matrix can also be very useful
• To understand this, we’ll look at a less geometric interpretation of what SVD is doing
Fei-Fei Li 30
![Page 31: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/31.jpg)
SVD Applications
• Look at how the multiplication works out, left to right:
• Column 1 of U gets scaled by the first value from Σ.
• The resulting vector gets scaled by row 1 of VT to produce a contribution to the columns of A
Fei-Fei Li 31
![Page 32: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/32.jpg)
SVD Applications
• Each product of (column i of U)∙(value i from Σ)∙(row i of VT) produces a component of the final A.
Fei-Fei Li 32
+
=
![Page 33: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/33.jpg)
SVD Applications
• We’re building A as a linear combination of the columns of U
• Using all columns of U, we’ll rebuild the original matrix perfectly
• But, in real-world data, often we can just use the first few columns of U and we’ll get something close (e.g. the first Apartial, above)
Fei-Fei Li 33
![Page 34: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/34.jpg)
SVD Applications
• We can call those first few columns of U the Principal Components of the data
• They show the major patterns that can be added to produce the columns of the original matrix
• The rows of VT show how the principal componentsare mixed to produce the columns of the matrix
Fei-Fei Li 34
![Page 35: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/35.jpg)
SVD Applications
We can look at Σ to see that the first column has a large effect
Fei-Fei Li 35
while the second column has a much smaller effect in this example
![Page 36: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/36.jpg)
Principal Component Analysis
• Remember, columns of U are the Principal Componentsof the data: the major patterns that can be added to produce the columns of the original matrix
• One use of this is to construct a matrix where each column is a separate data sample
• Run SVD on that matrix, and look at the first few columns of U to see patterns that are common among the columns
• This is called Principal Component Analysis (or PCA) of the data samples
Fei-Fei Li 36
![Page 37: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/37.jpg)
Principal Component Analysis
• Often, raw data samples have a lot of redundancy and patterns
• PCA can allow you to represent data samples as weights on the principal components, rather than using the original raw form of the data
• By representing each sample as just those weights, you can represent just the “meat” of what’s different between samples
• This minimal representation makes machine learning and other algorithms much more efficient
Fei-Fei Li 37
![Page 38: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/38.jpg)
• Images of faces: represent each as the concatenation of its pixels (column vectors)
• Stack together horizontally to get matrix A
• [U, S, V] = svd(A);
• First four columns of U:
• Can represent each face as a linear combination of the first few columns of U
Image from Alexander Ihler
Example: Eigenfaces
![Page 39: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/39.jpg)
Addendum: How is SVD computed?
• For this class: tell MATLAB to do it. Use the result.
• But, if you’re interested, one computer algorithm to do it makes use of Eigenvectors
– The following material is presented to make SVD less of a “magical black box.” But you will do fine in this class if you treat SVD as a magical black box, as long as you remember its properties from the previous slides.
Fei-Fei Li 39
![Page 40: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/40.jpg)
Eigenvector definition
• Suppose we have a square matrix A. We can solve for vector x and scalar λ such that Ax= λx
• In other words, find vectors where, if we transform them with A, the only effect is to scale them with no change in direction
• These vectors are called eigenvectors (German for “self vector” of the matrix), and the scaling factors λ are called eigenvalues
• An m x m matrix will have ≤ m eigenvectors where λ is nonzero
Fei-Fei Li 40
![Page 41: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/41.jpg)
Finding eigenvectors
• Computers can find an x such that Ax= λx using this iterative algorithm:
– x=random unit vector– while(x hasn’t converged)
• x=Ax• normalize x
• x will quickly converge to an eigenvector• Some simple modifications will let this algorithm
find all eigenvectors
Fei-Fei Li 41
![Page 42: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/42.jpg)
Finding SVD
• Eigenvectors are for square matrices, but SVD is for all matrices
• To do svd(A), computers can do this:
– Take eigenvectors of AAT (matrix is always square).
• These eigenvectors are the columns of U.
• Square root of eigenvalues are the singular values (the entries of Σ).
– Take eigenvectors of ATA (matrix is always square).
• These eigenvectors are columns of V (or rows of VT)
Fei-Fei Li 42
![Page 43: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/43.jpg)
Finding SVD
• Moral of the story: SVD is fast, even for large matrices
• It’s useful for a lot of stuff• There are also other algorithms to compute SVD
or part of the SVD– MATLAB’s svd() command has options to efficiently
compute only what you need, if performance becomes an issue
Fei-Fei Li 43
A detailed geometric explanation of SVD is here:http://www.ams.org/samplings/feature-column/fcarc-svd
![Page 44: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/44.jpg)
Matlab Tutorialhttp://www.cs.pitt.edu/~kovashka/cs2750_sp17/tutorial.m
http://www.cs.pitt.edu/~kovashka/cs2750_sp17/myfunction.mhttp://www.cs.pitt.edu/~kovashka/cs2750_sp17/myotherfunction.m
Please cover whatever we don’t finish at home.
![Page 45: CS 2750: Machine Learningkovashka/cs2750_sp17/ml_02... · 2017-01-31 · Announcement • TA won’t be back until the last week of January ... • Multiplication example: Fei-Fei](https://reader034.vdocuments.us/reader034/viewer/2022042712/5f917fdb4316f413aa3c6e05/html5/thumbnails/45.jpg)
Tutorials and Exercises
• https://people.cs.pitt.edu/~milos/courses/cs2750/Tutorial/
• http://www.math.udel.edu/~braun/M349/Matlab_probs2.pdf
• http://www.facstaff.bucknell.edu/maneval/help211/basicexercises.html
– Do Problems 1-8, 12
– Most also have solutions
– Ask the TA if you have any problems