Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_1.htm[6/14/2012 3:51:38 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

The Lecture Contains:

Singular Value Decomposition (SVD)

SVD of real symmetric matrix

Transformation using SVD

ExampleExample: compact form

Dimensionality reduction using SVD

Example of dimensionality reduction (k = 1)

Compact way of dimensionality reduction (k = 1)

How many dimensions to retain?

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_2.htm[6/14/2012 3:51:39 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD)Factorization of a matrix

If A is of size , then is , is and is matrix

Columns of are eigenvectors of

Left singular vectors

(orthonormal)

Columns of are eigenvectors of

Right singular vectors

(orthonormal)

are the singular values

is diagonal

Singular values are positive squareroots of eigenvalues of or

(assuming singular values)

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_3.htm[6/14/2012 3:51:39 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

SVD of real symmetric matrix

A is real symmetric of size

since

is of size and contains eigenvectors of

This is called spectral decomposition of

contains singular values

Eigenvectors of = eigenvectors of

Eigenvalues of = squareroot of eigenvalues of

Eigenvalues of = singular values of

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_4.htm[6/14/2012 3:51:39 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVDTransformed data

is called SVD transform matrix

Essentially, is just a rotation of

Dimensionality of is

different basis vectors than the original space

Columns of give the basis vectors in rotated space

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_5.htm[6/14/2012 3:51:39 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVDTransformed data

is called SVD transform matrix

Essentially, is just a rotation of

Dimensionality of is

different basis vectors than the original space

Columns of give the basis vectors in rotated space

shows how each dimension can be represented as a linear combination of otherdimensions

Columns are input basis vectors

shows how each object can be represented as a linear combination of other objects

Columns are output basis vectors

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_6.htm[6/14/2012 3:51:39 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVDTransformed data

is called SVD transform matrix

Essentially, is just a rotation of

Dimensionality of is

different basis vectors than the original space

Columns of give the basis vectors in rotated space

shows how each dimension can be represented as a linear combination of otherdimensions

Columns are input basis vectors

shows how each object can be represented as a linear combination of other objects

Columns are output basis vectors

Lengths of vectors are preserved

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_7.htm[6/14/2012 3:51:40 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Example

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_8.htm[6/14/2012 3:51:40 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Example: compact form

If is of size , then is , is and is matrix

Works because there at most non-zero singular values in

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_9.htm[6/14/2012 3:51:40 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Dimensionality reduction using SVD

Use only dimensions

Retain first columns for and and first values for

First columns of give the basis vectors in reduced space

Best rank approximation in terms of sum squared error

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_10.htm[6/14/2012 3:51:40 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Example of dimensionality reduction ( = 1)

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_11.htm[6/14/2012 3:51:40 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

Compact way of dimensionality reduction ( = 1)

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_12.htm[6/14/2012 3:51:41 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_13.htm[6/14/2012 3:51:41 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?There is no easy answer

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_14.htm[6/14/2012 3:51:41 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?There is no easy answer

Concept of energy of a dataset

Total energy is sum of squares of singular values (aka spread or variance)

Retain dimensions such that of the energy is retained

Generally, is between to

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_15.htm[6/14/2012 3:51:41 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?There is no easy answer

Concept of energy of a dataset

Total energy is sum of squares of singular values (aka spread or variance)

Retain dimensions such that of the energy is retained

Generally, is between to

In the above example, = 1 retains of the energy

Objectives_template

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_16.htm[6/14/2012 3:51:41 PM]

Module 6: Dimensionality Reduction Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?There is no easy answer

Concept of energy of a dataset

Total energy is sum of squares of singular values (aka spread or variance)

Retain dimensions such that of the energy is retained

Generally, is between to

In the above example, = 1 retains of the energy

Running time: for of size and rank

29_1Local DiskObjectives_template

29_2Local DiskObjectives_template

29_3Local DiskObjectives_template

29_4Local DiskObjectives_template

29_5Local DiskObjectives_template

29_6Local DiskObjectives_template

29_7Local DiskObjectives_template

29_8Local DiskObjectives_template

29_9Local DiskObjectives_template

29_10Local DiskObjectives_template

29_11Local DiskObjectives_template

29_12Local DiskObjectives_template

29_13Local DiskObjectives_template

29_14Local DiskObjectives_template

29_15Local DiskObjectives_template

29_16Local DiskObjectives_template