DIMENSION REDUCTION FOR HYPERSPECTRAL DATA USING RANDOMIZED PCA AND LAPLACIANEIGENMAPSYIRAN LI

APPLIED MATHEMATICS, STATISTICS AND SCIENTIFIC COMPUTING

ADVISOR: DR. WOJTEK CZAJA, DR. JOHN BENEDETTO

DEPARTMENT OF MATHEMATICS

UNIVERSITY OF MARYLAND, COLLEGE PARK

BACKGROUND: HYPERSPECTRAL IMAGING

• Light is described in terms of its wavelength

• A reflectance spectrum shows the reflectance of a material measured across a

range of wavelengths. It helps identify certain materials uniquely

• We measure reflectance at many narrow, closely spaced wavelength bands

• When a spectrometer is used in an imaging sensor, the resulting images record

a reflectance spectrum for each pixel in the images

(Shippert, 2003)

SPECTRUM AND HYPERSPECTRAL IMAGERY

• Left: Reflectance spectra measured by laboratory spectrometers for three

materials: a green bay laurel leaf, the mineral talc, and a silty loam soil.

• Right: The concept of hyperspectral imagery. (Shippert, 2003)

MULTISPECTRAL VS HYPERSPECTRAL

• Multispectral imaging measures reflectance at discrete and somewhat narrow

bands. Multispectral images do not produce the "spectrum" of an object

• Hyperspectral deals with imaging narrow spectral bands over a continuous spectral

range, and produce the spectra of all pixels in the scene.

• So a sensor with only 20 bands can also be hyperspectral when it covers the range

from 500 to 700 nm with 20 bands each 10 nm wide.

(Wikipedia: hyperspectral imaging)

AN EXAMPLE: SALINAS VALLEY, CALIFORNIA

• Left: sample band collected by 224-band sensor. It includes vegetables, bare soils,

and vineyard fields. Right: Groundtruth of Salinas dataset (16 classes)

(IC: Hyperspectral Remote Sensing Scenes)

PROBLEM

• Hyperspectral images are three dimensional (x-coordinate, y-coordinate, b)

• Each pixel has a different spectrum that represents different materials

• Sometimes over 100 bands and with large number of pixels

• Dimension reduction reduces the number of bands of a hyperspectral image

• It maps dimensional data into a lower dimension while preserving the main

features of the original data.

(hyperspectral imaging, Wikipedia)

PROJECT GOAL

• Reduce dimensionality of hyperspectral imaging

• Compare two algorithms to be implemented

METHODS

Existing methods (partial) :

• Principal Component Analysis( PCA)

• Local Linear Embedding

• Neighborhood Preserving Embedding

• Classical multidimensional scaling

• Isomap

• Stochastic Proximity Embedding

My Methods:

• Randomized PCA

• Laplacian Eigenmaps

(Delft University)

COMPARISON BETWEEN TWO ALGORITHMS

Compare two algorithms,

Randomized PCA and Laplacian Eigenmaps, in terms of:

• Implementation

• Running time

• Results

• Difficulties during implementation

ALGORITHM 1: LAPLACIAN EIGENMAPS

• Consider the problem of mapping the weighted graph G to a line so that connected

points stay as close together as possible, let 𝑦 = 𝑦1, 𝑦2, … 𝑦𝑛T be such a map. Our

goal is to minimize

𝑖,𝑗 𝑦𝑖 − 𝑦𝑗2𝑊𝑖𝑗

Since 𝑖,𝑗 𝑦𝑖 − 𝑦𝑗2𝑊𝑖𝑗 = 2yTLy, the problem of finding 𝑎𝑟𝑔𝑚𝑖𝑛 𝑦𝑇𝐿𝑦 given that

𝑦𝑇𝐷𝑦 = 1, 𝑦𝑇𝐷1 = 0 becomes the minimum eigenvalue problem:

𝐿𝑓 = 𝜆𝐷𝑓

(Belkin, Niyogi, 2002)

ALGORITHM 1: THE ALGORITHM

• Step 1: Constructing the Adjacency Graph

• Construct a weighted graph with n nodes (n number of data points), and a set of edges connecting

neighboring points.

• A) 𝜀 neighborhood: connected if

𝑥𝑖 − 𝑥𝑗2< 𝜀

• B) n nearest neighbors

• Step 2: Choosing the weights

• A) Heat Kernel:

𝑊𝑖𝑗 = 𝑒−𝑥𝑖−𝑥𝑗

2

𝑡

• B) Simple Minded: 𝑊𝑖𝑗 = 1 if connected and 𝑊𝑖𝑗 = 0 otherwise

• Step 3: Compute eigenvalues and eigenvectors for the generalized eigenvector

problem:

𝐿𝑓 = 𝜆𝐷𝑓 (1)

Where 𝑊 is the weight matrix defined earlier, 𝐷 is diagonal weight matrix, with

𝐷𝑖𝑖 = 𝑗𝑊𝑗𝑖, and

𝐿 = 𝐷 −𝑊

• Let 𝑓0, 𝑓1, … , 𝑓𝑛−1 be the solutions of equation (1), ordered such that

0 = 𝜆0 ≤ 𝜆1 ≤ … ≤ 𝜆𝑛−1

• Then the first m eigenvectors (excluding 𝑓0) ,

{𝑓1, 𝑓2, … , 𝑓𝑚}

are the desired vectors for embedding in m-dimensional Euclidean space

(Belkin, Niyogi, 2002)

ALGORITHM 2: RANDOMIZED PCA INTRODUCTION

• Canonical construction of the best possible rank-k approximation to a real 𝑚 × 𝑛

matrix 𝐴 uses singular value decomposition (SVD) of 𝐴,

𝐴 = 𝑈Σ𝑉𝑇 ,

Where 𝑈 real unitary 𝑚 ×𝑚 matrix, 𝑉 is real unitary 𝑛 × 𝑛 matrix, and Σ is real

𝑚 × 𝑛 diagonal matrix with nonnegative, non increasing diagonal entries

• Best Approximation of 𝐴:

𝐴 ≈ 𝑈 Σ 𝑉𝑇 ,

Where 𝑈 leftmost 𝑚 × 𝑘 block of 𝑈, Σ 𝑘 × 𝑘 upper left block of Σ, 𝑉 leftmost 𝑛 × 𝑘

block of 𝑉

(Rokhlin, Szlam, Tygert, 2009)

• Best because it minimizes the spectral norm 𝐴 − 𝐵 for a rank-k matrix 𝐵 = 𝑈 Σ 𝑉𝑇. In fact ,

𝐴 − 𝑈 Σ 𝑉𝑇 = 𝜎𝑘+1,

Where 𝜎𝑘+1 is the 𝑘 + 1 𝑡ℎgreatest singular value

• Randomized PCA generates 𝐵 such that

𝐴 − 𝐵 ≤ 𝐶𝑚1

4𝑖+2𝜎𝑘+1

with high probability (1 − 10−15) , where 𝑖 is specified by user, and C depends

on parameters of algorithm

(Rokhlin, Szlam, Tygert, 2009)

ALGORITHM 2: THE ALGORITHM

• Choose 𝑙 > k such that 𝑙 ≤ 𝑚 − 𝑘

• Step 1: Generate a real 𝑙 × 𝑚 matrix 𝐺 whose entries are i.i.d normal Gaussian

random variables, compute

𝑅 = 𝐺 𝐴𝐴𝑇 𝑖𝐴

• Step 2: Using SVD, form a real 𝑛 × 𝑘 matrix 𝑄 whose columns are orthonormal, such

that

𝑄𝑆 − 𝑅𝑇 ≤ 𝜌𝑘+1

for some 𝑘 × 𝑙 matrix 𝑆, where 𝜌𝑘+1 is the 𝑘 + 1 𝑡ℎ greatest singular value of 𝑅

• Step 3: Compute

𝑇 = 𝐴𝑄

• Step 4: Form an SVD of T:

𝑇 = 𝑈Σ𝑊𝑇,

where 𝑈 is a real 𝑚 × 𝑘 matrix whose columns are orthonormal, 𝑊 is a real 𝑘 × 𝑘

matrix whose columns are orthonormal, Σ is a real diagonal 𝑘 × 𝑘 matrix with

nonnegative diagonal entries

• Step 5: Compute

𝑉 = 𝑄𝑊

• In this way, we get 𝑈, Σ, 𝑉 as desired, and 𝐵 = 𝑈Σ𝑉𝑇

(Rokhlin, Szlam, Tygert, 2009)

IMPLEMENTATION

• Hardware: Personal laptop/Computers in the math computer lab

• Software: Matlab

• Database: 12 Band Moderate Dimension Image: June 1966 aircraft scanner

Flightline C1 (Portion of Southern Tippecanoe County, Indiana)

• 220 Band Hyperspectral Image: June 12, 1992 AVIRIS image Indian Pine Test Site 3

(2 x 2 mile portion of Northwest Tippecanoe County, Indiana)

• 220 Band Hyperspectral Image: June 12, 1992 AVIRIS image North-South flight line

(25 x 6 mile portion of Northwest Tippecanoe County, Indiana)

• Hyperspectral data from Norbert Weiner Center

• Data can be large (with 10,000^2 pixels, 200 bands, for example)

VALIDATION METHODS

• Delft University has developed Matlab toolbox for dimension reduction, which

includes many methods, and is publically available

• Use algorithms from DR matlab toolbox to run on the same data and compare results

• For randomized PCA, check error bound:

𝐴 − 𝐵 ≤ 𝐶𝑚1

4𝑖+2𝜎𝑘+1 (Rohklin, 2009)

• Compare with ground truth images for the test cases

TEST PROBLEMS FOR VERIFICATION

• Test on known data set (as provided earlier), and compare results with ground

truth classifications and images

• Test on smaller scales at first, and then move to large data set

EXPECTED RESULTS/CONCLUDING REMARKS

• Laplacian Eigenmaps should be easier to implement, but may take longer to

run because it deals with solving the eigenvalue problem of large matrices

• Randomized PCA will be more difficult to implement, but will give desired

results under unfavorable conditions with reasonable speed, and it should

perform better than Laplacian eigenmaps when dealing with very large

matrices

TIMELINE/MILESTONES

• October 17th: Project proposal

• Now to November, 2014: Implement and test laplacian eigenmaps, prepare

for implementation of randomized PCA

• December, 2014: Midyear report and presentation

• January to March: Implement and test randomized PCA, compare two

methods in various situations

• April to May: Final presentation and Final report

DELIVERABLES

• Presentation of data sets with reduced dimensions of both algorithms

• Comparison charts in terms of running time and accuracy of two different

methods

• Comparison charts with other methods that are available from the DR matlab

toolbox

• Data sets, Matlab codes, presentations, proposals, mid-year report, final

report

BIBLIOGRAPHY

• Shippert, Peg. Introduction to Hyperspectral Image Analysis. Online Journal of Space

Communication, issue No. 3: Remote Sensing of Earth via Satellite. Winter 2003.

http://spacejournal.ohio.edu/pdf/shippert.pdf

• Hyperspectral Imaging. From Wikipedia. Oct. 6th, 2014.

http://en.wikipedia.org/wiki/Hyperspectral_imaging

• Belkin, Mikhail; Niyogi, Partha. Laplacian Eigenmaps for Dimensionality Reduction and

Data Representation. Neural Computation, vol 15. Dec. 8th, 2002. Web.

http://web.cse.ohio-state.edu/~mbelkin/papers/LEM_NC_03.pdf

• Rokhlin, Vladimir; Szlam, Arthur; Tygert, Mark. A Randomized Algorithm for

Principal Component Analysis. SIAM Journal on Matrix Analysis and

Applications Volume 31 Issue 3. August 2009. Web.

ftp://ftp.math.ucla.edu/pub/camreport/cam08-60.pdf

• Matlab Toolbox for Dimension Reduction. Delft University. Web. Oct. 6th,

2014.

http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_Dimensionality_Redu

ction.html

• IC: Hyperspectral Remote Sensing Scenes. Web. Oct. 6th, 2014.

http://www.ehu.es/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_

Scenes

• Hyperspectral Images. Web. Oct. 6th, 2014.

https://engineering.purdue.edu/~biehl/MultiSpec/hyperspectral.html