Richard Baraniuk

Rice Universitydsp.rice.edu/cs

Lecture 3:CompressiveClassification

Compressive Sampling

Signal Sparsity

widebandsignalsamples

largeGabor (TF)coefficients

Fourier matrix

Compressive Sampling

• Random measurements

measurements signal

sparsein basis

Compressive Sampling

• Universality

Why Does It Work? (1)• Random projection not full rank, but stably embeds

– sparse/compressible signal models (CS) – point clouds (JL)

into lower dimensional space with high probability• Stable embedding: preserves structure

– distances between points, angles between vectors, …

provided M is large enough: Compressive Sampling

K-dim planes

K-sparsemodel

Why Does It Work? (2)• Random projection not full rank, but stably embeds

– sparse/compressible signal models (CS) – point clouds (JL)

into lower dimensional space with high probability• Stable embedding: preserves structure

– distances between points, angles between vectors, …

provided M is large enough: Johnson-Lindenstrauss

Q points

Information Scalability

• In many CS applications, full signal recovery may not be required

• If we can reconstruct a signal from compressive measurements, then we should be able to perform other kinds of statistical signal processing:

– detection– classification– estimation …

CompressiveDetection/Classification

Classification of Signals in Noise

• Observe one of P known signals in noise

• Probability density function of the noiseex: zero mean, white Gaussian noise

(AWGN)

• Probability density function of signal

>>> mean shifted to

Multiclass Likelihood Ratio Test (LRT)

• Observe one of P known signals in noise

• Classify according to:

• AWGN: nearest-neighbor classification

• Sufficient statistic:

Compressive LRT

• Compressive observations:

[Waagen et al 05, Davenport et al 06, Haupt et al 06]

by the JL Lemmathese distancesare preserved

Compressive LRT

• Compressive observations:

• If are normalized,

these anglesare preserved

Performance of Compressive LRT

• ROC curve for Neyman-Pearson detector (2 classes):

• From JL lemma, for random orthoprojector

• Thus

• Penalty for compressive measurements:

false alarmprobability

Performance of Compressive LRT

fewer measurements

better SNR

Summary

• If CS system is a random orthoprojector, then detection/classification problems in the signal space map into analogous problems in the measurement space

• SNR penalty for compressive measurements

• Note: Signal sparsity unexploited!

Matched Filtering

Matched Filter

• In many applications, signals are transformedwith an unknown parameter; ex: translation

• Elegant solution: matched filterCompute

for all

• Simultaneously: estimates parameterclassifies signal

convolution of measurementwith template reversed in time

Compressive Matched Filter

• Challenge: Extend matched filter concept to compressive measurements

• GLRT:

• GLRT approach extends to any case where each class can be parameterized with K parameters

• If mapping from parameters to signal is well-behaved, then each class forms a manifold in

Signal Manifolds

What is a Manifold?

“Manifolds are a bit like pornography: hard to define, but you know one when you see one.”

– S. Weinberger [Lee]

• Locally Euclidean topological space

• Roughly speaking:– a collection of mappings of open sets of RK glued together (“coordinate charts”)

– can be an abstract space, not a subset of Euclidean space• e.g., SO3, Grassmannian

• Typically for signal processing: – nonlinear K-dimensional “surface” in signal space RN

Examples

• Circle in RN

– parameter: angle

• Chirp in RN

– parameters: start frequencyend frequency

• Image appearance manifold– parameters:

position of object, camera, lighting, etc.

Object Rotation Manifold

K=1

each imageis a pointin RN

Up/Down Left/Right Manifold

[Tenenbaum, de Silva, Langford]

K=2

ISOMAP HLLELaplacian

Eigenmaps

R4096

Manifold Learning from Training Data

• Translating diskparameters: left/right, up/down shift (K=2)

• Generate training data by sampling from the manifold

• “Learn” the structure of the manifold

Manifold Classification

Manifold Classification

• Now suppose data is drawn from one of P possible manifolds:

• AWGN: nearest manifold classification

M1

M2M3

Compressive Manifold Classification

• Compressive observations:

• Good news: structure of smoothmanifolds is preserved by randomprojection provided

– distances, geodesic distances, angles, …

[Wakin et al, 06, Haupt et al 07]

Aside:Random Projectionsof Smooth Manifolds

Stable Manifold EmbeddingTheorem:

Let F ⊂ RN be a compact K-dimensional manifold with

– condition number 1/τ (curvature, self-avoiding)

– volume V

Let Φ be a random MxN orthoprojector with

[Wakin et al 06, Haupt et al 07]

Then with probability at least 1-ρ, the followingstatement holds: For every pair x,y ∈ F

Stable Manifold Embedding

• Theorem tells us that random projectionspreserve smooth manifold

– dimensionality– ambient distances– geodesic distances– local angles– topology– local neighborhoods– Volume

• Also there exists extension to some kinds of non-smooth manifolds

Manifold Learning from Compressive Measurements

ISOMAP HLLELaplacian

Eigenmaps

R4096

RM

M=15 M=15M=20

Let Φ be a random MxN orthoprojector with

Multiple Manifold Embedding

Corollary:Let M1, … ,MP ⊂ RN be compact K-dimensional manifolds with

– condition number 1/τ (curvature, self-avoiding)

– volume V

– min dist(Mj,Mk) > τ (can be relaxed)

Then with probability at least 1-ρ, the followingstatement holds: For every pair x,y ∈ U Mj

Compressive Manifold

Classification

Compressive Manifold Classification

• Compressive observations:

• Good news: structure of smoothmanifolds is preserved by randomprojection provided

– distances, geodesic distances, angles, …

[Wakin et al, 06, Haupt et al 07]

Smashed Filter• Compressive manifold classification with GLRT

– nearest-manifold classifier based on manifolds

M1

M2M3

Φ M1

Φ M2Φ M3

[Davenport et al 06, Healy and Rohode 07]

Smashed Filter – Experiments

• 3 image classes: tank, school bus, SUV

• N = 65,536 pixels

• Imaged using single-pixel CS camera with– unknown shift– unknown rotation

Smashed Filter – Unknown Position

• Object shifted at random (K=2 manifold)• Noise added to measurements• Goal: identify most likely position for each image class

identify most likely class using nearest-neighbor test

number of measurements Mnumber of measurements M

avg.

shift

estim

ate

erro

r

clas

sifica

tion r

ate

(%)

more noise

more noise

Smashed Filter – Unknown Rotation

• Object rotated each 10o

• Goals: identify most likely rotation for each image classidentify most likely class using nearest-neighbor test

• Perfect classification withas few as 6 measurements

• Good estimates of rotation with under 10 measurements

number of measurements M

avg.

rot.

est

. er

ror

Summary

• Compressive measurements areinformation scalable

reconstruction > estimation > classification > detection

• Random projections preserve structure of smooth manifolds (analogous to sparse signals)

• Smashed filter: dimension-reduced GLRT for parametrically transformed signals– exploits compressive measurements and manifold structure– broadly applicable: targets do not have to have sparse

representation in any basis– effective for detection/classification

Open Issues

• Compressive classification does not exploit sparse signal structure to improve performance

• Non-smooth manifolds and local minima in GLRT– one approach: multiscale random projections

• Experiments with real data

Some References

• R. G. Baraniuk, M. Davenport, R. DeVore, M. Wakin, “A simple proof of the restricted isometry property for random matrices,” to appear in Constructive Approximation, 2007.

• R. G. Baraniuk and M. Wakin, “Random projections of smooth manifolds,”2006; see also ICASSP 2006.

• M. Davenport, M. Duarte, M. Wakin, J. Laska, D. Takhar, K. Kelly, R. G. Baraniuk, “The smashed filter for compressive classification and target recognition,” Proc. of Computational Imaging V at SPIE Electronic Imaging, San Jose, California, January 2007.

• M. Wakin, D. Donoho, H. Choi, R. G. Baraniuk. “The multiscale structure of non-differentiable image manifolds,” Proc. Wavelets XI at SPIE Optics and Photonics, 2005.

• J. Haupt, R. Castro, R. Nowak, G. Fudge, A. Yeh, “Compressive sampling for signal classification,” Proc. Asilomar Conference on Signals, Systems, and Computers, 2006.

• D. Healy, G. Rohode, “Fast global image registration using random projections, 2007.

for more, see dsp.rice.edu/cs