compressive sensing a new approach to image acquisition and processing richard baraniuk mona sheikh...

CompressiveSensingA New Approach to Image Acquisition and Processing

Richard BaraniukMona Sheikh

Rice University

The Digital Universe

• Size: 281 billion gigabytes generated in 2007digital bits > stars in the universegrowing by a factor of 10 every 5 years > Avogadro’s number (6.02x1023) in 15 years

• Growth fueled by multimedia data audio, images, video, surveillance cameras, sensor nets, …(2B fotos on Flickr, 4.2M security cameras in UK, …)

• In 2007 digital data generated > total storageby 2011, ½ of digital universe will have no home

[Source: IDC Whitepaper “The Diverse and Exploding Digital Universe” March 2008]

Act I What’s wrong with today’s multimedia sensor systems?why go to all the work to acquire massiveamounts of multimedia data only to throw much/most of it away?

Act II One way out: dimensionality reduction (compressive sensing)enables the design of radically new sensors and systems

Act III Compressive sensing in actionnew cameras, imagers, ADCs, …

Postlude Open-access education

Sense by Sampling

sample

Sense by Sampling

sample too much data!

Sense then Compress

compress

decompress

sample

JPEGJPEG2000

…

Sparsity

pixels largewaveletcoefficients

(blue = 0)

Sparsity

pixels largewaveletcoefficients

(blue = 0)

widebandsignalsamples

largeGabor (TF)coefficients

time

frequency

What’s Wrong with this Picture?

• Why go to all the work to acquire N samples only to discard all but K pieces of data?

compress

decompress

sample

What’s Wrong with this Picture?linear processinglinear signal model(bandlimited subspace)

compress

decompress

sample

nonlinear processingnonlinear signal model(union of subspaces)

Compressive Sensing

• Directly acquire “compressed” data via dimensionality reduction

• Replace samples by more general “measurements”

compressive sensing

recover

Compressive SensingTheory

A Geometrical Perspective

• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain

Sampling

sparsesignal

nonzeroentries

• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain

• Sampling

sparsesignal

nonzeroentries

measurements

Sampling

Compressive Sensing

• When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction

measurements sparsesignal

nonzeroentries

How Can It Work?

• Projection not full rank…

… and so loses information in general

• Ex: Infinitely many ’s map to the same(null space)

How Can It Work?

• Projection not full rank…

… and so loses information in general

• But we are only interested in sparse vectors

columns

How Can It Work?

• Projection not full rank…

… and so loses information in general

• But we are only interested in sparse vectors

• is effectively MxK

columns

How Can It Work?

• Projection not full rank…

… and so loses information in general

• But we are only interested in sparse vectors

• Design so that each of its MxK submatrices are full rank (ideally close to orthobasis)– Restricted Isometry Property (RIP)

columns

Geometrical Interpretation of RIP

sparsesignal

nonzeroentries

• An information preserving projection preserves the geometry of the set of sparse signals

Geometrical Interpretation of RIP

sparsesignal

nonzeroentries

• An information preserving projection preserves the geometry of the set of sparse signals

• Model: union of K-dimensional subspaces aligned w/ coordinate axes (highly nonlinear!)

RIP = Stable Embedding• An information preserving projection preserves

the geometry of the set of sparse signals

• RIP ensures that

K-dim subspaces

RIP = Stable Embedding• An information preserving projection preserves

the geometry of the set of sparse signals

• RIP ensures that

How Can It Work?

• Projection not full rank…

… and so loses information in general

• Design so that each of its MxK submatrices are full rank (RIP)

• Unfortunately, a combinatorial, NP-complete design problem

columns

Insight from the 70’s [Kashin, Gluskin]

• Draw at random– iid Gaussian– iid Bernoulli

…

• Then has the RIP with high probability provided

columns

Insight from the 70’s [Kashin, Gluskin]

• Draw at random– iid Gaussian– iid Bernoulli

…

• Then has the RIP with high probability provided

columns

Randomized Sensing

• Measurements = random linear combinations of the entries of

• No information loss for sparse vectors whp

measurements sparsesignal

nonzeroentries

CS Signal Recovery

• Goal: Recover signal from measurements

• Problem: Randomprojection not full rank(ill-posed inverse problem)

• Solution: Exploit the sparse/compressiblegeometry of acquired signal

CS Signal Recovery• Random projection

not full rank

• Recovery problem:givenfind

• Null space

• Search in null space for the “best” according to some criterion– ex: least squares (N-M)-dim hyperplane

at random angle

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Closed-form solution:

Signal Recovery

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Closed-form solution:

• Wrong answer!

Signal Recovery

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Closed-form solution:

• Wrong answer!

Signal Recovery

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

Signal Recovery

“find sparsest vectorin translated nullspace”

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Correct!

• But NP-Complete alg

Signal Recovery

“find sparsest vectorin translated nullspace”

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Convexify the optimization

Signal Recovery

Candes Romberg Tao Donoho

• Recovery: given(ill-posed inverse problem) find (sparse)

• Optimization:

• Convexify the optimization

• Correct!

• Polynomial time alg(linear programming)

Signal Recovery

Compressive Sensing

• Signal recovery via optimization [Candes, Romberg, Tao; Donoho]

random measurements

sparsesignal

nonzeroentries

Compressive Sensing

• Signal recovery via iterative greedy algorithm

– (orthogonal) matching pursuit [Gilbert, Tropp]

– iterated thresholding [Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; …]

– CoSaMP [Needell and Tropp]

random measurements

sparsesignal

nonzeroentries

Iterated Thresholding

update signal estimate

prune signal estimate(best K-term approx)

update residual

Summary: CS

• Encoding: = random linear combinations of the entries of

• Decoding: Recover from via optimization

measurements sparsesignal

nonzeroentries

CS Hallmarks

• Stable– acquisition/recovery process is numerically stable

• Asymmetrical (most processing at decoder) – conventional: smart encoder, dumb decoder– CS: dumb encoder, smart decoder

• Democratic– each measurement carries the same amount of information– robust to measurement loss and quantization– “digital fountain” property

• Random measurements encrypted

• Universal – same random projections / hardware can be used for

any sparse signal class (generic)

Compressive SensingIn Action

Gerhard Richter 4096 Farben / 4096 Colours

1974254 cm X 254 cmLaquer on CanvasCatalogue Raisonné: 359

Museum Collection:Staatliche Kunstsammlungen Dresden (on loan)

Sales history: 11 May 2004Christie's New York Post-War and Contemporary Art (Evening Sale), Lot 34US$3,703,500  

“Single-Pixel” CS Camera

randompattern onDMD array

DMD DMD

single photon detector

imagereconstructionorprocessing

w/ Kevin Kelly

scene

“Single-Pixel” CS Camera

randompattern onDMD array

DMD DMD

single photon detector

imagereconstructionorprocessing

scene

• Flip mirror array M times to acquire M measurements• Sparsity-based (linear programming) recovery

…

First Image Acquisition

target 65536 pixels

1300 measurements (2%)

11000 measurements (16%)

Utility?

DMD DMD

single photon detector

Fairchild100Mpixel

CCD

CS Low-Light Imaging with PMT

true color low-light imaging

256 x 256 image with 10:1 compression

[Nature Photonics, April 2007]

CS Infrared Camera

20% 5%

CS Hyperspectral Imager

spectrometer

hyperspectral data cube450-850nm

1M space x wavelength voxels200k random sums

CS MRI

• Lustig, Pauly, Donoho et al at Stanford

• Design MRI sampling patterns (in frequency/k-space) to be close to random

• Speed up acquisition by reducing number of samples required for a given image reconstruction quality

Multi-Slice Brain Imaging [M. Lustig]

3DFT Angiography [M. Lustig]

CS In Action• CS makes sense when measurements

are expensive

• Ultrawideband A/D converters[DARPA “Analog to Information” program]

• Camera networks– sensing/compression/fusion

• Radar, sonar, array processing– exploit spatial sparsity of targets

• DNA microarrays– smaller, more agile arrays for

bio-sensing [Milenkovic, Orlitsky, B]

Summary

• Compressive sensing– randomized dimensionality reduction– integrates sensing, compression, processing– exploits signal sparsity information– enables new sensing modalities, architectures, systems– relies on large-scale optimization

• Why CS works: preserves information in signals with concise geometric

structure sparse signals | compressible signals | manifolds

Open Research Issues

• Links with information theory– new encoding matrix design via codes (LDPC, fountains)– new decoding algorithms (BP, etc.)– quantization and rate distortion theory

• Links with machine learning– Johnson-Lindenstrauss, manifold embedding, RIP

• Processing/inference on random projections– filtering, tracking, interference cancellation, …

• Multi-signal CS– array processing, localization, sensor networks, …

• CS hardware– ADCs, receivers, cameras, imagers, radars, …

Beyond Sparsity

• Sparse signal model captures simplistic primary structure

wavelets:natural images

Gabor atoms:chirps/tones

pixels:background subtracted

images

Structured Sparsity

• Sparse signal model captures simplistic primary structure

• Modern compression/processing algorithms capture richer secondary coefficient structure

wavelets:natural images

Gabor atoms:chirps/tones

pixels:background subtracted

images

Wavelet Tree-Sparse Recovery

target signal CoSaMP, (RMSE=1.12)

Tree-sparse CoSaMP (RMSE=0.037)

N=1024M=80

L1-minimization(RMSE=0.751)

dsp.rice.edu/cs

vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective

create

share

usefreely

re-useopenly

vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective

create

share

usefreely

re-useopenly

vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective

create share

use re-usefreely openly

why?

today’s textbooks/courses lock up educational ideas – closed formats– closed copyrights

create share

use re-usefreely openly

open education

OEenablers

enabler 1: technology

Web/XMLmodularizationcommon framework for sharing

Internetvirtually free distributionvirtually infinite, permanent

storage

Connexions – primordial state

textbook / course

personalize

reuse

enabler 2: new IP

intellectual propertyand copyright

make content safe to share

common legal vocabulary

inspiration: open-source software (ex: Linux)

today’s textbook pipeline

authoring

editing

quality control

publishing

distribution

open education ecosystem

authoring

editing

quality control

publishing

distribution

feedbackpeersuserslearning

Connexions (cnx.org)

non-profit OE platform/communitybased at Rice U since 1999($6m+ in foundation funding)

950+ open textbooks/courses15000+ reusable Lego modules1.5 million+ unique users/month

free on-linelow-cost in print

Universality

• Random measurements can be used for signals sparse in any basis

Universality

• Random measurements can be used for signals sparse in any basis

Universality

• Random measurements can be used for signals sparse in any basis

sparsecoefficient

vector

nonzeroentries