compressive sensing a new approach to image acquisition and processing richard baraniuk mona sheikh...
TRANSCRIPT
CompressiveSensingA New Approach to Image Acquisition and Processing
Richard BaraniukMona Sheikh
Rice University
The Digital Universe
• Size: 281 billion gigabytes generated in 2007digital bits > stars in the universegrowing by a factor of 10 every 5 years > Avogadro’s number (6.02x1023) in 15 years
• Growth fueled by multimedia data audio, images, video, surveillance cameras, sensor nets, …(2B fotos on Flickr, 4.2M security cameras in UK, …)
• In 2007 digital data generated > total storageby 2011, ½ of digital universe will have no home
[Source: IDC Whitepaper “The Diverse and Exploding Digital Universe” March 2008]
Act I What’s wrong with today’s multimedia sensor systems?why go to all the work to acquire massiveamounts of multimedia data only to throw much/most of it away?
Act II One way out: dimensionality reduction (compressive sensing)enables the design of radically new sensors and systems
Act III Compressive sensing in actionnew cameras, imagers, ADCs, …
Postlude Open-access education
Sense by Sampling
sample
Sense by Sampling
sample too much data!
Sense then Compress
compress
decompress
sample
JPEGJPEG2000
…
Sparsity
pixels largewaveletcoefficients
(blue = 0)
Sparsity
pixels largewaveletcoefficients
(blue = 0)
widebandsignalsamples
largeGabor (TF)coefficients
time
frequency
What’s Wrong with this Picture?
• Why go to all the work to acquire N samples only to discard all but K pieces of data?
compress
decompress
sample
What’s Wrong with this Picture?linear processinglinear signal model(bandlimited subspace)
compress
decompress
sample
nonlinear processingnonlinear signal model(union of subspaces)
Compressive Sensing
• Directly acquire “compressed” data via dimensionality reduction
• Replace samples by more general “measurements”
compressive sensing
recover
Compressive SensingTheory
A Geometrical Perspective
• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain
Sampling
sparsesignal
nonzeroentries
• Signal is -sparse in basis/dictionary– WLOG assume sparse in space domain
• Sampling
sparsesignal
nonzeroentries
measurements
Sampling
Compressive Sensing
• When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction
measurements sparsesignal
nonzeroentries
How Can It Work?
• Projection not full rank…
… and so loses information in general
• Ex: Infinitely many ’s map to the same(null space)
How Can It Work?
• Projection not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
columns
How Can It Work?
• Projection not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
• is effectively MxK
columns
How Can It Work?
• Projection not full rank…
… and so loses information in general
• But we are only interested in sparse vectors
• Design so that each of its MxK submatrices are full rank (ideally close to orthobasis)– Restricted Isometry Property (RIP)
columns
Geometrical Interpretation of RIP
sparsesignal
nonzeroentries
• An information preserving projection preserves the geometry of the set of sparse signals
Geometrical Interpretation of RIP
sparsesignal
nonzeroentries
• An information preserving projection preserves the geometry of the set of sparse signals
• Model: union of K-dimensional subspaces aligned w/ coordinate axes (highly nonlinear!)
RIP = Stable Embedding• An information preserving projection preserves
the geometry of the set of sparse signals
• RIP ensures that
K-dim subspaces
RIP = Stable Embedding• An information preserving projection preserves
the geometry of the set of sparse signals
• RIP ensures that
How Can It Work?
• Projection not full rank…
… and so loses information in general
• Design so that each of its MxK submatrices are full rank (RIP)
• Unfortunately, a combinatorial, NP-complete design problem
columns
Insight from the 70’s [Kashin, Gluskin]
• Draw at random– iid Gaussian– iid Bernoulli
…
• Then has the RIP with high probability provided
columns
Insight from the 70’s [Kashin, Gluskin]
• Draw at random– iid Gaussian– iid Bernoulli
…
• Then has the RIP with high probability provided
columns
Randomized Sensing
• Measurements = random linear combinations of the entries of
• No information loss for sparse vectors whp
measurements sparsesignal
nonzeroentries
CS Signal Recovery
• Goal: Recover signal from measurements
• Problem: Randomprojection not full rank(ill-posed inverse problem)
• Solution: Exploit the sparse/compressiblegeometry of acquired signal
CS Signal Recovery• Random projection
not full rank
• Recovery problem:givenfind
• Null space
• Search in null space for the “best” according to some criterion– ex: least squares (N-M)-dim hyperplane
at random angle
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
Signal Recovery
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
• Wrong answer!
Signal Recovery
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Closed-form solution:
• Wrong answer!
Signal Recovery
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
Signal Recovery
“find sparsest vectorin translated nullspace”
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Correct!
• But NP-Complete alg
Signal Recovery
“find sparsest vectorin translated nullspace”
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Convexify the optimization
Signal Recovery
Candes Romberg Tao Donoho
• Recovery: given(ill-posed inverse problem) find (sparse)
• Optimization:
• Convexify the optimization
• Correct!
• Polynomial time alg(linear programming)
Signal Recovery
Compressive Sensing
• Signal recovery via optimization [Candes, Romberg, Tao; Donoho]
random measurements
sparsesignal
nonzeroentries
Compressive Sensing
• Signal recovery via iterative greedy algorithm
– (orthogonal) matching pursuit [Gilbert, Tropp]
– iterated thresholding [Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; …]
– CoSaMP [Needell and Tropp]
random measurements
sparsesignal
nonzeroentries
Iterated Thresholding
update signal estimate
prune signal estimate(best K-term approx)
update residual
Summary: CS
• Encoding: = random linear combinations of the entries of
• Decoding: Recover from via optimization
measurements sparsesignal
nonzeroentries
CS Hallmarks
• Stable– acquisition/recovery process is numerically stable
• Asymmetrical (most processing at decoder) – conventional: smart encoder, dumb decoder– CS: dumb encoder, smart decoder
• Democratic– each measurement carries the same amount of information– robust to measurement loss and quantization– “digital fountain” property
• Random measurements encrypted
• Universal – same random projections / hardware can be used for
any sparse signal class (generic)
Compressive SensingIn Action
Gerhard Richter 4096 Farben / 4096 Colours
1974254 cm X 254 cmLaquer on CanvasCatalogue Raisonné: 359
Museum Collection:Staatliche Kunstsammlungen Dresden (on loan)
Sales history: 11 May 2004Christie's New York Post-War and Contemporary Art (Evening Sale), Lot 34US$3,703,500
“Single-Pixel” CS Camera
randompattern onDMD array
DMD DMD
single photon detector
imagereconstructionorprocessing
w/ Kevin Kelly
scene
“Single-Pixel” CS Camera
randompattern onDMD array
DMD DMD
single photon detector
imagereconstructionorprocessing
scene
• Flip mirror array M times to acquire M measurements• Sparsity-based (linear programming) recovery
…
First Image Acquisition
target 65536 pixels
1300 measurements (2%)
11000 measurements (16%)
Utility?
DMD DMD
single photon detector
Fairchild100Mpixel
CCD
CS Low-Light Imaging with PMT
true color low-light imaging
256 x 256 image with 10:1 compression
[Nature Photonics, April 2007]
CS Infrared Camera
20% 5%
CS Hyperspectral Imager
spectrometer
hyperspectral data cube450-850nm
1M space x wavelength voxels200k random sums
CS MRI
• Lustig, Pauly, Donoho et al at Stanford
• Design MRI sampling patterns (in frequency/k-space) to be close to random
• Speed up acquisition by reducing number of samples required for a given image reconstruction quality
Multi-Slice Brain Imaging [M. Lustig]
3DFT Angiography [M. Lustig]
CS In Action• CS makes sense when measurements
are expensive
• Ultrawideband A/D converters[DARPA “Analog to Information” program]
• Camera networks– sensing/compression/fusion
• Radar, sonar, array processing– exploit spatial sparsity of targets
• DNA microarrays– smaller, more agile arrays for
bio-sensing [Milenkovic, Orlitsky, B]
Summary
• Compressive sensing– randomized dimensionality reduction– integrates sensing, compression, processing– exploits signal sparsity information– enables new sensing modalities, architectures, systems– relies on large-scale optimization
• Why CS works: preserves information in signals with concise geometric
structure sparse signals | compressible signals | manifolds
Open Research Issues
• Links with information theory– new encoding matrix design via codes (LDPC, fountains)– new decoding algorithms (BP, etc.)– quantization and rate distortion theory
• Links with machine learning– Johnson-Lindenstrauss, manifold embedding, RIP
• Processing/inference on random projections– filtering, tracking, interference cancellation, …
• Multi-signal CS– array processing, localization, sensor networks, …
• CS hardware– ADCs, receivers, cameras, imagers, radars, …
Beyond Sparsity
• Sparse signal model captures simplistic primary structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
Structured Sparsity
• Sparse signal model captures simplistic primary structure
• Modern compression/processing algorithms capture richer secondary coefficient structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
Wavelet Tree-Sparse Recovery
target signal CoSaMP, (RMSE=1.12)
Tree-sparse CoSaMP (RMSE=0.037)
N=1024M=80
L1-minimization(RMSE=0.751)
dsp.rice.edu/cs
vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective
create
share
usefreely
re-useopenly
vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective
create
share
usefreely
re-useopenly
vibrantinteractivecommunityconnectedinnovativeup-to-dateefficienteffective
create share
use re-usefreely openly
why?
today’s textbooks/courses lock up educational ideas – closed formats– closed copyrights
create share
use re-usefreely openly
open education
OEenablers
enabler 1: technology
Web/XMLmodularizationcommon framework for sharing
Internetvirtually free distributionvirtually infinite, permanent
storage
Connexions – primordial state
textbook / course
personalize
reuse
enabler 2: new IP
intellectual propertyand copyright
make content safe to share
common legal vocabulary
inspiration: open-source software (ex: Linux)
today’s textbook pipeline
authoring
editing
quality control
publishing
distribution
open education ecosystem
authoring
editing
quality control
publishing
distribution
feedbackpeersuserslearning
Connexions (cnx.org)
non-profit OE platform/communitybased at Rice U since 1999($6m+ in foundation funding)
950+ open textbooks/courses15000+ reusable Lego modules1.5 million+ unique users/month
free on-linelow-cost in print
Universality
• Random measurements can be used for signals sparse in any basis
Universality
• Random measurements can be used for signals sparse in any basis
Universality
• Random measurements can be used for signals sparse in any basis
sparsecoefficient
vector
nonzeroentries