shriram sarvotham dror baron richard baraniuk
DESCRIPTION
Measurements and Bits: Compressed Sensing meets Information Theory. Shriram Sarvotham Dror Baron Richard Baraniuk. ECE Department Rice University dsp.rice.edu/cs. CS encoding. Replace samples by more general encoder based on a few linear projections (inner products) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/1.jpg)
ECE DepartmentRice Universitydsp.rice.edu/cs
Measurements and Bits: Compressed Sensing
meets Information Theory
Shriram Sarvotham
Dror Baron
Richard Baraniuk
![Page 2: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/2.jpg)
CS encoding• Replace samples by more general encoder
based on a few linear projections (inner products)• Matrix vector multiplication
measurements sparsesignal
# non-zeros
![Page 3: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/3.jpg)
The CS revelation –• Of the infinitely many solutions seek the one
with smallest L1 norm
![Page 4: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/4.jpg)
• Of the infinitely many solutions seek the onewith smallest L1 norm
• If then perfect reconstruction w/ high probability [Candes et al.; Donoho]
• Linear programming
The CS revelation –
![Page 5: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/5.jpg)
Compressible signals• Polynomial decay of signal components
• Recovery algorithms– reconstruction performance:
– also requires– polynomial complexity (BPDN) [Candes et al.]
• Cannot reduce order of [Kashin,Gluskin]
squared of best term approximation
constant
![Page 6: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/6.jpg)
Fundamental goal: minimize
• Compressed sensing aims to minimize resource consumption due to measurements
• Donoho: “Why go to so much effort to acquire all the data when most of what we get will be thrown away?”
![Page 7: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/7.jpg)
Measurement reduction for sparse signals
• Ideal CS reconstruction of -sparse signal• Of the infinitely many solutions seek sparsest one• If M · K then w/ high probability this can’t be done• If M ¸ K+1 then perfect reconstruction
w/ high probability [Bresler et al.; Wakin et al.]
• But not robust and combinatorial complexity
number of nonzero entries
![Page 8: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/8.jpg)
Why is this a complicated problem?
![Page 9: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/9.jpg)
Rich design space
• What performance metric to use?– Wainwright: determine support set of nonzero entries
this is distortion metric but why let tiny nonzero entries spoil the fun?
– metric? ?
• What complexity class of reconstruction algorithms?– any algorithms? – polynomial complexity?– near-linear or better?
• How to account for imprecisions?– noise in measurements?– compressible signal model?
![Page 10: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/10.jpg)
How many measurements do we need?
![Page 11: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/11.jpg)
Measurement noise
• Measurement process is analog• Analog systems add noise, non-linearities, etc.
• Assume Gaussian noise for ease of analysis
![Page 12: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/12.jpg)
Setup
• Signal is iid• Additive white Gaussian noise• Noisy measurement process
![Page 13: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/13.jpg)
Measurement and reconstruction quality
• Measurement signal to noise ratio
• Reconstruct using decoder mapping
• Reconstruction distortion metric
• Goal: minimize CS measurement rate
![Page 14: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/14.jpg)
Measurement channel
• Model process as measurement channel
• Capacity of measurement channel
• Measurements are bits!
![Page 15: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/15.jpg)
Main result
• Theorem: For a sparse signal with rate-distortion function , lower bound on measurement rate subject to measurement quality and reconstruction distortion satisfies
• Direct relationship to rate-distortion content
![Page 16: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/16.jpg)
Main result
• Theorem: For a sparse signal with rate-distortion function , lower bound on measurement rate subject to measurement quality and reconstruction distortion satisfies
• Proof sketch:– each measurement provides bits– information content of source bits– source-channel separation for continuous amplitude sources – minimal number of measurements
– Obtain measurement rate via normalization by
![Page 17: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/17.jpg)
Example
• Spike process - spikes of uniform amplitude• Rate-distortion function• Lower bound
• Numbers:– signal of length 107
– 103 spikes– SNR=10 dB – SNR=-20 dB
![Page 18: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/18.jpg)
Upper bound (achievable) in progress…
![Page 19: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/19.jpg)
CS reconstruction meets channel coding
![Page 20: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/20.jpg)
Why is reconstruction expensive?
measurementssparsesignal
nonzeroentries
Culprit: dense, unstructured
![Page 21: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/21.jpg)
Fast CS reconstruction
measurementssparsesignal
nonzeroentries
• LDPC measurement matrix (sparse)• Only 0/1 in • Each row of contains randomly placed 1’s
• Fast matrix multiplication fast encoding and reconstruction
![Page 22: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/22.jpg)
Ongoing work: CS using BP [Sarvotham et al.]
• Considering noisy CS signals• Application of Belief Propagation
– BP over real number field– sparsity is modeled as prior in graph
• Low complexity
• Provable reconstruction with noisy measurements using
• Success of LDPC+BP in channel coding carried over to CS!
![Page 23: Shriram Sarvotham Dror Baron Richard Baraniuk](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681591d550346895dc64539/html5/thumbnails/23.jpg)
Summary
• Determination of measurement rates in CS– measurements are bits: each measurement provides
bits– lower bound on measurement rate
– direct relationship to rate-distortion content
• Compressed sensing meets information theory
• Additional research directions– promising results with LDPC measurement matrices– upper bound (achievable) on number of measurements
dsp.rice.edu/cs