![Page 1: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/1.jpg)
Connecting Bayesian and
Denoising-Based Approximate
Message
Chris Metzler, Richard Baraniuk
Rice University
Arian Maleki
Columbia University
![Page 2: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/2.jpg)
Compressive Sensing Problem
![Page 3: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/3.jpg)
Solution: Assume Structure
• Sparse: OMP [Tropp 04], IST [Figueiredo et al. 07], AMP [Donoho et al. 09]
• Minimal Total Variation: TVAL3 [Li et al. 09], TV-AMP [Donoho et al. 13]
• Tree-Sparse: Model-CoSaMP [Baraniuk et al. 10],Turbo-AMP [Som and Schniter 12]
• Group-Sparse: NLR-CS [Dong et al. 14]
![Page 4: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/4.jpg)
Using Structure is Hard
• Write as penalty or constraint
• How to efficiently solve for non-convex R(x)?
• What is R(x) for natural images?
• What is R(x) for RF, microscopy, and other applications?
![Page 5: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/5.jpg)
This Talk
• Develop algorithm that can easily use almost any structure
• Predict performance with accurate state evolution
• Derive theoretical guarantees• Measurements required
• Robust to noise
• Optimality and suboptimality
• Demonstrate state-of-the-art performance• 10dB better than wavelet sparsity
• 50x faster than group-sparsity
![Page 6: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/6.jpg)
Insight: Denoisers Use Structure
• Gaussian Kernel• Smooth
• Soft Wavelet Thresholding [Donoho and Johnstone 94]
• Sparse wavelet representation
• BLS-GSM [Portilla et al. 03]
• Coefficients follow Gaussian Mixture Model
• NLM [Baudes et al. 05]
• Correlated structures
• BM3D [Dabov et al. 07]
• Group-sparse in DCT/Wavelet representation
• BM3D-SAPCA [Dabov et al. 09]
• Group-sparse in adaptive basis
![Page 7: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/7.jpg)
Denoisers as Black Boxes
Denoiser
![Page 8: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/8.jpg)
Denoisers as Projections
C
![Page 9: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/9.jpg)
Naïve Algorithm: Denoising-based
Iterative Thresholding (D-IT)
![Page 10: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/10.jpg)
Naïve Algorithm: D-IT
Our prior on x
![Page 11: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/11.jpg)
Naïve Algorithm: D-IT
![Page 12: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/12.jpg)
Naïve Algorithm: D-IT
![Page 13: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/13.jpg)
Naïve Algorithm: D-IT
![Page 14: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/14.jpg)
Naïve Algorithm: D-IT
![Page 15: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/15.jpg)
Failure of D-IT: Systematic Errors
![Page 16: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/16.jpg)
Systematic Errors: Overshooting
![Page 17: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/17.jpg)
Systematic Errors: Overshooting
Too High
![Page 18: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/18.jpg)
Systematic Errors: Overshooting
Too High Too Low
Too High Too Low
![Page 19: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/19.jpg)
Systematic Errors: Overshooting
Too High Too Low
Too High Too Low
![Page 20: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/20.jpg)
Systematic Errors: Overshooting
Too High Too Low
Too High Too Low
![Page 21: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/21.jpg)
Systematic Errors: Overshooting
Too High Too Low
Too High Too Low
Use residuals from
previous iterations to
avoid over/under-
shooting
![Page 22: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/22.jpg)
New Algorithm: D-AMP
Onsager Correction
![Page 23: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/23.jpg)
D-AMP Benefits
D-IT
• Updates proportional to residual (P-controller).
• 5dB improvement over L1
D-AMP
• Updates proportional to previous residual (PI-controller).
• 10dB improvement over L1
(state-of-the-art)
• Onsager Correction
![Page 24: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/24.jpg)
Onsager Correction:
• Where did it come from?• Approximation of message passing algorithm
![Page 25: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/25.jpg)
Onsager Correction:
• Where did it come from?• Approximation of message passing algorithm
• Why does it help?• zt stores residuals over many iterations (momentum)
• Corrects for bias in denoiser solutions
• Makes errors uncorrelated (Gaussian) and thus easy to remove
![Page 26: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/26.jpg)
Onsager Correction:
• Where did it come from?• Approximation of message passing algorithm
• Why does it help?• zt stores residuals over many iterations (momentum)
• Corrects for bias in denoiser solutions
• Makes errors uncorrelated (Gaussian) and thus easy to remove
• How is it calculated?• Approximation from Monte Carlo SURE [Ramani et al. 08]
![Page 27: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/27.jpg)
D-AMP Avoids Systematic Errors
![Page 28: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/28.jpg)
D-AMP Theoretical Properties
• State evolution predicts performance
• Explicit phase transition
• Robust to noise
• No algorithm can uniformly outperform D-AMP
• Single-class suboptimal
• Easy to tune
![Page 29: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/29.jpg)
Bayesian-AMP and Bayesian
State Evolution
• Algorithm
• State Evolution
![Page 30: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/30.jpg)
Denoising-based AMP and
Deterministic State Evolution
• Algorithm
• State Evolution
![Page 31: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/31.jpg)
State Evolution Comparison
• With minimax denoiser, suprema of Deterministic and Bayesian state evolutions are equivalent:
• Significance of deterministic state evolution: • Can apply without knowing x’s distribution
• Can apply to natural images and other complex signals
![Page 32: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/32.jpg)
State Evolution of D-IT and D-AMP
![Page 33: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/33.jpg)
State Evolution is Accurate for
Many Denoisers
![Page 34: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/34.jpg)
State Evolution for Discontinuous
Denoisers
![Page 35: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/35.jpg)
State Evolution for Smoothed
Discontinuous Denoisers
![Page 36: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/36.jpg)
Main Theoretical Results
• Denoiser Performance
• Phase Transition: Determined by denoiser
• Noise Sensitivity: Graceful failure
• No algorithm can uniformly outperform D-AMP
• D-AMP is single-class suboptimal
![Page 37: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/37.jpg)
Parameter Tuning
• Denoiser parameters
• Problem: Tune denoiser parameters over multiple iterations
• Result: Greedy parameter selection is optimal
![Page 38: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/38.jpg)
3x Under-Sampling
![Page 39: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/39.jpg)
20x Under-Sampling
Wavelet Sparse (L1) BM3D-AMP (our algorithm)
![Page 40: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/40.jpg)
10x Under-Sampling with Noise
NLR-CS BM3D-AMP
![Page 41: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/41.jpg)
Performance without Noise
![Page 42: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/42.jpg)
Computation Time
30x Faster
70x Faster
![Page 43: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/43.jpg)
Performance with Noise
![Page 44: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/44.jpg)
D-AMP Summary
• Arbitrary denoiser• NLM
• BM3D
• Useful state evolution
• State-of-the-art performance
• Resilient to noise
• >97% reduction in average computation time
C. Metzler, A. Maleki, R. G. Baraniuk, “From Denoising to
Compressed Sensing,” arXiv:1406.4175.pdf
![Page 45: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/45.jpg)
D-AMP vs. AMP1, G-AMP
2, Turbo-
AMP3, TV-AMP
4, GrAMPA
5, etc.
Similarities
• Same basic AMP iterations
• Solve a series of denoisingproblems
• Better denoisers lead to better phase transition and noise sensitivity
Differences
• Separable denoisers without scale invariance
• Signal x can be denoised but need not have generalized-sparsity nor known px
• Approximate Onsager correction
• New deterministic state evolution
• State evolution holds for separable, but continuous, denoisers
• Derive phase transition and noise sensitivity of non-sparse signals
• Derive optimality/sub-optimality
• Optimal tuning strategy
1. Donoho et al. 09
2. Rangan 12
3. Som and Schniter 12
4. Donoho et al. 13
5. Borgerding et al. 14
![Page 46: Connecting Bayesian and Denoising-Based Approximate …cam6.web.rice.edu/talks/Asilomar_v4.pdfD-AMP Benefits D-IT •Updates proportional to residual (P-controller). •5dB improvement](https://reader034.vdocuments.us/reader034/viewer/2022042309/5ed68acdff0e593c0b640a68/html5/thumbnails/46.jpg)
Near Proper Denoiser
• Denoiser Performance
• Phase Transition: Determined by denoiser
• Noise Sensitivity: Graceful failure