bayesian methods for sparse signal recovery · bayesian methods for sparse signal recovery bhaskar...
TRANSCRIPT
![Page 1: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/1.jpg)
Bayesian Methods for Sparse Signal Recovery
Bhaskar D Rao1
University of California, San Diego
1
Thanks to David Wipf, Jason Palmer, Zhilin Zhang and Ritwik Giri
Bhaskar D Rao University of California, San Diego
![Page 2: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/2.jpg)
Motivation
Sparse Signal Recovery is an interesting area with many potentialapplications.
Methods developed for solving sparse signal recovery problem can be avaluable tool for signal processing practitioners.
Many interesting developments in recent past that make the subjecttimely.
Bayesian Framework o↵ers some interesting options.
Bhaskar D Rao University of California, San Diego
![Page 3: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/3.jpg)
Motivation
Sparse Signal Recovery is an interesting area with many potentialapplications.
Methods developed for solving sparse signal recovery problem can be avaluable tool for signal processing practitioners.
Many interesting developments in recent past that make the subjecttimely.
Bayesian Framework o↵ers some interesting options.
Bhaskar D Rao University of California, San Diego
![Page 4: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/4.jpg)
Motivation
Sparse Signal Recovery is an interesting area with many potentialapplications.
Methods developed for solving sparse signal recovery problem can be avaluable tool for signal processing practitioners.
Many interesting developments in recent past that make the subjecttimely.
Bayesian Framework o↵ers some interesting options.
Bhaskar D Rao University of California, San Diego
![Page 5: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/5.jpg)
Motivation
Sparse Signal Recovery is an interesting area with many potentialapplications.
Methods developed for solving sparse signal recovery problem can be avaluable tool for signal processing practitioners.
Many interesting developments in recent past that make the subjecttimely.
Bayesian Framework o↵ers some interesting options.
Bhaskar D Rao University of California, San Diego
![Page 6: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/6.jpg)
Motivation
Sparse Signal Recovery is an interesting area with many potentialapplications.
Methods developed for solving sparse signal recovery problem can be avaluable tool for signal processing practitioners.
Many interesting developments in recent past that make the subjecttimely.
Bayesian Framework o↵ers some interesting options.
Bhaskar D Rao University of California, San Diego
![Page 7: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/7.jpg)
Outline
Sparse Signal Recovery (SSR) Problem and some Extensions
Applications
Bayesian Methods
MAP estimationEmpirical Bayes
Summary
Bhaskar D Rao University of California, San Diego
![Page 8: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/8.jpg)
Outline
Sparse Signal Recovery (SSR) Problem and some Extensions
Applications
Bayesian Methods
MAP estimationEmpirical Bayes
Summary
Bhaskar D Rao University of California, San Diego
![Page 9: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/9.jpg)
Outline
Sparse Signal Recovery (SSR) Problem and some Extensions
Applications
Bayesian Methods
MAP estimationEmpirical Bayes
Summary
Bhaskar D Rao University of California, San Diego
![Page 10: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/10.jpg)
Outline
Sparse Signal Recovery (SSR) Problem and some Extensions
Applications
Bayesian Methods
MAP estimationEmpirical Bayes
Summary
Bhaskar D Rao University of California, San Diego
![Page 11: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/11.jpg)
Outline
Sparse Signal Recovery (SSR) Problem and some Extensions
Applications
Bayesian Methods
MAP estimationEmpirical Bayes
Summary
Bhaskar D Rao University of California, San Diego
![Page 12: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/12.jpg)
Problem Description: Sparse Signal Recovery (SSR)
y is a N ⇥ 1 measurement vector.
� is N ⇥M dictionary matrix where M >> N.
x is M ⇥ 1 desired vector which is sparse with k non zero entries.
v is the measurement noise.Bhaskar D Rao University of California, San Diego
![Page 13: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/13.jpg)
Problem Statement: SSR
Noise Free Case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to y = �x
I (.) is the indicator function.
Noisy case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to ky � �xk2
< �
Bhaskar D Rao University of California, San Diego
![Page 14: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/14.jpg)
Problem Statement: SSR
Noise Free Case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to y = �x
I (.) is the indicator function.
Noisy case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to ky � �xk2
< �
Bhaskar D Rao University of California, San Diego
![Page 15: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/15.jpg)
Problem Statement: SSR
Noise Free Case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to y = �x
I (.) is the indicator function.
Noisy case
Given a target signal y and dictionary �, find the weights x that solve,
minx
X
i
I (xi
6= 0) subject to ky � �xk2
< �
Bhaskar D Rao University of California, San Diego
![Page 16: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/16.jpg)
Useful Extensions
Block Sparsity
Multiple Measurement Vectors (MMV)
Block MMV
MMV with time varying sparsity
Bhaskar D Rao University of California, San Diego
![Page 17: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/17.jpg)
Useful Extensions
Block Sparsity
Multiple Measurement Vectors (MMV)
Block MMV
MMV with time varying sparsity
Bhaskar D Rao University of California, San Diego
![Page 18: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/18.jpg)
Useful Extensions
Block Sparsity
Multiple Measurement Vectors (MMV)
Block MMV
MMV with time varying sparsity
Bhaskar D Rao University of California, San Diego
![Page 19: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/19.jpg)
Useful Extensions
Block Sparsity
Multiple Measurement Vectors (MMV)
Block MMV
MMV with time varying sparsity
Bhaskar D Rao University of California, San Diego
![Page 20: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/20.jpg)
Block Sparsity
Variations include equal blocks, unequal blocks, block boundary known orunknown.
Bhaskar D Rao University of California, San Diego
![Page 21: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/21.jpg)
Multiple Measurement Vectors (MMV)
Multiple measurements: L measurements
Common Sparsity Profile: k nonzero rows
Bhaskar D Rao University of California, San Diego
![Page 22: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/22.jpg)
Applications
Signal Representation (Mallat, Coifman, Donoho,..)
EEG/MEG (Leahy, Gorodnitsky,Ioannides,..)
Robust Linear Regression and Outlier Detection (Jin, Giannakis, ..)
Speech Coding (Ozawa, Ono, Kroon,..)
Compressed Sensing (Donoho, Candes, Tao,..)
Magnetic Resonance Imaging (Lustig,..)
Sparse Channel Equalization (Fevrier, Proakis,...)
Face Recognition (Wright, Yang, ...)
Cognitive Radio (Eldar, ..)
and many more.........
Bhaskar D Rao University of California, San Diego
![Page 23: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/23.jpg)
Applications
Signal Representation (Mallat, Coifman, Donoho,..)
EEG/MEG (Leahy, Gorodnitsky,Ioannides,..)
Robust Linear Regression and Outlier Detection (Jin, Giannakis, ..)
Speech Coding (Ozawa, Ono, Kroon,..)
Compressed Sensing (Donoho, Candes, Tao,..)
Magnetic Resonance Imaging (Lustig,..)
Sparse Channel Equalization (Fevrier, Proakis,...)
Face Recognition (Wright, Yang, ...)
Cognitive Radio (Eldar, ..)
and many more.........
Bhaskar D Rao University of California, San Diego
![Page 24: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/24.jpg)
MEG/EEG Source Localization
?
)�
source space (x) sensor space (y)
i Forward model dictionary ) can be computed using Maxwell’s equations [Sarvas,1987].
i In many situations the active brain regions may be relatively sparse, and so solving a sparse inverse problem is required.
�
[Baillet et al., 2001]
Bhaskar D Rao University of California, San Diego
![Page 25: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/25.jpg)
Sparse Channel Estimation
Potential Application: Underwater Acoustics
Bhaskar D Rao University of California, San Diego
![Page 26: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/26.jpg)
Speech Modeling and Deconvolution
Speech Model
Speech specific assumptions: Voiced excitation is block sparse and thefilter is an all pole filter 1
A(z)
Bhaskar D Rao University of California, San Diego
![Page 27: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/27.jpg)
Compressive Sampling (CS)
Transform Coding
is the transform and b is the original data/image.
Bhaskar D Rao University of California, San Diego
![Page 28: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/28.jpg)
Compressive Sampling (CS)
Transform Coding
is the transform and b is the original data/image.
Bhaskar D Rao University of California, San Diego
![Page 29: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/29.jpg)
Compressive Sampling (CS)
Computation:
Solve for x such that �x = y .
Reconstruction: b = x
Issues:
Need to recover sparse signal x with constraint �x = y .
Need to design sampling matrix A.
Bhaskar D Rao University of California, San Diego
![Page 30: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/30.jpg)
Compressive Sampling (CS)
Computation:
Solve for x such that �x = y .
Reconstruction: b = x
Issues:
Need to recover sparse signal x with constraint �x = y .
Need to design sampling matrix A.
Bhaskar D Rao University of California, San Diego
![Page 31: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/31.jpg)
Compressive Sampling (CS)
Computation:
Solve for x such that �x = y .
Reconstruction: b = x
Issues:
Need to recover sparse signal x with constraint �x = y .
Need to design sampling matrix A.
Bhaskar D Rao University of California, San Diego
![Page 32: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/32.jpg)
Model noise
w: Sparse Component,
Outliers
ε: Gaussian Component, Regular error
y X c n Robust Linear Regression X, y: data; c: regression coeffs.; n: model noise;
Transform into overcomplete representation:
Y = X c + Φ w + ε, where Φ=I, or Y = [X , Φ] + ε
cwª º« »¬ ¼
Bhaskar D Rao University of California, San Diego
![Page 33: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/33.jpg)
Potential Algorithmic Approaches
Finding the Optimal Solution is NP hard. So need low complexityalgorithms with reasonable performance.
Greedy Search Techniques
Matching Pursuit (MP), Orthogonal Matching Pursuit (OMP).
Minimizing Diversity Measures
Indicator function is not continuous. Define Surrogate Cost functionsthat are more tractable and whose minimization leads to sparse solutions,e.g. `
1
minimization.
Bayesian Methods
Make appropriate Statistical assumptions on the solution and applyestimation techniques to identify the desired sparse solution.
Bhaskar D Rao University of California, San Diego
![Page 34: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/34.jpg)
Bayesian Methods
1. MAP Estimation Framework (Type I)
2. Hierarchical Bayesian Framework (Type II)
Bhaskar D Rao University of California, San Diego
![Page 35: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/35.jpg)
MAP Estimation Framework (Type I)
Problem Statement
x̂ = argmaxx
P(x |y) = argmaxx
P(y |x)P(x)
Choice of P(x) = a
2
e�a|x| as Laplacian and P(y |x) as Gaussian will leadto the familiar LASSO framework.
Bhaskar D Rao University of California, San Diego
![Page 36: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/36.jpg)
Hierarchical Bayesian Framework (Type II)
Bhaskar D Rao University of California, San Diego
![Page 37: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/37.jpg)
Hierarchical Bayesian Framework (Type II)
Problem Statement
�̂ = argmax�
P(�|y) = argmax�
ZP(y |x)P(x |�)P(�)dx
Using this estimate of � we can compute our concerned posteriorP(x |y ; �̂).
Example: Bayesian LASSO
Laplacian prior P(x) can be represented as a Gaussian Scale Mixture inthis fashion,
P(x) =
ZP(x |�)P(�)d�
=
Z1p2⇡�
exp(� x2
2�)⇥ a2
2exp(�a2
2�)d�
=a
2exp(�a|x |)
Bhaskar D Rao University of California, San Diego
![Page 38: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/38.jpg)
Hierarchical Bayesian Framework (Type II)
Problem Statement
�̂ = argmax�
P(�|y) = argmax�
ZP(y |x)P(x |�)P(�)dx
Using this estimate of � we can compute our concerned posteriorP(x |y ; �̂).
Example: Bayesian LASSO
Laplacian prior P(x) can be represented as a Gaussian Scale Mixture inthis fashion,
P(x) =
ZP(x |�)P(�)d�
=
Z1p2⇡�
exp(� x2
2�)⇥ a2
2exp(�a2
2�)d�
=a
2exp(�a|x |)
Bhaskar D Rao University of California, San Diego
![Page 39: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/39.jpg)
MAP Estimation
Problem Statement
x̂ = argmaxx
P(x |y) = argmaxx
P(y |x)P(x)
Advantages
Many options to promote sparsity, i.e. choose some sparse priorover x .
Growing options for solving the underlying optimization problem.
Can be related to LASSO and other `1
minimization techniques byusing suitable P(x).
Bhaskar D Rao University of California, San Diego
![Page 40: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/40.jpg)
MAP Estimation
Assumption: Gaussian Noise
x̂ = argmaxx
P(y |x)P(x)
= argminx
�logP(y |x)� logP(x)
= argminx
ky � �xk22
+ �mX
i=1
g(|xi
|)
Theorem
If g is non decreasing and strictly concave function for x 2 R+, the localminima of the above optimization problem will be the extreme points, i.e.have max of N non-zero entries.
Bhaskar D Rao University of California, San Diego
![Page 41: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/41.jpg)
MAP Estimation
Assumption: Gaussian Noise
x̂ = argmaxx
P(y |x)P(x)
= argminx
�logP(y |x)� logP(x)
= argminx
ky � �xk22
+ �mX
i=1
g(|xi
|)
Theorem
If g is non decreasing and strictly concave function for x 2 R+, the localminima of the above optimization problem will be the extreme points, i.e.have max of N non-zero entries.
Bhaskar D Rao University of California, San Diego
![Page 42: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/42.jpg)
Special cases of MAP estimation
Gaussian Prior
Gaussian assumption of P(x) leads to `2
norm regularized problem
x̂ = argminx
ky � �xk22
+ �kxk22
Laplacian Prior
Laplacian assumption of P(x) leads to standard `1
norm regularizedproblem i.e. LASSO.
x̂ = argminx
ky � �xk22
+ �kxk1
Bhaskar D Rao University of California, San Diego
![Page 43: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/43.jpg)
Special cases of MAP estimation
Gaussian Prior
Gaussian assumption of P(x) leads to `2
norm regularized problem
x̂ = argminx
ky � �xk22
+ �kxk22
Laplacian Prior
Laplacian assumption of P(x) leads to standard `1
norm regularizedproblem i.e. LASSO.
x̂ = argminx
ky � �xk22
+ �kxk1
Bhaskar D Rao University of California, San Diego
![Page 44: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/44.jpg)
Examples of Sparse Distributions
Sparse distributions can be viewed using a general framework ofsupergaussian distribution.
P(x ;�, p) =p
2 p
p2��( 1
p
)e
�|x|p2�p , p 1
If a unit variance distribution is desired � becomes a function of p.
Bhaskar D Rao University of California, San Diego
![Page 45: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/45.jpg)
Example of Sparsity Penalties
Practical Selections
g(xi
) = log(x2i
+ ✏), [Chartrand and Yin, 2008]g(x
i
) = log(|xi
|+ ✏), [Candes et al., 2008]g(x
i
) = |xi
|p, [Rao et al., 1999]
Di↵erent choices favor di↵erent levels of sparsity.
Bhaskar D Rao University of California, San Diego
![Page 46: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/46.jpg)
Which Sparse prior to choose?
x̂ = argminx
ky � �xk22
+ �MX
l=1
|xl
|p
Two issues:
If the prior is too sparse, i.e. p ⇠ 0, then we may get stuck at alocal minima which results in convergence error.
If the prior is not sparse enough, i.e. p ⇠ 1, then though globalminima can be found, it may not be the sparsest solution, whichresults in a structural error.
Bhaskar D Rao University of California, San Diego
![Page 47: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/47.jpg)
MAP Estimation
Underlying Optimization problem is
x̂ = argminx
ky � �xk22
+ �mX
i=1
g(|xi
|)
Useful algorithms exist to minimize the cost function with a strictlyconcave penalty function g on R+ (Reweighted `
2
/`1
algorithms).
The essence of this algorithm is to create a bound for the concavepenalty function and follow the steps of a Majorize-Minimization(MM) algorithm.
Bhaskar D Rao University of California, San Diego
![Page 48: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/48.jpg)
MAP Estimation
Underlying Optimization problem is
x̂ = argminx
ky � �xk22
+ �mX
i=1
g(|xi
|)
Useful algorithms exist to minimize the cost function with a strictlyconcave penalty function g on R+ (Reweighted `
2
/`1
algorithms).
The essence of this algorithm is to create a bound for the concavepenalty function and follow the steps of a Majorize-Minimization(MM) algorithm.
Bhaskar D Rao University of California, San Diego
![Page 49: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/49.jpg)
MAP Estimation
Underlying Optimization problem is
x̂ = argminx
ky � �xk22
+ �mX
i=1
g(|xi
|)
Useful algorithms exist to minimize the cost function with a strictlyconcave penalty function g on R+ (Reweighted `
2
/`1
algorithms).
The essence of this algorithm is to create a bound for the concavepenalty function and follow the steps of a Majorize-Minimization(MM) algorithm.
Bhaskar D Rao University of California, San Diego
![Page 50: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/50.jpg)
Reweighted `1 optimization
Assume: g(xi
) = h(|xi
|) with h concave.
Now we have to bound this concave penalty function.
Bhaskar D Rao University of California, San Diego
![Page 51: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/51.jpg)
Reweighted `1 optimization
Assume: g(xi
) = h(|xi
|) with h concave.
Updates
x (k+1) ! argminx
ky � �xk22
+ �X
i
w(k)
i
|xi
|
wk+1
i
! @g(xi
)
@|xi
| |x
i
=x
(k+1)
i
Bhaskar D Rao University of California, San Diego
![Page 52: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/52.jpg)
Reweighted `1 optimization
Assume: g(xi
) = h(|xi
|) with h concave.
Updates
x (k+1) ! argminx
ky � �xk22
+ �X
i
w(k)
i
|xi
|
wk+1
i
! @g(xi
)
@|xi
| |x
i
=x
(k+1)
i
Bhaskar D Rao University of California, San Diego
![Page 53: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/53.jpg)
Reweighted `1 optimization
Candes et al., 2008
Penalty: g(xi
) = log(|xi
|+ ✏), 0✏ � 0
Weight Update: w (k+1)
i
! [|x (k+1)
i
+ ✏]�1
Bhaskar D Rao University of California, San Diego
![Page 54: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/54.jpg)
Reweighted `2 optimization
Assume: g(xi
) = h(x2i
) with h concave
Upper bound h(.) as before.
Bound will be quadratic in the variables leading to a weighted2-norm optimization problem
Updates
x (k+1) ! argminx
ky � �xk22
+ �X
i
w(k)
i
x2i
= W̃ (k)�T (�I + �W̃ (k)�T )�1y
wk+1
i
! @g(xi
)
@x2i
|x
i
=x
(k+1)
i
, W̃ (k+1) ! diag [w (k+1)]�1
Bhaskar D Rao University of California, San Diego
![Page 55: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/55.jpg)
Reweighted `2 optimization
Assume: g(xi
) = h(x2i
) with h concave
Upper bound h(.) as before.
Bound will be quadratic in the variables leading to a weighted2-norm optimization problem
Updates
x (k+1) ! argminx
ky � �xk22
+ �X
i
w(k)
i
x2i
= W̃ (k)�T (�I + �W̃ (k)�T )�1y
wk+1
i
! @g(xi
)
@x2i
|x
i
=x
(k+1)
i
, W̃ (k+1) ! diag [w (k+1)]�1
Bhaskar D Rao University of California, San Diego
![Page 56: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/56.jpg)
Reweighted `2 optimization: Examples
FOCUSS Algorithm[Rao et al., 2003]
Penalty: g(xi
) = |xi
|p, 0 p 2
Weight Update: w (k+1)
i
! |x (k+1)
i
|p�2
Properties: Well-characterized convergence rates; very susceptibleto local minima when p is small.
Chartrand and Yin (2008) Algorithm
Penalty: g(xi
) = log(x2i
+ ✏), ✏ � 0
Weight Update: w (k+1)
i
! [(x (k+1)
i
)2 + ✏]�1
Properties: Slowly reducing ✏ to zero smoothes out local minimainitially allowing better solutions to be found;
Bhaskar D Rao University of California, San Diego
![Page 57: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/57.jpg)
Reweighted `2 optimization: Examples
FOCUSS Algorithm[Rao et al., 2003]
Penalty: g(xi
) = |xi
|p, 0 p 2
Weight Update: w (k+1)
i
! |x (k+1)
i
|p�2
Properties: Well-characterized convergence rates; very susceptibleto local minima when p is small.
Chartrand and Yin (2008) Algorithm
Penalty: g(xi
) = log(x2i
+ ✏), ✏ � 0
Weight Update: w (k+1)
i
! [(x (k+1)
i
)2 + ✏]�1
Properties: Slowly reducing ✏ to zero smoothes out local minimainitially allowing better solutions to be found;
Bhaskar D Rao University of California, San Diego
![Page 58: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/58.jpg)
Empirical Comparison
Bhaskar D Rao University of California, San Diego
![Page 59: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/59.jpg)
Empirical Comparison
Figure: Probability of Successful recovery vs Number of non zerocoe�cients
Bhaskar D Rao University of California, San Diego
![Page 60: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/60.jpg)
Limitation of MAP based methods
To retain the same maximally sparse global solution as the `0
norm ingeneral conditions, then any possible MAP algorithm will possess O
⇥�M
N
�⇤
local minima.
Bhaskar D Rao University of California, San Diego
![Page 61: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/61.jpg)
Hierarchical Bayes: Sparse Bayesian Learning(SBL)
MAP estimation is just a penalized regression, hence BayesianInterpretation has not contributed much as of now.
MAP methods were interested in the mode of the posterior but SBL usesposterior information beyond the mode, i.e. posterior distribution.
Problem
For all sparse priors it is not possible to compute the normalized posteriorP(x |y), hence some approximations are needed.
Bhaskar D Rao University of California, San Diego
![Page 62: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/62.jpg)
Hierarchical Bayes: Sparse Bayesian Learning(SBL)
MAP estimation is just a penalized regression, hence BayesianInterpretation has not contributed much as of now.
MAP methods were interested in the mode of the posterior but SBL usesposterior information beyond the mode, i.e. posterior distribution.
Problem
For all sparse priors it is not possible to compute the normalized posteriorP(x |y), hence some approximations are needed.
Bhaskar D Rao University of California, San Diego
![Page 63: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/63.jpg)
Hierarchical Bayes: Sparse Bayesian Learning(SBL)
MAP estimation is just a penalized regression, hence BayesianInterpretation has not contributed much as of now.
MAP methods were interested in the mode of the posterior but SBL usesposterior information beyond the mode, i.e. posterior distribution.
Problem
For all sparse priors it is not possible to compute the normalized posteriorP(x |y), hence some approximations are needed.
Bhaskar D Rao University of California, San Diego
![Page 64: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/64.jpg)
Hierarchical Bayes: Sparse Bayesian Learning(SBL)
MAP estimation is just a penalized regression, hence BayesianInterpretation has not contributed much as of now.
MAP methods were interested in the mode of the posterior but SBL usesposterior information beyond the mode, i.e. posterior distribution.
Problem
For all sparse priors it is not possible to compute the normalized posteriorP(x |y), hence some approximations are needed.
Bhaskar D Rao University of California, San Diego
![Page 65: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/65.jpg)
Hierarchical Bayesian Framework (Type II)
In order for this framework to be useful, we need tractablerepresentations: Gaussian Scaled Mixtures
Bhaskar D Rao University of California, San Diego
![Page 66: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/66.jpg)
Hierarchical Bayesian Framework (Type II)
In order for this framework to be useful, we need tractablerepresentations: Gaussian Scaled Mixtures
Bhaskar D Rao University of California, San Diego
![Page 67: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/67.jpg)
Construction of Sparse priors
Separability: P(x) =Q
i
P(xi
)
Gaussian Scale Mixture :
P(xi
) =
ZP(x
i
|�i
)P(�i
)d�i
=
ZN(x
i
; 0, �i
)P(�i
)d�i
Most of the sparse priors over x (including those with concave g) can berepresented in this GSM form, and di↵erent scale mixing density i.e,P(�
i
) will lead to di↵erent sparse priors. [Palmer et al., 2006]
Instead of solving a MAP problem in x , in the Bayesian framework oneestimates the hyperparameters � leading to an estimate of the posteriordistribution for x , i.e. P(x |y ; �̂). (Sparse Bayesian Learning)
Bhaskar D Rao University of California, San Diego
![Page 68: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/68.jpg)
Construction of Sparse priors
Separability: P(x) =Q
i
P(xi
)
Gaussian Scale Mixture :
P(xi
) =
ZP(x
i
|�i
)P(�i
)d�i
=
ZN(x
i
; 0, �i
)P(�i
)d�i
Most of the sparse priors over x (including those with concave g) can berepresented in this GSM form, and di↵erent scale mixing density i.e,P(�
i
) will lead to di↵erent sparse priors. [Palmer et al., 2006]
Instead of solving a MAP problem in x , in the Bayesian framework oneestimates the hyperparameters � leading to an estimate of the posteriordistribution for x , i.e. P(x |y ; �̂). (Sparse Bayesian Learning)
Bhaskar D Rao University of California, San Diego
![Page 69: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/69.jpg)
Construction of Sparse priors
Separability: P(x) =Q
i
P(xi
)
Gaussian Scale Mixture :
P(xi
) =
ZP(x
i
|�i
)P(�i
)d�i
=
ZN(x
i
; 0, �i
)P(�i
)d�i
Most of the sparse priors over x (including those with concave g) can berepresented in this GSM form, and di↵erent scale mixing density i.e,P(�
i
) will lead to di↵erent sparse priors. [Palmer et al., 2006]
Instead of solving a MAP problem in x , in the Bayesian framework oneestimates the hyperparameters � leading to an estimate of the posteriordistribution for x , i.e. P(x |y ; �̂). (Sparse Bayesian Learning)
Bhaskar D Rao University of California, San Diego
![Page 70: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/70.jpg)
Construction of Sparse priors
Separability: P(x) =Q
i
P(xi
)
Gaussian Scale Mixture :
P(xi
) =
ZP(x
i
|�i
)P(�i
)d�i
=
ZN(x
i
; 0, �i
)P(�i
)d�i
Most of the sparse priors over x (including those with concave g) can berepresented in this GSM form, and di↵erent scale mixing density i.e,P(�
i
) will lead to di↵erent sparse priors. [Palmer et al., 2006]
Instead of solving a MAP problem in x , in the Bayesian framework oneestimates the hyperparameters � leading to an estimate of the posteriordistribution for x , i.e. P(x |y ; �̂). (Sparse Bayesian Learning)
Bhaskar D Rao University of California, San Diego
![Page 71: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/71.jpg)
Examples of Gaussian Scale Mixture
Laplacian density
P(x ; a) =a
2exp(�a|x |)
Scale mixing density: P(�) = a
2
2
exp(� a
2
2
�), � � 0.
Student-t Distribution
P(x ; a, b) =ba�(a+ 1/2)
(2⇡)0.5�(a)
1
(b + x2/2)a+1/2
Scale mixing density: Gamma Distribution.
Generalized Gaussian
P(x ; p) =1
2�(1 + 1
p
)e�|x|p
Scale mixing density: Positive alpha stable density of order p/2.
Bhaskar D Rao University of California, San Diego
![Page 72: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/72.jpg)
Examples of Gaussian Scale Mixture
Laplacian density
P(x ; a) =a
2exp(�a|x |)
Scale mixing density: P(�) = a
2
2
exp(� a
2
2
�), � � 0.
Student-t Distribution
P(x ; a, b) =ba�(a+ 1/2)
(2⇡)0.5�(a)
1
(b + x2/2)a+1/2
Scale mixing density: Gamma Distribution.
Generalized Gaussian
P(x ; p) =1
2�(1 + 1
p
)e�|x|p
Scale mixing density: Positive alpha stable density of order p/2.
Bhaskar D Rao University of California, San Diego
![Page 73: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/73.jpg)
Examples of Gaussian Scale Mixture
Laplacian density
P(x ; a) =a
2exp(�a|x |)
Scale mixing density: P(�) = a
2
2
exp(� a
2
2
�), � � 0.
Student-t Distribution
P(x ; a, b) =ba�(a+ 1/2)
(2⇡)0.5�(a)
1
(b + x2/2)a+1/2
Scale mixing density: Gamma Distribution.
Generalized Gaussian
P(x ; p) =1
2�(1 + 1
p
)e�|x|p
Scale mixing density: Positive alpha stable density of order p/2.
Bhaskar D Rao University of California, San Diego
![Page 74: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/74.jpg)
Sparse Bayesian Learning (Tipping)
y = �x + v
Solving for the optimal �
�̂ = argmax�
P(�|y) = argmax�
P(y |�)P(�)
= argmin�
log |⌃y
|+ yT⌃�1
y
y � 2X
i
logP(�i
)
where, ⌃y
= �2I + ���T and � = diag(�)
Empirical Bayes
Choose P(�i
) to be a non-informative prior
Bhaskar D Rao University of California, San Diego
![Page 75: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/75.jpg)
Sparse Bayesian Learning (Tipping)
y = �x + v
Solving for the optimal �
�̂ = argmax�
P(�|y) = argmax�
P(y |�)P(�)
= argmin�
log |⌃y
|+ yT⌃�1
y
y � 2X
i
logP(�i
)
where, ⌃y
= �2I + ���T and � = diag(�)
Empirical Bayes
Choose P(�i
) to be a non-informative prior
Bhaskar D Rao University of California, San Diego
![Page 76: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/76.jpg)
Sparse Bayesian Learning
Computing Posterior
Now because of our convenient choice posterior can be easily computed,i.e, P(x |y ; �̂) = N(µ
x
,⌃x
) where,
µx
= E [x |y ; �̂] = �̂�T (�2I + ��̂�T )�1y
⌃x
= Cov [x |y ; �̂] = �̂� �̂�T (�2I + ��̂�T )�1��̂
Updating �
Using EM algorithm with a non informative prior over �, the update rulebecomes:
�i
µx
(i)2 + ⌃x
(i , i)
Bhaskar D Rao University of California, San Diego
![Page 77: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/77.jpg)
Sparse Bayesian Learning
Computing Posterior
Now because of our convenient choice posterior can be easily computed,i.e, P(x |y ; �̂) = N(µ
x
,⌃x
) where,
µx
= E [x |y ; �̂] = �̂�T (�2I + ��̂�T )�1y
⌃x
= Cov [x |y ; �̂] = �̂� �̂�T (�2I + ��̂�T )�1��̂
Updating �
Using EM algorithm with a non informative prior over �, the update rulebecomes:
�i
µx
(i)2 + ⌃x
(i , i)
Bhaskar D Rao University of California, San Diego
![Page 78: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/78.jpg)
SBL properties
Local minima are sparse, i.e. have at most N nonzero �i
Bayesian inference cost is generally much smoother than associatedMAP estimation. Fewer local minima.
In high signal to noise ratio, the global minima is the sparsestsolution. No structural problems.
Bhaskar D Rao University of California, San Diego
![Page 79: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/79.jpg)
SBL properties
Local minima are sparse, i.e. have at most N nonzero �i
Bayesian inference cost is generally much smoother than associatedMAP estimation. Fewer local minima.
In high signal to noise ratio, the global minima is the sparsestsolution. No structural problems.
Bhaskar D Rao University of California, San Diego
![Page 80: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/80.jpg)
SBL properties
Local minima are sparse, i.e. have at most N nonzero �i
Bayesian inference cost is generally much smoother than associatedMAP estimation. Fewer local minima.
In high signal to noise ratio, the global minima is the sparsestsolution. No structural problems.
Bhaskar D Rao University of California, San Diego
![Page 81: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/81.jpg)
SBL properties
Local minima are sparse, i.e. have at most N nonzero �i
Bayesian inference cost is generally much smoother than associatedMAP estimation. Fewer local minima.
In high signal to noise ratio, the global minima is the sparsestsolution. No structural problems.
Bhaskar D Rao University of California, San Diego
![Page 82: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/82.jpg)
Empirical Comparison
For each test case
1 Generate a random dictionary � with 50 rows and 250 columnsfrom the normal distribution and normalize each column to have2-norm of 1.
2 Select the support for the true sparse coe�cient vector x0
randomly.
3 Generate the non-zero components of x0
from the normaldistribution.
4 Compute signal, y = �x0
(Noiseless case).
5 Compare SBL with previous methods with regard to estimating x0
.
6 Average over 1000 independent trials.
Bhaskar D Rao University of California, San Diego
![Page 83: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/83.jpg)
Empirical Comparison: 1000 trials
Figure: Probability of Successful recovery vs Number of non zerocoe�cients
Bhaskar D Rao University of California, San Diego
![Page 84: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/84.jpg)
Empirical Comparison: Multiple Measurement Vectors(MMV)
Generate data matrix via Y = �X0
(noiseless), where:
1 X0
is 100-by-5 with random non-zero rows.
2 � is 50-by-100 with Gaussian iid entries.
Bhaskar D Rao University of California, San Diego
![Page 85: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/85.jpg)
Empirical Comparison: 1000 trials
Bhaskar D Rao University of California, San Diego
![Page 86: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/86.jpg)
Summary
Bayesian methods o↵er interesting and useful options to the SparseSignal Recovery problem
MAP estimation (Reweighted `2
/`1
algorithms)Sparse Bayesian Learning
Versatile and can be more easily employed in problems withstructure
Algorithms can often be justified by studying the resulting objectivefunctions.
Bhaskar D Rao University of California, San Diego
![Page 87: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/87.jpg)
Summary
Bayesian methods o↵er interesting and useful options to the SparseSignal Recovery problem
MAP estimation (Reweighted `2
/`1
algorithms)Sparse Bayesian Learning
Versatile and can be more easily employed in problems withstructure
Algorithms can often be justified by studying the resulting objectivefunctions.
Bhaskar D Rao University of California, San Diego
![Page 88: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/88.jpg)
Summary
Bayesian methods o↵er interesting and useful options to the SparseSignal Recovery problem
MAP estimation (Reweighted `2
/`1
algorithms)Sparse Bayesian Learning
Versatile and can be more easily employed in problems withstructure
Algorithms can often be justified by studying the resulting objectivefunctions.
Bhaskar D Rao University of California, San Diego
![Page 89: Bayesian Methods for Sparse Signal Recovery · Bayesian Methods for Sparse Signal Recovery Bhaskar D Rao1 University of California, San Diego ... Robust Linear Regression and Outlier](https://reader034.vdocuments.us/reader034/viewer/2022042418/5f34793c8bccd723615fe5a5/html5/thumbnails/89.jpg)
Summary
Bayesian methods o↵er interesting and useful options to the SparseSignal Recovery problem
MAP estimation (Reweighted `2
/`1
algorithms)Sparse Bayesian Learning
Versatile and can be more easily employed in problems withstructure
Algorithms can often be justified by studying the resulting objectivefunctions.
Bhaskar D Rao University of California, San Diego