eleg 867 - compressive sensing and sparse signal...

ELEG 867 - Compressive Sensing and SparseSignal Representations

Gonzalo R. ArceDepart. of Electrical and Computer Engineering

University of Delaware

Fall 2011

Compressive Sensing G. Arce Fall, 2011 1 / 42

Outline

Compressive sensing signal reconstruction

Geometry of CSReconstruction Algorithms

Matching PursuitOrthogonal Matching PursuitIterative Hard ThresholdingWeighted Median Regression

Compressive Sensing G. Arce Fall, 2011 2 / 42

Compressive Sensing Signal Reconstruction

Goal:Recover signalx from measurementsy

Problem:Random projectionΦ not full rank (ill-posed inverseproblem)

Solution:Exploit the sparse/compressible geometry of acquiredsignalx

y Φ x

Compressive Sensing G. Arce CS Signal Reconstruction Fall, 2011 3 / 42

Suppose there is aS-sparse solution toy = Φx

Combinatorial optimization problem

(P0) minx

‖x‖0 s.t. Φx = y

Convex optimization problem

(P1) minx

‖x‖1 s.t. Φx = y

If 2δ3s + δ2s ≤ 1, the solutions(P0) and(P1) are the same†.† E. Candes.”Compressive Sampling”. Proc. Intern. Congress of Math., China, April 2006.

Compressive Sensing G. Arce CS Signal Reconstruction Fall, 2011 4 / 42

The Geometry of CS

Sparse signals have few non-zero coefficients

1-sparse 2-sparse

R3 R3X 1

X 2

X 3

X 1

X 2

X 3

Compressive Sensing G. Arce Geometry of CS Fall, 2011 5 / 42

ℓ0 Recovery

Reconstruction should be

Consistent with the model:x as sparse as possible min‖x‖0

Consistent with the measurements:y = ΦxX 1

X 3

R3

X 2x

x : y = Φx

Compressive Sensing G. Arce Geometry of CS Fall, 2011 6 / 42

ℓ0 Recovery

minx

‖x‖0 s.t. Φx = y

‖x‖0 number of nonzero elements

Sparsest signal consistent with the measurementsy

Requires onlyM << N measurements

Combinatorial NP-hard problem: forx ∈ RN with sparsityS, the

complexity isO(NS)

Too slow for implementation

Compressive Sensing G. Arce Geometry of CS Fall, 2011 7 / 42

ℓ1 Recovery

A more realistic model

minx

‖x‖1 s.t. Φx = y

Theℓ1 norm also induces sparsity.

The constraint is given byy = Φx.

x : y = Φx

x

X 1

X 3

R3

X 2

Compressive Sensing G. Arce Geometry of CS Fall, 2011 8 / 42

Phase Transition Diagram

In theℓ1 minimization, there is a defined region on(M/N, S/M) whichensures successful recovery

0.1

0.2

0.3

0.4

0.5

0.1 0.2 0.3 0.4 0.5

0.6

0.6

0.6

0.7

0.7

0.7

0.8

0.8

0.8

0.9

0.9

0.9 1

1

1

0.1

0.2

0.3

0.4

0.5

δ = M/N is a normalized measure of the problem indeterminacy.

ρ = S/M is a normalized measure of the sparsity.

Red region- unsuccessful recovery or exact reconstruction typicallyfails.

Blue region- successful recovery or exact reconstruction typicallyoccurs.

Compressive Sensing G. Arce Geometry of CS Fall, 2011 9 / 42

ℓ2 Recovery

minx

‖x‖2 s.t. Φx = y

Least square solutionx = (ΦTΦ)−1ΦTy

Solved by using quadratic programming:X Least squares solutionX Interior-point methods

Solution is almost never sparse

X 2

R3

X 1

x

x'

X 3

x : y = Φx

Compressive Sensing G. Arce Geometry of CS Fall, 2011 10 / 42

ℓ2 Recovery

Problem:smallℓ2 does not imply sparsity

x

x'

x′ has smallℓ2 norm but it is not sparse

Compressive Sensing G. Arce Geometry of CS Fall, 2011 11 / 42

CS Example in the Time Domain

Random

Projection

Nonlinear

Reconstruction

min ||x||1 s.t.

Time domain sparse signal

Error signal

Φ y = x

Φ y = x

Compressive Sensing G. Arce Geometry of CS Fall, 2011 12 / 42

CS Example in the Wavelet Domain

Reconstruction of an image (N = 1 megapixel) sparse(S = 25, 000) in the wavelet domain fromM = 96, 000incoherent measurements.†

Original (25K wavelets) Recovered Image† E. J. Candes and J. Romberg ”Sparsity and Incoherence in Compressive Sampling.” Inverse Problems.

vol.23, pp.969-985. 2006.

Compressive Sensing G. Arce Geometry of CS Fall, 2011 13 / 42

Reconstruction Algorithms

Different formulations and implementations have been proposedto find the sparsestx subject toy = Φx

Difficult to compare results obtained by different methodsThose are broadly classified in:X Regularization formulations (Replace combinatorial problem with

convex optimization)X Greedy algorithms (Iterative refinement of a sparse solution)X Bayesian framework (Assume prior distribution of sparse

coefficients)

Compressive Sensing G. Arce Reconstruction Algorithms Fall, 2011 14 / 42

Greedy Algorithms

Iterative algorithms that select an optimal subset of the desired signalxat each iteration. Some examples:

Matching Pursuit (MP) minx ‖x‖0 s.t. Φx = y

Orthogonal Matching Pursuit(OMP) minx ‖x‖0 s.t. Φx = y

Compressive Sampling Matching Pursuit(CoSaMP) minx ‖y − Φx‖2

2 s.t. ‖x‖0 ≤ S

Iterative Hard Thresholding (ITH) minx ‖y − Φx‖22 s.t. ‖x‖0 ≤ S

Weighted Median Regression minx ‖y − Φx‖1 + λ‖x‖0

Those methods build an approximation of the sparse signal ateach iteration.

Compressive Sensing G. Arce Reconstruction Algorithms Fall, 2011 15 / 42

Matching Pursuit (MP)

Sparse approximation algorithm used to find the solution to

minx

‖x‖0 s.t. Φx = y

Key ideas:

Measurementsy = Φx are composed of sum ofS columns ofΦ.

The algorithm needs to identify whichS columns contribute toy.y x

S

Compressive Sensing G. Arce Matching Pursuit Fall, 2011 16 / 42

Matching Pursuit (MP) Algorithm

Input and Observation vectoryInitialize Measurement MatrixΦParameters Initial Solutionx(0) = 0

Initial residualr(0) = y

Iteration k

Compute the current error:c(k)j = 〈φj, r(k)〉

Identify the indexj such that:j = maxj |c(k)j |

Updatex(k)

j= x(k−1)

j+ c(k)

j

Updater(k) = r(k−1) − c(k)

jφj

D. Donoho et. al, SparseLab Version 2.0. 2007.

Compressive Sensing G. Arce Matching Pursuit Fall, 2011 17 / 42

MP Reconstruction Example

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

reconstruc. signal

Iteration 0

0 50 100 150−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

residue - Iteration 0

0 50 100 150 200 250 3000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

reconstruc. signal

Iteration 5

0 50 100 150−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

residue - Iteration 5

Compressive Sensing G. Arce Matching Pursuit Fall, 2011 18 / 42

MP Reconstruction Example

0 50 100 150 200 250 3000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0 50 100 150 200 250 3000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

reconstruc. signal

Iteration 32

Original Signal

0 50 100 150−6

−4

−2

0

2

4

6

8x 10

−5

Residue Iteration 32

Compressive Sensing G. Arce Matching Pursuit Fall, 2011 19 / 42

Orthogonal Matching Pursuit (MP)

Sparse approximation algorithm used to find the solution to

minx

‖x‖0 s.t. Φx = y

Key ideas:

Measurementsy = Φx are composed of sum ofS columns ofΦ.

The algorithm needs to identify whichS columns contribute toy.y x

S

Compressive Sensing G. Arce Orthogonal Matching Pursuit Fall, 2011 20 / 42

Orthogonal Matching Pursuit (OMP) Algorithm

Input and Observation vectoryInitialize Measurement MatrixΦParameters Initial Solutionx(0) = 0

Initial Solution SupportS(0)=support{x(0)} = 0Initial residualr(0) = y

Iteration k Compute the current error:c(k)j = 〈φj, r(k)〉

Identify the indexj such that:j = maxj |c(k)j |

Update the supportS(k)=S(k−1)∪jUpdate the matrixΦS(k) = [0, ..., φi, ..., 0, ..., φj, ..., 0]Update the solutionx(k) = (ΦT

S(k)ΦS(k))−1ΦTS(k)y

Updater(k) = y − ΦS(k) x(k)

D. Donoho et. al, SparseLab Version 2.0. 2007.

Compressive Sensing G. Arce Orthogonal Matching Pursuit Fall, 2011 21 / 42

Remarks on the OMP Algorithm

S(k) is the support ofx at the iterationk andΦS(k) is the matrix thatcontains the columns fromΦ that belongs to this support.

The updated solution gives thex(k) that solves the minimizationproblem‖ΦS(k)x − y‖2

2. The solution is given by

ΦTS(k)(ΦS(k)x − y) = 0

−ΦTS(k)r(k) = 0

This solution shows that the columns inΦTS(k) are orthogonal tor(k).

Compressive Sensing G. Arce Orthogonal Matching Pursuit Fall, 2011 22 / 42

OMP Reconstruction Example

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

0 50 100 150−5

−4

−3

−2

−1

0

1

2

3

4

5

reconstruc. signal

Iteration 0residue - Iteration 0

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

reconstruc. signal

Iteration 5

0 50 100 150−3

−2

−1

0

1

2

3

4

residue - Iteration 5

Compressive Sensing G. Arce Orthogonal Matching Pursuit Fall, 2011 23 / 42

OMP Reconstruction Example

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

0 50 100 150−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

reconstruc. signal

Iteration � residue - Iteration 9

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

0 50 100 150−1

−0.5

0

0.5

1

1.5x 10

−15

reconstruc. signal

Iteration 10residue - Iteration 10

Compressive Sensing G. Arce Orthogonal Matching Pursuit Fall, 2011 24 / 42

Iterative Hard Thresholding

Sparse approximation algorithm that solves the problem

minx

‖y − Φx‖22 + λ‖x‖0.

Iterative algorithm that computes a descent direction given by thegradient of theℓ2-norm.

The solution at each iteration is given by finding:

x(k+1) = Hλ(x(k) + ΦT(y − Φx(k)))

Hλ is a non-linear operator that only retains the coefficients withthe largest magnitude.

† T. Blumensath and M. Davies. ”Iterative Hard Thresholding for Compressed Sensing.” Journal of Appl. Comp. Harm.Anal. vol. 27, no. 3, pp. 265-274, 2009.

Compressive Sensing G. Arce Iterative Hard Thresholding Fall, 2011 25 / 42

Iterative Hard Thresholding

Input and Observation vectoryInitialize Measurement MatrixΦParameters x(0) = 0

Iteration kCompute an update:a(k+1) = x(k) +ΦT(y − Φx(k)))

Select the largest elements:x(k+1) = Hλ(a(k+1)).Pull other elements toward zero.

D. Donoho et. al, SparseLab Version 2.0. 2007.

Compressive Sensing G. Arce Iterative Hard Thresholding Fall, 2011 26 / 42

ITH Reconstruction Example

0 50 100 150 200 250 300−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

reconstruc. signal

Iteration 1

0 50 100 150−5

−4

−3

−2

−1

0

1

2

3

4

5

residue - Iteration 1

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

reconstruc. signal

Iteration 5

0 50 100 150−0.04

−0.03

−0.02

−0.01

0

0.01

0.02

0.03

0.04

0.05

residue - Iteration 5

Compressive Sensing G. Arce Iterative Hard Thresholding Fall, 2011 27 / 42

ITH Reconstruction Example

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

0 50 100 150−6

−4

−2

0

2

4

6

8x 10

−4

0 50 100 150 200 250 300−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Original Signal reconstruc. signal

Iteration 10

residue - Iteration 10

Compressive Sensing G. Arce Iterative Hard Thresholding Fall, 2011 28 / 42

Weighted Median Regression ReconstructionAlgorithm

When the random measurements are corrupted by noise the mostwidely used CS reconstruction algorithms solve the problem

minx

‖y − Φx‖22 + τ‖x‖1.

whereτ is the regularization parameter that balances the conflictingtask of minimizing theℓ2 norm while yielding a sparse solution.

As τ increases the solution is sparser.

As τ decreases the solution approximates to least square solution.

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 29 / 42

In the solution to the problem

minx

‖y − Φx‖22 + τ‖x‖1.

Theℓ2 term is the data fitting term, induced by the Gaussianassumption of the noise contamination.

Theℓ1 term is the sparsity promoting term.

However, theℓ2 term leads to the least square solution which is knownto be very sensitive to high-noise level and outliers present in thecontamination.

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 30 / 42

Example of signal reconstruction when outliers are presentin therandom projections

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 31 / 42

To mitigate the effect of the impulsive noise in the compressivemeasurements, a more robust norm for the data-fitting term is

minx

‖y − Φx‖1︸ ︷︷ ︸+ λ‖x‖0︸ ︷︷ ︸ (1)

Data fitting Sparsity

LAD offers robustness to a broad class of noise, in particular to heavy tailnoise.

Optimum under the maximum likelihood principle when the underlyingcontamination follows a Laplacian-distributed model.

† J. L. Paredes and G. Arce. ”Compressive Sensing Signal Reconstruction by Weighted Median Regression Estimate.”ICASSP 2010.

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 32 / 42

Drawbacks:

Solvingℓ0-LAD optimization isNp-hard problem.

Direct solution is unfeasible even for modest-sized signal.

To solve this multivariable minimization problem:

Solve a sequence of scalar minimization subproblems.

Each subproblem improves the estimate of the solution byminimizing along a selected coordinate with all other coordinatesfixed.

Closed form solution exists for the scalar minimization problem.

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 33 / 42

Signal reconstruction

x = minx

M∑

i=1

|(y − Φx)i|+ λ‖x‖0

The solution usingCoordinate Descent Methodis given by

xn = minxn

M∑

i=1

|yi −

N∑

j=1,j6=n

φi,jxj − φi,nxn|+ λ|xn|0 +

N∑

j=1,j6=n

λ|xj|0

xn = minxn

M∑

i=1

|φin||yi −

∑Nj=1,j6=n φi,jxj

φin− xn|

︸ ︷︷ ︸+ λ|xn|0︸ ︷︷ ︸

Weighted least absolute deviation Regulariz.

Q(xn)

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 34 / 42

The solution to the minimization problem is given by

xn = minxn

Q(xn) + λ|xn|0

xn = MEDIAN(|φin| ⋄yi −

∑Nj=1,j6=n φi,jxj

φin) |Mi=1; n = 1, 2, ...,N

At kth iteration:

x(k)n =

{xn, if Q(xn) + λ ≤ Q(0)0, otherwise

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 35 / 42

The iterative algorithm detects the non-zero entries of thesparse vectorby

Robust signal estimation: computing a rough estimate of thesignal that minimizes the LAD by the weighted median operator.

Basis selection problem: applying a hard-threshold operator onthe estimated values.

Removing the entry’s influence from the observation vector.

Repeating these steps again until a performance criterion isreached.

IterationOriginal Signal

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 36 / 42

Input and Observation vectoryInitialize Measurement MatrixΦParameters Number of IterationsK0

Iteration counter:k = 1Hard-thresholding value:λ = λi

Estimation atk = 1, x(0) = 0IterationStep A For then-th entry ofx, n = 1, 2, ...,N compute

xn = MEDIAN

{φin ⋄

yi−∑N

j=1;j 6=n φijxj

φin

}|ki=1

x(k)n =

{xn, if Q(xn) + λ ≤ Q(0)0, otherwise,

Step BUpdate the hard-thresholding parameter and the estimationof xx(k+1) = x(k)

λ = λiβk

Step CCheck stopping criteriumIf k ≤ K0 then setk = k + 1 and go to Step A; otherwise, end.

OutputRecovered sparse signalx

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 37 / 42

Example Signal Reconstruction from Projectionsin Non-Gaussian Noise

Original signal and

noisy projections

CoSaMP L1-Ls

Matching Pursuit Weighted Median

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 38 / 42

Performance Comparisons

Exact reconstruction for noiseless signals (normalized error is smallerthan 0.1% of the signal energy).

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 39 / 42

Performance Comparisons

SNR vs. Sparsity for different types of noise

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 40 / 42

Convex Relaxation Approach:

Theℓ0-LAD problem can be relaxed by solving the following problem

minx

‖y − Φx‖1︸ ︷︷ ︸+ λ‖x‖1︸ ︷︷ ︸ . (2)

Data fitting Sparsity

This problem is convex and can be solved by linear programming.

ℓ1-norm may not induce the necessary sparsity in the solution.

It requires more measurements to achieve a target reconstructionerror.

Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 41 / 42

The solution for the problem

x = minx

M∑

i=1

|yi − Φix|+ λ‖x‖1

is given by

x = MEDIAN(λ ⋄ 0, |φ1| ⋄y1

φ1, ..., |φM| ⋄

yM

φM).

The regularization process induced by theℓ1-term leads to aweighting operation with the zero-valued sample in the WMoperation.

Large values of the regularization parameter implies largevaluefor the weight corresponding to the zero-valued sample (thisfavors sparsity).

The solution is biased.Compressive Sensing G. Arce Weighted Median Regression Fall, 2011 42 / 42