AMSC 663 1

Sparse Solutions of Linear Systems of Equations and Sparse Modeling of

Signals and Images

Alfredo Nava-Tudela

John J. Benedetto, advisor

AMSC 663 2

Outline

- Problem: Find “sparse solutions” of Ax = b- Why do we care?- An application in signal processing: compression- Working definition of “sparse solution”- Why things eventually work, for some cases- How do we find sparse solutions? OMP algorithm- Case study: What happens when we combine the

DCT and DWT?- Conclusions/Recap- Project timeline

AMSC 663 3

ProblemLet A be an n by m matrix with n < m, and rank(A)=n.We want to solve

Ax = b, where b is a data or signal vector,

and x is the solution with the fewest number of non-zero entries possible, that is, the “sparsest” one.

Observations:- A is underdetermined and, since rank(A)=n,there is an infinite number of solutions. Good!- How do we find the “sparsest” solution? What does this mean in practice? Is there a unique sparsest solution?

AMSC 663 4

But, why do we care?

QuickTime™ and a decompressor

are needed to see this picture.

231 kb, uncompressed, 320x240x3x8 bit

QuickTime™ and a decompressor

are needed to see this picture.

74 kb, compressed 3.24:1JPEG

AMSC 663 5

But, why do we care?

QuickTime™ and a decompressor

are needed to see this picture.

512 x 512 Pixels,24-Bit RGB,

Size 786 Kbyte

QuickTime™ and a decompressor

are needed to see this picture.

75:1, 10.6 KbyteJPEG2000

AMSC 663 6

“Sparsity” equals compression

Both JPEG and JPEG2000 achieve their compressionmainly because at their core one finds a linear transform(DCT and DWT, respectively) that reduces the numberof non-zero entries required to represent the data, withinan acceptable error.

We can then think of signal compression in terms of ourproblem

Ax = b, x is sparse, b is dense, store x!

AMSC 663 7

Working definitions of “sparse”

- Convenient to introduce the l0 “norm”:

||x||0 = # {k : xk ≠ 0}

- (P0): minx ||x||0 subject to ||Ax - b||2 = 0

- (P0e): minx ||x||0 subject to ||Ax - b||2 < e

Observation: Solving (P0) is NP-hard, bummer.

AMSC 663 8

Some theoretical results

Definition: The spark of a matrix A is the minimumnumber of linearly dependent columns of A. We writespark(A) to represent this number.

Theorem: If there is a solution x to Ax = b, and||x||0 < spark(A) / 2, then x is the sparsest solution.That is, if y ≠ x also solves the equation, then||x||0 < ||y||0.

Observation: Computing spark(A) is combinatorial,therefore hard. Alternative?

AMSC 663 9

More theoretical resultsDefinition: The mutual coherence of a matrix A is thenumber

Lemma: spark(A) ≥ 1+1/mu(A).

Theorem: If x solves Ax = b, and ||x||0 < (1+mu(A)-1)/2,then x is the sparsest solution, as before.

Observation: mu(A) is a lot easier and faster to compute,but 1+1/mu(A) far worse bound than spark(A), in general.

AMSC 663 10

Finding sparse solutions:OMPOrthogonal Matching Pursuit algorithm:

AMSC 663 11

Revisiting compressionPropose to study the compression properties of thematrix

A = [DCT,DWT]

and compare it with the compression properties of DCT or DWT alone.

Study the behavior of OMP for this problem.

Many wavelet options available to try, e.g., reversible5/3 or floating-point 9/7 Daubechies, as in JPEG2000.

Interested in compression vs error graph properties.

AMSC 663 12

Conclusions/Recapitulation- Finding sparse solutions to the linear system ofequations Ax = b, when A is an n by m full rank matrix and n < m, is of interest to the signal processing community.

- There are simple criteria to assert the uniqueness of a given sparse solution.

- There are algorithms to find sparse solutions, e.g.,OMP; and their convergence can be guaranteed when there are “sufficiently sparse” solutions.

- Studies on the performance of OMP missing when A is the concatenation of unitary matrices.

AMSC 663 13

Project timeline- Oct. 15, 2010: Complete project proposal.- Oct. 18 - Nov. 5: Implement OMP.- Nov. 8 - Nov. 26: Validate OMP.- Nov. 29 - Dec. 3: Write mid-year report.- Dec. 6 - Dec. 10: Prepare mid-year oral presentation.- Some time after that, give mid-year oral presentation.- Jan. 24 - Feb. 11: Testing of OMP. Reproduce paper results.- Jan. 14 - Apr. 8: Testing of OMP on A = [DCT,DWT].- Apr. 11 - Apr. 22: Write final report.- Apr. 25 - Apr. 29: Prepare final oral presentation.- Some time after that, give final oral presentation.

AMSC 663 14

ReferencesA. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), pp. 34–81.

S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, 1998.

B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM Journal on Computing, 24 (1995), pp. 227-234.

G. W. Stewart, Introduction to Matrix Computations, Academic Press, 1973.

D. S. Taubman and M. W. Mercellin, JPEG 2000: Image Compression Funda- mentals, Standards and Practice, Kluwer Academic Publishers, 2001.

G. K. Wallace, The JPEG still picture compression standard, Communications of the ACM, 34 (1991), pp. 30-44.

http://www.stanford.edu/class/ee398a/handouts/lectures/08-JPEG.pdf