Matrix Extensions to Sparse Recovery

Yi Ma1,2 Allen Yang3 John Wright1

CVPR Tutorial, June 20, 2009

1Microsoft Research Asia

3University of California Berkeley

2University of Illinois at Urbana-Champaign

FINAL TOPIC – Generalizations: sparsity to degeneracy

The tools and phenomena underlying sparse recovery generalize very nicely to low-rank matrix recovery

???

FINAL TOPIC – Generalizations: sparsity to degeneracy

The tools and phenomena underlying sparse recovery generalize very nicely to low-rank matrix recovery

Matrix completion: Given an incomplete subset of the entries of a low-rank matrix, fill in the missing values.

Robust PCA: Given a low-rank matrix which has been grossly corrupted, recover the original matrix.

???

Face images

Degeneracy: illumination models

Errors: occlusion, corruption

Relevancy data

Degeneracy: user preferences co-predict

Errors: Missing rankings, manipulation

Video

Degeneracy: temporal, dynamic structures

Errors: anomalous events, mismatches…

Examples of degenerate data:

THIS TALK – From sparse recovery to low-rank recovery

KEY ANALOGY – Connections between rank and sparsity

Sparse recovery Rank minimization

Unknown Vector x Matrix A

Observations y = Ax y = L[A] (linear map)

Combinatorial objective

Convex relaxation

Algorithmic tools

Linear programming Semidefinite programming

KEY ANALOGY – Connections between rank and sparsity

This talk: exploiting this connection for matrix completion and RPCA

Sparse recovery Rank minimization

Unknown Vector x Matrix A

Observations y = Ax y = L[A] (linear map)

Combinatorial objective

Convex relaxation

Algorithmic tools

Linear programming Semidefinite programming

CLASSICAL PCA – Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

CLASSICAL PCA – Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

Principal Component Analysis via singular value decomposition:

• Stable, efficient computation

• Optimal estimate of under iid Gaussian noise

• Fundamental statistical tool, huge impact in vision, search, bioinformatics

CLASSICAL PCA – Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

But… PCA breaks down under even a single corrupted observation.

Principal Component Analysis via singular value decomposition:

• Stable, efficient computation

• Optimal estimate of under iid Gaussian noise

• Fundamental statistical tool, huge impact in vision, search, bioinformatics

ROBUST PCA – Problem formulation

……

D - observation A – low-rank E – sparse error

…

Properties of the errors:

• Each multivariate data sample (column) may be corrupted in some entries

• Corruption can be arbitrarily large in magnitude (not Gaussian!)

Problem: Given recover .A0

Low-rank structure Sparse errors

ROBUST PCA – Problem formulation

Problem: Given recover .A0

Low-rank structure Sparse errors

Numerous heuristic methods in the literature:

• Random sampling [Fischler and Bolles ‘81] • Multivariate trimming [Gnanadesikan and Kettering ‘72] • Alternating minimization [Ke and Kanade ‘03] • Influence functions [de la Torre and Black ‘03]

•

No polynomial-time algorithm with strong performance guarantees!

……

D - observation A – low-rank E – sparse error

…

ROBUST PCA – Semidefinite programming formulation

Seek the lowest-rank that agrees with the data up to some sparse error:A

ROBUST PCA – Semidefinite programming formulation

Seek the lowest-rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

A

ROBUST PCA – Semidefinite programming formulation

Seek the lowest-rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

A

Semidefinite program, solvable in polynomial time

Convex envelope over

MATRIX COMPLETION – Motivation for the nuclear norm

Related problem: we observe only a small known subset

of entries of a rank- matrix . Can we exactly recover ?

MATRIX COMPLETION – Motivation for the nuclear norm

Related problem: recover a rank matrix from a known subset of entries

Convex optimization heuristic [Candes and Recht] :

Spectral trimming also succeeds with for

For incoherent , exact recovery with

[Keshavan, Montanari and Oh]

[Candes and Tao]

ROBUST PCA – Exact recovery?

CONJECTURE: If with sufficiently low-rank and

exactly recovers .

Sparsity of error

sufficiently sparse, then solving

Empirical evidence: probability of correct recovery vs rank and sparsity

Perfect recovery

Rank

Decompose as or ?

ROBUST PCA – Which matrices and which errors?

Fundamental ambiguity – very sparse matrices are also low-rank:

rank-1 rank-00-sparse 1-sparse

Obviously we can only hope to uniquely recover that are incoherent with the standard basis.

Can we recover almost all low-rank matrices from almost all sparse errors?

ROBUST PCA – Which matrices and which errors?

Random orthogonal model (of rank r) [Candes & Recht ‘08]:

independent samples from invariant measure on Steifel manifold of orthobases of rank r.

arbitrary.

ROBUST PCA – Which matrices and which errors?

Random orthogonal model (of rank r) [Candes & Recht ‘08]:

independent samples from invariant measure on Steifel manifold of orthobases of rank r.

arbitrary.

Bernoulli error signs-and-support (with parameter ):

Magnitude of is arbitrary.

MAIN RESULT – Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from errors affecting of the observations!”

BONUS RESULT – Matrix completion in proportional growth

“Convex optimization exactly recovers matrices of rank , even with entries missing!”

MATRIX COMPLETION – Contrast with literature

• [Candes and Tao 2009]:

Correct completion whp for

Does not apply to the large-rank case

• This work:

Correct completion whp for even with

Proof exploits rich regularity and independence in random orthogonal model.

Caveats:

- [C-T ‘09] tighter for small r. - [C-T ‘09] generalizes better to other matrix ensembles.

MAIN RESULT – Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from errors affecting of the observations!”

ROBUST PCA – Solving the convex program

Semidefinite program in millions of unknowns.

Scalable solution: apply a first-order method with convergence to

Sequence of quadratic approximations [Nesterov, Beck & Teboulle]:

Solved via soft thresholding (E), and singular value thresholding (A).

ROBUST PCA – Solving the convex program

• Iteration complexity for suboptimal solution.

• Dramatic practical gains from continuation

SIMULATION – Recovery in various growth scenarios

Correct recovery with and fixed, increasing.

Empirically, almost constant number of iterations:

Provably robust PCA at only a constant factor more computation than conventional PCA.

SIMULATION – Phase Transition in Rank and Sparsity

Fraction of successes with , varying (10 trials)

Fraction of successes with , varying (65 trials)

[0,.5] x [0,.5][0,1] x [0,1]

[0,1] x [0,1] [0,.4] x [0,.4]

EXAMPLE – Background modeling from video

Video Low-rank appx. Sparse errorStatic camera surveillance video

200 frames, 72 x 88 pixels,

Significant foregroundmotion

EXAMPLE – Background modeling from video

Video Low-rank appx. Sparse errorStatic camera surveillance video

550 frames, 64 x 80 pixels,

significant illuminationvariation

Background variation

Anomalous activity

EXAMPLE – Faces under varying illumination

…

…RPCA

29 images of one person under varying lighting:

EXAMPLE – Faces under varying illumination

…

…RPCA

29 images of one person under varying lighting:

Self- shadowing

Specularity

EXAMPLE – Face tracking and alignment

Initial alignment, inappropriate for recognition:

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

EXAMPLE – Face tracking and alignment

Final result: per-pixel alignment

EXAMPLE – Face tracking and alignment

Final result: per-pixel alignment

SIMULATION – Phase Transition in Rank and Sparsity

Fraction of successes with , varying (10 trials)

Fraction of successes with , varying (65 trials)

[0,.5] x [0,.5][0,1] x [0,1]

[0,1] x [0,1] [0,.4] x [0,.4]

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

Hypothesized breakdown behavior as m ∞

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

What we know so far:

This work

Classical PCA

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

CONJECTURE I: convex programming succeeds in proportional growth

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

CONJECTURE II: for small ranks ,

any fraction of errors can eventually be corrected.

Similar to Dense Error Correction via L1 Minimization, Wright and Ma ‘08

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

CONJECTURE III: for any rank fraction, ,

there exists a nonzero fraction of errors that can eventually be

corrected with high probability.

CONJECTURES – Phase Transition in Rank and Sparsity

1

100

CONJECTURE IV: there is an asymptotically sharp phase transition

between correct recovery with overwhelming probability, and

failure with overwhelming probability.

CONJECTURES – Connections to Matrix Completion

Our results also suggest the possibility of a proportional growth phase transition for matrix completion.

1

100

• How do the two breakdown points compare?

• How much is gained by knowing the location of the corruption?

Robust PCA

Matrix Completion

Similar to Recht, Xu and Hassibi ‘08

Matrix CompletionRobust PCA

FUTURE WORK – Stronger results on RPCA?

• RPCA with noise and errors:

Tradeoff between estimation error and robustness to corruption?

• Deterministic conditions on the matrix

• Simultaneous error correction and matrix completion:

bounded noise(e.g., Gaussian)

Conjecture: stable recovery with

we observe

• Faster algorithms:

Smarter continuation strategies

Parallel implementations, GPU, multi-machine

• Further applications:

Computer vision: photometric stereo, tracking, video repair

Relevancy data: search, ranking and collaborative filtering

Bioinformatics

System Identification

FUTURE WORK – Algorithms and Applications

• Reference: Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices by Convex Optimization submitted to the Journal of the ACM

• Collaborators: Prof. Yi Ma (UIUC, MSRA)

Dr. Zhouchen Lin (MSRA)

Dr. Shankar Rao (UIUC)

Arvind Ganesh (UIUC)

Yigang Peng (MSRA)

• Funding:

Microsoft Research Fellowship (sponsored by Live Labs)Grants NSF CRS-EHS-0509151, NSF CCF-TF-0514955, ONR YIP N00014-04-1-0633, NSF IIS 07-03756

REFERENCES + ACKNOWLEDGEMENT

Questions, please?

THANK YOU!

John Wright Robust PCA: Exact Recovery of Corrupted Low-Rank Matrices