Depth Enhancement via Low-rank Matrix Completion Si Lu1, Xiaofeng Ren2, and Feng Liu1

Department of Computer Science, Portland State University1 Department of Computer Science and Engineering, University of Washington2

Depth maps captured by consumer RGB-D cameras are often noisy and miss values at some pixels. This paper presents a depth enhancement algorithm via low rank matrix completion that performs depth map completion and de-noising simultaneously.

SUMMARY

EXPERIMENTS

This work was supported in part by NSF grants IIS-1321119, CNS-1205746, and CNS-1218589.

FRAMEWORK

OBSERVATIONS: Similar RGB-D patches

approximately lie in a low-dimensional subspace.

The subspace constraint essentially captures the potentially scene-dependent image structures in the RGB-D patches in both the color and depth domain.

This low-rank subspace constraint can be enforced through incomplete matrix factorization.

Patch samples. (a): clean patch. (b): noisy patch. (c): the top ten eigen-vectors of the patch matrix formed by similar patches to each noisy patch.

Similar patch searching Rank prediction for patch matrix

Enhancement via low-rank matrix completion

Output color Output depth

Input color Input depth

Output color

Output depth

Input color Input depth

Output color

Output depth

Input color Input depth

Output color

Output depth

Features capturing

patch structure properties

Predicted rank

Training data

Noisy color Color edge

Depth edgeNoisy depth

(a) Ground truth (b) Noisy RGB-D image BM3D Our method

(c) Color denoising

(d) Joint bilateral filter (e) BM3D + joint bilateral filter

(f) Our depth enhancement result

Rank distribution of 30,000 RGB-D patch matrices

(a) (b) (c) (a) (b) (c) (a) (b) (c)

�̂�=𝐴𝐵

Regression model

Comparisons among depth enhancement methods.

patch matrix M matrix rank r