stereo matching with nonparametric smoothness priors in...

Stereo Matching with Nonparametric Smoothness Priors in Feature Space

Brandon M. Smith1, Li Zhang1, and Hailin Jin2

1 University of Wisconsin – Madison

2 Adobe Systems Incorporated1

Motivation

State-of-the-art two-view stereo methods9 out of top 10 employ image segmentation

UW-Madison Computer Graphics and Vision Group2

UW-Madison Computer Graphics and Vision Group

Motivation

Image segmentation problems

3

UW-Madison Computer Graphics and Vision Group

Motivation

Segmentation artifacts in video: temporal instability

1 of 5 input views 2nd-order smoothness method with segmentation[Woodford et al. CVPR ’08]

4

UW-Madison Computer Graphics and Vision Group

Inspiration: Adaptive Support Weight

Yoon & Kweon, Locally adapt. support-weight approach for vis. corr. search, CVPR ‘05

Intensity-encoded weightsClose-up views of matching window

5

UW-Madison Computer Graphics and Vision Group

Inspiration: Adaptive Support Weight

Can we incorporate this idea into a global inference algorithm?

Large, weighted smoothness nbrhoodClose-up views of matching window

6

UW-Madison Computer Graphics and Vision Group

Inspiration: Sparse Neighborhoods

O. Veksler, Stereo Correspondence by Dynamic Programming on a Tree, CVPR ‘05

Image close-up views Common smoothness neighborhood structure

Sparse tree smoothness neighborhood structure

Minimum spanning tree

algorithm

7

UW-Madison Computer Graphics and Vision Group

Our Approach

Minimum spanning tree

algorithm

Most important edgesLarge, weighted smoothness nbrhood

Global inference using large, sparse smoothness neighborhoods

8

Problem Formulation

UW-Madison Computer Graphics and Vision Group

2sm1sm21ph21 ,, DDDDDD

by minimizing:

compute disparity maps, and1D 2D

Given a stereo image pair, and , 1I 2I

9

Energy Minimization Function

2sm1sm21ph21 ,, DDDDDD

UW-Madison Computer Graphics and Vision Group

Photo consistency term[Kolmogorov & Zabih ECCV ‘02]

21ph , DD

Smoothness (regularization) terms

2sm1sm DD

10

UW-Madison Computer Graphics and Vision Group

Spatial Smoothness Terms

21ph21 ,, DDDD 2sm1sm DD

• 2nd-order smoothness priors

Previous global methods:• 1st-order smoothness priors

Another approach:• Kernel density estimation

Large neighborhood 11

Large neighborhood UW-Madison Computer Graphics and Vision Group

Kernel Density Estimation

dg

dd

d

qp

Nq p

p,qw

g(x)

x

p||

1 qqgcx

cc

c

xx

xNpp g

qp

Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]

21ph21 ,, DDDD 2sm1sm DD

12

Kernel function for disparity

Large neighborhood UW-Madison Computer Graphics and Vision Group

Upperbound Approximation

Nq p

p,qw

qp

21ph21 ,, DDDD 2sm1sm DD

dg

dd

d

qp

p||

1 qqgcx

cc

c

xx

xNp

g(x)

x

p g

Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]

Ip

Dsm

13

Kernel function for disparity

min( λ |dp dq|, τ)

Truncated linear difference function

UW-Madison Computer Graphics and Vision Group

Optimization Challenge

21ph21 ,, DDDD 2sm1sm DDsm is dense, expensive to minimize

p,qw

Nq pIp

Dsm

min( λ |dp dq|, τ)

Solution: use a sparse approximation

Large neighborhood 14

UW-Madison Computer Graphics and Vision Group

Sparse Graph Approximation

p,qw

Nq pIp

Dsm

min( λ |dp dq|, τ)

sm is dense, expensive to minimize

Solution: use a sparse graph approximation

DenseGraph

SparseGraph

15

UW-Madison Computer Graphics and Vision Group

Sparse Graph Approximation

DenseGraph

SparseGraph

1st Spanning Tree

ResidualGraph

2st Spanning Tree Nth Spanning Tree…

16

UW-Madison Computer Graphics and Vision Group

Dense smoothness neighborhood

Minimum spanning tree

algorithm

Sparse smoothness neighborhood

Graph Edges On Real Images

17

UW-Madison Computer Graphics and Vision Group

0% pixels connected 82% pixels connected 95% pixels connected 98% pixels connected 100% pixels connected

Connection to Image Segmentation

Kruskal’s minimum spanning tree algorithm

C. Zahn. Graph-theoretic methods for detecting and describinggestalt clusters. IEEE Trans. on Computing, 1971.

18

Results on Stereo Images

UW-Madison Computer Graphics and Vision Group

Left input image

Ground truth

Multi-view graph cuts result[Kolmogorov & Zabih, ‘02]

4.82% bad pixels

Our result

3.41% bad pixels 25.06% bad pixels

Second-order smoothness[Woodford et al. ’08]

19

UW-Madison Computer Graphics and Vision Group

2.48% bad pixels 4.21% bad pixels

Klaus et al. ‘06 results Multi-view graph cuts[Kolmogrov & Zabih ‘02]

(tailored parameters)

Left input image Ground truth depth

Results on Stereo Images

20

3.29% bad pixels

Our results(tailored parameters)

Results on Videos

UW-Madison Computer Graphics and Vision Group21

Future Work

UW-Madison Computer Graphics and Vision Group22

• Automatic parameter estimation (scale in kernel function)

• generalizes to any feature vector (not just x,y,r,g,b) explore other feature vectors

p,qw

• Better handle View-dependent brightness inconsistencies