stereo matching with nonparametric smoothness priors in...
TRANSCRIPT
Stereo Matching with Nonparametric Smoothness Priors in Feature Space
Brandon M. Smith1, Li Zhang1, and Hailin Jin2
1 University of Wisconsin – Madison
2 Adobe Systems Incorporated1
Motivation
State-of-the-art two-view stereo methods9 out of top 10 employ image segmentation
UW-Madison Computer Graphics and Vision Group2
UW-Madison Computer Graphics and Vision Group
Motivation
Image segmentation problems
3
UW-Madison Computer Graphics and Vision Group
Motivation
Segmentation artifacts in video: temporal instability
1 of 5 input views 2nd-order smoothness method with segmentation[Woodford et al. CVPR ’08]
4
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Yoon & Kweon, Locally adapt. support-weight approach for vis. corr. search, CVPR ‘05
Intensity-encoded weightsClose-up views of matching window
5
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Can we incorporate this idea into a global inference algorithm?
Large, weighted smoothness nbrhoodClose-up views of matching window
6
UW-Madison Computer Graphics and Vision Group
Inspiration: Sparse Neighborhoods
O. Veksler, Stereo Correspondence by Dynamic Programming on a Tree, CVPR ‘05
Image close-up views Common smoothness neighborhood structure
Sparse tree smoothness neighborhood structure
Minimum spanning tree
algorithm
7
UW-Madison Computer Graphics and Vision Group
Our Approach
Minimum spanning tree
algorithm
Most important edgesLarge, weighted smoothness nbrhood
Global inference using large, sparse smoothness neighborhoods
8
Problem Formulation
UW-Madison Computer Graphics and Vision Group
2sm1sm21ph21 ,, DDDDDD
by minimizing:
compute disparity maps, and1D 2D
Given a stereo image pair, and , 1I 2I
9
Energy Minimization Function
2sm1sm21ph21 ,, DDDDDD
UW-Madison Computer Graphics and Vision Group
Photo consistency term[Kolmogorov & Zabih ECCV ‘02]
21ph , DD
Smoothness (regularization) terms
2sm1sm DD
10
UW-Madison Computer Graphics and Vision Group
Spatial Smoothness Terms
21ph21 ,, DDDD 2sm1sm DD
• 2nd-order smoothness priors
Previous global methods:• 1st-order smoothness priors
Another approach:• Kernel density estimation
Large neighborhood 11
Large neighborhood UW-Madison Computer Graphics and Vision Group
Kernel Density Estimation
dg
dd
d
qp
Nq p
p,qw
g(x)
x
p||
1 qqgcx
cc
c
xx
xNpp g
qp
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
21ph21 ,, DDDD 2sm1sm DD
12
Kernel function for disparity
Large neighborhood UW-Madison Computer Graphics and Vision Group
Upperbound Approximation
Nq p
p,qw
qp
21ph21 ,, DDDD 2sm1sm DD
dg
dd
d
qp
p||
1 qqgcx
cc
c
xx
xNp
g(x)
x
p g
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
Ip
Dsm
13
Kernel function for disparity
min( λ |dp dq|, τ)
Truncated linear difference function
UW-Madison Computer Graphics and Vision Group
Optimization Challenge
21ph21 ,, DDDD 2sm1sm DDsm is dense, expensive to minimize
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
Solution: use a sparse approximation
Large neighborhood 14
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
sm is dense, expensive to minimize
Solution: use a sparse graph approximation
DenseGraph
SparseGraph
15
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
DenseGraph
SparseGraph
1st Spanning Tree
ResidualGraph
2st Spanning Tree Nth Spanning Tree…
16
UW-Madison Computer Graphics and Vision Group
Dense smoothness neighborhood
Minimum spanning tree
algorithm
Sparse smoothness neighborhood
Graph Edges On Real Images
17
UW-Madison Computer Graphics and Vision Group
0% pixels connected 82% pixels connected 95% pixels connected 98% pixels connected 100% pixels connected
Connection to Image Segmentation
Kruskal’s minimum spanning tree algorithm
C. Zahn. Graph-theoretic methods for detecting and describinggestalt clusters. IEEE Trans. on Computing, 1971.
18
Results on Stereo Images
UW-Madison Computer Graphics and Vision Group
Left input image
Ground truth
Multi-view graph cuts result[Kolmogorov & Zabih, ‘02]
4.82% bad pixels
Our result
3.41% bad pixels 25.06% bad pixels
Second-order smoothness[Woodford et al. ’08]
19
UW-Madison Computer Graphics and Vision Group
2.48% bad pixels 4.21% bad pixels
Klaus et al. ‘06 results Multi-view graph cuts[Kolmogrov & Zabih ‘02]
(tailored parameters)
Left input image Ground truth depth
Results on Stereo Images
20
3.29% bad pixels
Our results(tailored parameters)
Results on Videos
UW-Madison Computer Graphics and Vision Group21
Future Work
UW-Madison Computer Graphics and Vision Group22
• Automatic parameter estimation (scale in kernel function)
• generalizes to any feature vector (not just x,y,r,g,b) explore other feature vectors
p,qw
• Better handle View-dependent brightness inconsistencies