global methods 1 michael bleyer lva stereo vision
TRANSCRIPT
Global Methods 1
Michael BleyerLVA Stereo Vision
What happened last time? Local methods Pros and cons Adaptive windows Slanted surfaces Occlusion handling in local stereo
Outline Stereo as an Energy Minimization Problem Dynamic programming (DP)
• Basic Algorithm
• DP Algorithms- Scanline Optimization- Tree DP- Semi global matching- Simple Tree Method
Michael BleyerLVA Stereo Vision
Stereo as an Energy Minimization
Problem
Stereo as an Energy Minimization Problem Define an energy/cost function to measure the quality
of a disparity map:• High energy means that the disparity map is bad.
• Low energy means it is good.
Energy function is typically in the form of:
where- D is the disparity map of the left image- Edata measures photo consistency- Esmooth measures smoothness
Global methods express smoothness assumption in an explicit form (as a smoothness term).
Let us take a closer look at Edata and Esmooth.
)()()( DEDEDE smoothdata
The Data Term Measures the color dissimilarity for each pixel p of the
left image I:
where • dp is the disparity of p in the disparity map D
• m() is a function computing the color dissimilarity between pixels of left and right images.
),()(
Ippdata dppmDE
The Smoothness Term The smoothness assumption states that neighboring
pixels should be assigned to the same (or similar) disparities:
Nodes correspond to pixels of the left image
The Smoothness Term The smoothness assumption states that neighboring
pixels should be assigned to the same (or similar) disparities:
Edges represent interactions between pixels.
In our case, interactions occur between a pixel and its 4
spatial neighbours.
We state that the pixel should have the same disparity with
its neighbours
The Smoothness Term Let us write this idea as a term:
where• N is the set of all spatial neighbouring pixels in the left image.
• s() is a smoothness function that imposes a penalty if two disparities are different from each other.
• Let us use the following smoothness function:
• P is a user-defined penalty that balances data and smoothness terms
),()(,
pddsDE pNqp
smooth
),( qp dds0 if dp = dq P otherwise
The Smoothness Term Let us write this idea as a term:
where• N is the set of all spatial neighbouring pixels in the left image.
• s() is a smoothness function that imposes a penalty if two disparities are different from each other.
• Let us use the following smoothness function:
• P is a user-defined penalty that balances data and smoothness terms
),()(,
pddsDE pNqp
smooth
),( qp dds0 if dp = dq P otherwise
This particular smoothness function is called the Potts model
Balancing Data and Smoothness Terms
P = 0 P = 5 P = 10
P = 30 P = 50 P = 5000
Disparity maps generated by energy optimization via alpha-expansion algorithm (no global optimum)
The Smoothness Term Our smoothness term defines the following set of
(smoothness) interactions:
This is called the 4-connected grid.
Optimizing the 4-connected grid We are looking for a disparity map D
that has the minimum energy E(D) among all possible disparity maps.
This is a very difficult problem:• np-complete problem in the general case
=> It is not possible to compute the optimal disparity map in reasonable time (most likely).
Why is it difficult:• A pixel has influence on all other pixels in
the image: - Changing the disparity of the pixel in the
top-left corner might change the disparity of the pixel in the bottom left corner.
Optimizing the 4-connected grid We are looking for a disparity map D
that has the minimum energy E(D) among all possible disparity map.
In general, this is a very difficult problem:• np-complete problem in general => It is
not possible to compute the optimal disparity map in reasonable time.
Why is it difficult:• Every pixel is connected to every other
pixel.
• Changing the disparity of the pixel in the top-left corner might change the disparity of the pixel in the bottom left corner.
We will spend two sessions on optimization algorithms.
We will learn about the following algorithms:
• Dynamic Programming (This session)
• Belief Propagation (Next session)
• Graph-Cuts (Next session)
Application to Other CV Problems
Our energy function measures the quality of an assignment of pixels to labels.
In our case, labels correspond to disparities. However, labels can have a different meaning => We
can use energy minimization approaches to solve a lot of other computer vision problems.
15
Optical Flow (Very Similar to Stereo) Input:
• 2 consecutive frames of a video
Desired output:• Map of 2D vectors representing the
movement of each pixel
Labels:• All allowed 2D displacement vectors
Data Term:• Color dissimilarity between
corresponding pixels
Smoothness term:• Penalty if neighbouring pixels have
different 2D vectors16
Image Denoising Input:
• Noisy image
Desired output:• Noise-free image
Labels:• 255 intensity values
Data Term:• Dissimilarity between pixel’s intensity
and assigned intensity
Smoothness term:• Penalty if neighbouring pixels are
assigned to different intensities.
17
Inpainting Input:
• Image with partially missing information (red rectangle)
Desired output:• Complete image
Labels:• 255 intensity values
Data Term:• 0 for each label assignment
Smoothness term:• Penalty if neighbouring pixels are
assigned to different intensities.
18
Interactive Image Segmentation Input:
• Color Image
• Foreground and background scribbles provided by user.
Desired output:• Binary map
- Label 0: Pixel belongs to background- Label 1: Pixel belongs to foreground
Data Term:• Dissimilarity between a pixel’s color
and the color models of fore-/background.
Smoothness term:• Penalty on 0/1 label transitions
19
Interactive Image Segmentation Input:
• Color Image
• Foreground and background scribbles provided by user.
Desired output:• Binary map
- Label 0: Pixel belongs to background- Label 1: Pixel belongs to foreground
Data Term:• Dissimilarity between a pixel’s color
and the color models of fore-/background.
Smoothness term:• Penalty on 0/1 label transitions
20
There are many more computer vision
problems that can be modelled by our energy function.
Generality of energy functions Apart from smoothness, we can model other
assumptions in the energy function. Some examples for stereo:
• Energy gives infinite costs if uniqueness assumption is violated.
• Energy is lower if disparity borders coincide with intensity edges.
In general, if you have a computer vision problem:• Think about what a perfect solution should look like.
• Try to express the properties of this perfect solution as an energy function.
• Apply one of many existing optimization algorithm to find the solution that minimizes your energy function.
• ~30% of vision papers work like this.21
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem.- Ideally, the correct solution should have lower energy than all
other possible solutions.
• Energy minimization:- Your optimization algorithm delivers a solution that is far off from
the exact minimum of your energy.
Problem:• You usually do not know which of the two reasons is the
problem in your approach.
• However, there are strong indications that energy modeling is the major problem at the current state-of-the-art. 22
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem:- Ideally, the correct solution should have lower energy than all
other possible solutions.
23
If this is the disparity map that has lower energy than all other
possible disparity maps, you have done a good job in the
energy modelling step.
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem.- Ideally, the correct solution should have lower energy than all
other possible solutions.
• Energy minimization:- Your optimization algorithm delivers a solution that is far off from
the exact minimum of your energy.
Problem:• You usually do not know which of the two reasons is the
problem in your approach.
• However, there are strong indications that energy modeling is the major problem at the current state-of-the-art. 24
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem.- Ideally, the correct solution should have lower energy than all
other possible solutions.
• Energy minimization:- Your optimization algorithm delivers a solution that is far off from
the exact minimum of your energy.
Problem:• You usually do not know which of the two reasons is the
problem in your approach.
• However, there are strong indications that energy modeling is the major problem at the current state-of-the-art. 25
Result of applying two different optimization
algorithms on the same energy function
ICM Graph-Cuts
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem.- Ideally, the correct solution should have lower energy than all
other possible solutions.
• Energy minimization:- Your optimization algorithm delivers a solution that is far off from
the exact minimum of your energy.
Problem:• You usually do not know which of the two reasons is the
problem in your approach.
• However, there are strong indications that energy modeling is the major problem at the current state-of-the-art. 26
Limitations of Energy Minimization I have implemented an energy minimization approach,
but it gives poor results. Why? There are 2 reasons:
• Energy modeling:- Your energy represents a poor model of your problem.- Ideally, the correct solution should have lower energy than all
other possible solutions.
• Energy minimization:- Your optimization algorithm delivers a solution that is far off from
the exact minimum of your energy.
Problem:• You usually do not know which of the two reasons is the
problem in your approach.
• However, there are strong indications that energy modeling is the major problem at the current state-of-the-art. 27
We will spend this and next sessions on the energy optimization problem.
We will then focus on the modelling component.
Michael BleyerLVA Stereo Vision
Dynamic Programming
Special Case of our Energy Function Let us come back to our energy function
where Esmooth is implemented by the Potts model. Optimization of E is np-complete:
• There is most likely no algorithm that can give you the exact minimum in reasonable time.
However, there is a special case:• If the smoothness interactions form a tree in the grid graph,
the optimal solution can be efficiently computed using dynamic programming (DP).
)()()( DEDEDE smoothdata
Example of a tree
A tree is a graph that does not contain cycles.
Dynamic Programming - Algorithm
Function L(r) computes the exact energy optimum r represents the root of the tree (can be chosen arbitrarily) D is the set of all allowed disparities m(p,d) are the costs for matching pixel p at disparity d. s(d,d’) gives a penalty if the disparities d and d’ have different
values (smoothness function) Cp is the set of all siblings of p (Those pixels that have p as a
direct predecessor on the path to the root node).
rCcDd
dcEdrmrL ),(),(min)(
pCcDd
dcEddsdpmdpE ),()',(),(min)',(
Dynamic Programming - Algorithm
Function L(r) computes the exact energy optimum r represents the root of the tree (can be chosen arbitrarily) D is the set of all allowed disparities m(p,d) are the costs for matching pixel p at disparity d. s(d,d’) gives a penalty if the disparities d and d’ have different
values (smoothness function) Cp is the set of all siblings of p (Those pixels that have p as a
direct predecessor on the path to the root node).
rCcDd
dcEdrmrL ),(),(min)(
pCcDd
dcEddsdpmdpE ),()',(),(min)',(
If the graph was not a tree, this recursion would run forever.
Dynamic Programming – An Example We will use the Potts model to
implement s() with P = 10. Matching costs are given by
r
s
t u
))2,(10),1,(5min()( sEsErL
)2,()2,(1015
),1,()1,(20min)1,(
uEtE
uEtEsE
2030,1010min)2,(
101030,10min)1,(
tE
tE 2525,1020min)2,(
201025,20min)1,(
uE
uE
5070,50min
55)70,55min(
m(p,d) r s t u
d1 5 20 10 20
d2 10 15 30 25
rCcDd
dcEdrmrL ),(),(min)(
pCcDd
dcEddsdpmdpE ),()',(),(min)',(
60)60,60min )2,()2,(15
),1,()1,(1020min)2,(
uEtE
uEtEsE
Dynamic Programming – An Example We will use the Potts model to
implement s() with P = 10. Matching costs are given by
r
s
t u
))2,(10),1,(5min()( sEsErL
)2,()2,(1015
),1,()1,(20min)1,(
uEtE
uEtEsE
2030,1010min)2,(
101030,10min)1,(
tE
tE 2525,1020min)2,(
201025,20min)1,(
uE
uE
5070,50min
55)70,55min(
m(p,d) r s t u
d1 5 20 10 20
d2 10 15 30 25
rCcDd
dcEdrmrL ),(),(min)(
pCcDd
dcEddsdpmdpE ),()',(),(min)',(
60)60,60min )2,()2,(15
),1,()1,(1020min)2,(
uEtE
uEtEsE
The energy of the optimal disparity assignment is 55.
What is the disparity assignment that has led to
this optimal energy?
Dynamic Programming – An Example We can find the disparity sequence that has led to the
optimum by back-tracking. We look which disparity was chosen at each pixel and
follow this path.
r
s
t u
))2,(10),1,(5min()( sEsErL
)2,()2,(1015
),1,()1,(20min)1,(
uEtE
uEtEsE
2030,1010min)2,(
101030,10min)1,(
tE
tE 2525,1020min)2,(
201025,20min)1,(
uE
uE
5070,50min
55)70,55min(
60)60,60min )2,()2,(15
),1,()1,(1020min)2,(
uEtE
uEtEsE
dr = 1
ds = 1
dt = 1 du = 1
Dynamic Programming – An Example We can find the disparity sequence that has led to the
optimum by back-tracking. We look which disparity was chosen at each pixel and
follow this path.
r
s
t u
))2,(10),1,(5min()( sEsErL
)2,()2,(1015
),1,()1,(20min)1,(
uEtE
uEtEsE
2030,1010min)2,(
101030,10min)1,(
tE
tE 2525,1020min)2,(
201025,20min)1,(
uE
uE
5070,50min
55)70,55min(
60)60,60min )2,()2,(15
),1,()1,(1020min)2,(
uEtE
uEtEsE
dr = 1
ds = 1
dt = 1 du = 1
Setting all pixels to disparity 1 represents the optimal
disparity assignment in our example.
DP on the 4-connected Grid
Problem:
• The 4-connected grid is definitely not a tree!
DP on the 4-connected Grid
Problem:
• The 4-connected grid is definitely not a tree!
Idea: We can remove edges (smoothness interactions) so that the 4-connected grid becomes a
tree.
The following approaches only differ in the way how they erase edges.
Scanline DP
All vertical edges are deleted from the 4-connected grid. That is what the majority of DP-based approaches do. Oftentimes, these approaches implement the ordering
assumption (I will skip this, because this is not state-of-the-art anymore.)
Scanline DP
All vertical edges are deleted from the 4-connected grid. That is what the majority of DP-based approaches do. Oftentimes, these approaches implement the ordering
assumption (I will skip this, because this is not state-of-the-art anymore.)
Scanline DP
All vertical edges are deleted from the 4-connected grid. That is what the majority of DP-based approaches do. Oftentimes, these approaches implement the ordering
assumption (I will skip this, because this is not state-of-the-art anymore.)
Scanline DP
All vertical edges are deleted from the 4-connected grid. That is what the majority of DP-based approaches do. Oftentimes, these approaches implement the ordering
assumption (I will skip this, because this is not state-of-the-art anymore.)
What will be the problem of this approach?
The Scanline Streaking Problem
Deleting the vertical smoothness edges leads to horizontal streaks in the disparity maps.
The problem is that smoothness between neighbouring scanlines is not enforced.
Tree DP by [Veksler, CVPR2005]
We can obtain a tree structure in a smarter way. Observation:
• Disparity discontinuities are typically aligned with intensity edges, hence:- Two neighbouring pixels of similar intensities are very
likely to lie on the same disparity.- Two neighbouring pixels of different intensities are less
likely to lie on the same disparity.
=> The smoothness edges between neighbouring pixels of very different intensities are the least important ones.=> We should remove those.
Tree DP by [Veksler, CVPR2005]
Algorithm for obtaining the tree structure:• For each smoothness edge between two pixels p and q:
- Compute a weight w(p,q) by
where I(p) denotes pixel p’s intensity.
• Build the minimum spanning tree (MST) using the computed weights:- The MST is the tree connecting all pixels whose sum of weights
is minimum among all such trees.- The MST can be computed in linear time using standard graph
algorithms.
|)()(|),( qIpIqpw
Tree DP by [Veksler, CVPR2005]
Pixels of high intensity difference
(no smoothness edge)
Pixels of low intensity difference
(smoothness edge)
Tree DP by [Veksler, CVPR2005]
Horizontal streaks are effectively reduced. However, vertical streaks or now present as well.
Semi-Global Matching by [Hirschmueller, CVPR2005]
Disclaimer: • The original paper is written
from a completely different perspective (no tree DP).
Construct an individual tree at each pixel p.
Tree contains the vertical, horizontal and diagonal lines on which p resides (star shape).
p
Semi-Global Matching by [Hirschmueller, CVPR2005]
No streaks, but isolated pixels
Semi-Global Matching by [Hirschmueller, CVPR2005]
If the tree for pixel p does not capture texture, the algorithm will fail.
Texture
Simple Tree by [Bleyer, VISAPP2008]
p p
Horizontal Tree
Motivation: • Overcome the problem of [Hirschmueller, CVPR2005] in untextured
regions Idea:
• Also generate an individual tree at each pixel p.
• This tree contains all pixels of the reference view (=> The problem of missing texture is avoided)
2 tree structures:
Vertical Tree
Simple Tree by [Bleyer, VISAPP2008]
Texture cannot be missed by these trees.
p
Simple Tree by [Bleyer, VISAPP2008]
No streaks, no problems in untextured regions.
Dynamic Programming - Pros and Cons
Pros:• DP algorithms are very fast (comparable to local methods)
• Good tradeoff between speed and accuracy
Cons:• Can only be applied on tree structures.
• Erasing smoothness edges leads to performance degradations.
• Optimization algorithms that operate on the full 4-connected grid perform better (next session)
Summary
Principle of global methods:• Energy modeling
• Energy minimization
Energy minimization for CV problems different from stereo
Dynamic Programming:• Scanline optimization
• Tree DP
• Semi-global matching
• Simple Tree
References M. Bleyer, M. Gelautz, Simple but Effective Tree Structures for
Dynamic Programming-based Stereo Matching, VISAPP 2008. H. Hirschmueller, Accurate and Efficient Stereo Processing by Semi-
Global Matching and Mutual Information, CVPR 2005. O. Veksler, Stereo Correspondence by Dynamic Programming on a
Tree, CVPR 2005.