stereo & iterative graph-cuts alex rav-acha vision course hebrew university
Post on 20-Dec-2015
212 views
TRANSCRIPT
![Page 1: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/1.jpg)
Stereo &
Iterative Graph-Cuts
Alex Rav-AchaVision Course Hebrew University
![Page 2: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/2.jpg)
The stereo problem
![Page 3: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/3.jpg)
The stereo problem
•Both images are very similar (like images that you see with your two eyes)
• Most of the pixels in the left image are present in the right image (except for few occlusions)
![Page 4: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/4.jpg)
After rectification: all correspondences are along the same horizontal scan lines
The stereo problem
(pixels in one image simply shift horizontally in the other image)
![Page 5: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/5.jpg)
The stereo problem
•The horizontal shifts between the images are sometimes called: “disparities”
•The Disparities relates to depth: Closer objects have larger disparities.
![Page 6: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/6.jpg)
The stereo problem: compute the disparity map between two images
![Page 7: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/7.jpg)
Traditional Approaches• Matching rigid windows around each pixel• Each window is matched independently Modern approaches• Finding coherent correspondences for all pixels
- “Graph cuts”- “Belief Propagation”
![Page 8: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/8.jpg)
Window-Based Approach
•Compute a cost for each location
•Location with the lowest cost wins
![Page 9: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/9.jpg)
General Problem : Ambiguity
Left Right
scanline
![Page 10: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/10.jpg)
Window-Based Approach
Small Window Large Window
noisy in low texture areas blurred boundaries
![Page 11: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/11.jpg)
Results with best window size(still not good enough)
Window-based matching(best window size)
Ground truth
![Page 12: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/12.jpg)
Graph Cuts
Ground truthGraph cuts
![Page 13: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/13.jpg)
Maximum flow problem
• Max flow problem:– Each edge is a “pipe”– Find the largest flow F
of “water” that can be sent from the “source” to the “sink” along the pipes
– Edge weights give the pipe’s capacity
“source”
A graph with two terminals
S T
“sink”
a flow F
![Page 14: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/14.jpg)
Minimum cut problem
• Min cut problem:– Find the cheapest way
to cut the edges so that the “source” is completely separated from the “sink”
– Edge weights now represent cutting “costs”
a cut C
“source”
A graph with two terminals
S T
“sink”
![Page 15: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/15.jpg)
Max flow/Min cut theorem
• Max Flow = Min Cut:– Maximum flow saturates the
edges along the minimum cut.
– Ford and Fulkerson, 1962– Problem reduction!
• Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution
“source”
A graph with two terminals
S T
“sink”
![Page 16: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/16.jpg)
Min-Cut: Important Rule
No subset of the cut can also be a cut
This is not a minimal cut
![Page 17: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/17.jpg)
Energy Minimization Using Iterative Graph cuts
Fast Approximate Energy Minimization via Graph Cuts
Yuri Boykov, Olga Veksler and Ramin Zabih
Pami 2001
![Page 18: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/18.jpg)
To do better we need a better model of images
• We can make reasonable assumptions about the surfaces in the world
• Usually assume that the surfaces are smooth• Can pose the problem of finding the
corresponding points as an energy (or cost) minimization:
how well the pixels match up for different
disparities
neighboring pixels have similar disparities
)()()( fEfEfE smoothdata f-assignment
![Page 19: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/19.jpg)
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
To do better we need a better model of images
• We can make reasonable assumptions about the surfaces in the world
• Usually assume that the surfaces are smooth• Can pose the problem of finding the
corresponding points as an energy (or cost) minimization:
Data term is calculated for
each pixels
Smoothness is calculated on
neighbor pixels
f-assignment
p,q-pixels
![Page 20: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/20.jpg)
Example for Smoothness terms
• Quadratic
• L1
• Truncated L1
• Potts model )(),( qpqp ffKffv
),min( Kff qp
qp ff
2)( qp ff
![Page 21: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/21.jpg)
Constructing a Graph to Solve the Stereo Problem
![Page 22: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/22.jpg)
Constructing a Graph to Solve the Stereo Problem
![Page 23: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/23.jpg)
Constructing a Graph to Solve the Stereo Problem
The labels of each pixel are the possible disparity values
![Page 24: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/24.jpg)
The labels of each pixel are the possible disparity values
Constructing a Graph to Solve the Stereo Problem
![Page 25: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/25.jpg)
Relation between the Energy and the Graph labeling problem
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
Data term Smoothness term
p q1
10
{fp=10}
{fq=2}
![Page 26: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/26.jpg)
Relation between the Energy and the Graph labeling problem
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
Data term Smoothness term
p q1
10
Dp(10)
V(p,q)(1, 10)
![Page 27: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/27.jpg)
Iterative graph-cuts
• Use an iterative scheme to find a “good” local optimum of the energy function.
• In each iteration: convert the original multi-label problem to a binary one, and solve it by finding a minimal graph-cut (max-flow).
• The most popular scheme is the expansion move.
• -expansion: set the label of each pixel to be either or the current label.
![Page 28: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/28.jpg)
Types of Moves
Problem: A lot of local minima
A Single Pixel Move
![Page 29: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/29.jpg)
Types of Moves
Any pixel can change its label to alpha
Expansion Move
![Page 30: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/30.jpg)
Types of Moves
Claim (without proof): The difference between the optimal solution and the solution from the iterative expansion moves is bounded
Expansion Move
![Page 31: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/31.jpg)
Energy Minimization Algorithm
1. Start with arbitrary labeling f
2. Set success = 0
3. For each label – Find– If set and success =1
4. If success =1 goto (2)
5. Return f
expansion- one within among )(minargˆ ffEf L
)()( fEfE ff :
![Page 32: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/32.jpg)
Conditions on the Smoothness for using expansion moves:
In other words: V should be a metric
Note : The Quadratic smoothness is not a metric
![Page 33: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/33.jpg)
For each pair of vertices such that we add a ‘dummy’ vertex (together with the respective edges as shown in the table).
Nqp },{Nqp },{
![Page 34: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/34.jpg)
The Relation between the cut and the Energy
• Given a cut C, we define a labelling fc by:
• The cost of a cut C is |C| = E(fC) (plus a constant)
oldp
p ff
If the cut C separates p and If the cut C separates p and
![Page 35: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/35.jpg)
The Relation between the cut and the Energy
![Page 36: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/36.jpg)
The Relation between the cut and the Energy
![Page 37: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/37.jpg)
Conditions on the Smoothness for using expansion moves:
In other words: V should be a metric
![Page 38: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/38.jpg)
![Page 39: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/39.jpg)
![Page 40: Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University](https://reader031.vdocuments.us/reader031/viewer/2022032800/56649d4b5503460f94a28a69/html5/thumbnails/40.jpg)