CSE486, Penn StateRobert Collins
Lecture 09: Stereo Algorithms
CSE486, Penn StateRobert Collins
Recall: Simple Stereo System
Camps, PSU
X
Y Z
located atlocated at(0,0,0)(0,0,0)
leftcamera
zz
xx
yy
(X,Y,Z)
Image coords of point (X,Y,Z) Left Camera:
TTxx
rightcameralocated atlocated at(T(Txx,0,0),0,0)
zz
xx
yy
Right Camera:
( , )
( , )
CSE486, Penn StateRobert Collins
Recall: Stereo Disparity
Left camera
Right camera
Stereo Disparity
Important equation!
baseline
disparity
depth
CSE486, Penn StateRobert Collins
Recall: Stereo Disparity
Important equation!
disparity
depth
Left camera
Right camera
Note: Depth andstereo disparity areinversely proportional
CSE486, Penn StateRobert Collins
Stereo Example
Left Image Right Image
From Middlebury stereo evaluation pagehttp://www.middlebury.edu/stereo/
CSE486, Penn StateRobert Collins
Stereo Example
Disparity values (0-64)Left image
Right image
Note how disparity is larger(brighter) for closer surfaces.
CSE486, Penn StateRobert Collins
Computing Disparity
• Correspondence Problem:
– Determining which pixel in the right image
corresponds to each pixel in the left image.
– Disp = x_coord(left) - x_coord(right)
Camps, PSU
Recall our discussion of computing correspondencesof image patches ( Lecture 7 ).
SSD - sum of squared difference measureNCC - normalized cross correlation measure
CSE486, Penn StateRobert Collins
Epipolar Constraint
Important Concept:
For stereo matching, we don’t have to search the whole 2D right image for a corresponding point.
The “epipolar constraint” reduces the search spaceto a one-dimensional line.
CSE486, Penn StateRobert Collins
Left camera
Right camera
Recall : Simple Stereo System
zz
xx
yyzz
xx
yy
TTxx
Same Y Coord!
Epipolar line
P
CSE486, Penn StateRobert Collins
Matching using Epipolar LinesLeft Image Right Image
Match Score Values
For a patch in left image
Compare with patches along same row in right image
CSE486, Penn StateRobert Collins
Matching using Epipolar LinesLeft Image Right Image
Match Score Values
Select patch with highestmatch score.
Repeat for all pixels inleft image.
CSE486, Penn StateRobert Collins Example: 5x5 windows
NCC match score
Computed disparities Ground truth
Black pixels: bad disparity values,or no matching patch in right image
CSE486, Penn StateRobert Collins
Occlusions: No matches
Left image
Right image
CSE486, Penn StateRobert Collins
Effects of Patch Size
5x5 patches 11x11patches
Smoother in some areas Loss of finer details
CSE486, Penn StateRobert Collins
Adding Inter-Scanline Consistency
So far, each left image patch has been matchedindependently along the right epipolar line.
This can lead to errors.
We would like to enforce some consistencyamong matches in the same row (scanline).
CSE486, Penn StateRobert Collins
Disparity Space Image
First we introduce the concept of DSI.The DSI for one row represents pairwise match scores between patches along that row in the left and right image.
Pixels along left scanline
Pix
els
alon
g ri
ght
scan
line
Pixel i
Pixel j
C(i,j) = Match scorefor patch centeredat left pixel i withpatch centered atright pixel j.
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image Right Image
Dissimilarity Values(1-NCC) or SSD
CSE486, Penn StateRobert Collins
Disparity Space Image (DSI)Left Image
Dissimilarity Values
DSI
Enter each vector ofmatch scores as a column in the DSI
CSE486, Penn StateRobert Collins
Disparity Space ImageLeft scanline
Rig
ht s
canl
ine
CSE486, Penn StateRobert Collins
Disparity Space Image
Invalid entries due to constraintthat disparity >= low value(0 in this case)
Invalid entries due to constraintthat disparity <= high value64 in this case)
Left scanlineR
ight
sca
nlin
e
CSE486, Penn StateRobert Collins
Disparity Space ImageN cols in left scanline
M c
ols
in r
ight
sca
nlin
e
However, I’m going to keep the full image around, including invalid values(I think it is easier to understand thepixel coordinates involved)
If we rearrange the diagonal band of valid values into a rectangular array (in this caseof size 64 x N), that is what is traditionally known as the DSI
coordinate in left scanline (e.g. N)
Disparity(e.g. 64)
Disparity Space Image
CSE486, Penn StateRobert Collins
DSI and Scanline ConsistencyAssigning disparities to all pixels in left scanline now amounts to finding a connected path through the DSI
Start
End
CSE486, Penn StateRobert Collins
Lowest Cost Path
Want one with lowest “cost” (Lowest sum ofdissimilarity scores along the path)
We would like to choose the “best” path.
?
?
?
CSE486, Penn StateRobert Collins
Constraints on Path
It is common to impose an ordering constrainton the path. Intuitively, the path is not allowedto “double back” on itself.
CSE486, Penn StateRobert Collins
Ordering Constraint
Ordering constraint…
A
B CD
A B C D
A B C D
…and its failure
A B C
D
A B C D
D A B C
CSE486, Penn StateRobert Collins
Dealing with Occlusions
… …Left scanline Right scanline
CSE486, Penn StateRobert Collins
Dealing with Occlusions
… …Left scanline Right scanline
Match
Match
MatchOccluded fromright scanline
Occluded fromleft scanline
CSE486, Penn StateRobert Collins
An Optimal Scanline Strategy
• We want to find best path, taking intoaccount ordering constraint and thepossibility of occlusions.
Algorithm we will discuss now is fromCox, Hingorani, Rao, Maggs, “A MaximumLikelihood Stereo Algorithm,” ComputerVision and Image Understanding, Vol 63(3), May 1996, pp.542-567.
CSE486, Penn StateRobert Collins
Cox et.al. Stereo Matching
matchOccludedfrom left
Occludedfrom right
Three cases:– Matching patches. Cost = dissimilarity score– Occluded from right. Cost is some constant value.– Occluded from left. Cost is some constant value.
match
Occludedfrom left
Occludedfrom right
i-1,j-1 i-1,j
i,j-1 i,j
C(i,j)= min([ C(i-1,j-1) + dissimilarity(i,j) C(i-1,j) + occlusionConstant, C(i,j-1) + occlusionConstant]);
CSE486, Penn StateRobert Collins
Cox et.al. Stereo Matching
Recap: want to find lowestcost path from upper left tolower right of DSI image.
At each point on the path wehave three choices: step left,step down, step diagonally.
Each choice has a well-definedcost associated with it.
This problem just screams out for Dynamic Programming!(which, indeed, is how Cox et.al. solve the problem)
End
Start
CSE486, Penn StateRobert Collins
DP cost matrix(cost of optimal path from each point to END)
DSIReal Scanline Example
Every pixel in left column now is marked witheither a disparity value, or an occlusion label.
Proceed for every scanline in left image.
CSE486, Penn StateRobert Collins
Example
Result of DP alg. Black pixels = occluded.
Result of DP alg Result without DP (independent pixels)
CSE486, Penn StateRobert Collins
Occlusion Filling
Simple trick for filling in gaps caused by occlusion.
= left occluded
Fill in left occluded pixels with value from the nearest valid pixel preceding it in the scanline.
Similarly, for right occluded, look for valid pixel to the right.
CSE486, Penn StateRobert Collins
Example
Result of DP alg with occlusion filling.
CSE486, Penn StateRobert Collins
ExampleResult of DP alg with occlusion filling. Result without DP (independent pixels)
CSE486, Penn StateRobert Collins
ExampleResult of DP alg with occlusion filling. Ground truth
CSE486, Penn StateRobert Collins www.middlebury.edu/stereo/
CSE486, Penn StateRobert Collins
State-of-the-Art Results
J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum.“Symmetric stereo matching for occlusion handling”. IEEE Conference on Computer Vision and Pattern Recognition, June 2005.
Algorithm Results Ground truth