lecture 09: stereo algorithms - penn state college of ...rtc12/cse486/lecture09.pdf · cse486, penn...

CSE486, Penn StateRobert Collins

Lecture 09: Stereo Algorithms

CSE486, Penn StateRobert Collins

Recall: Simple Stereo System

Camps, PSU

X

Y Z

located atlocated at(0,0,0)(0,0,0)

leftcamera

zz

xx

yy

(X,Y,Z)

Image coords of point (X,Y,Z) Left Camera:

TTxx

rightcameralocated atlocated at(T(Txx,0,0),0,0)

zz

xx

yy

Right Camera:

( , )

( , )

CSE486, Penn StateRobert Collins

Recall: Stereo Disparity

Left camera

Right camera

Stereo Disparity

Important equation!

baseline

disparity

depth

CSE486, Penn StateRobert Collins

Recall: Stereo Disparity

Important equation!

disparity

depth

Left camera

Right camera

Note: Depth andstereo disparity areinversely proportional

CSE486, Penn StateRobert Collins

Stereo Example

Left Image Right Image

From Middlebury stereo evaluation pagehttp://www.middlebury.edu/stereo/

CSE486, Penn StateRobert Collins

Stereo Example

Disparity values (0-64)Left image

Right image

Note how disparity is larger(brighter) for closer surfaces.

CSE486, Penn StateRobert Collins

Computing Disparity

• Correspondence Problem:

– Determining which pixel in the right image

corresponds to each pixel in the left image.

– Disp = x_coord(left) - x_coord(right)

Camps, PSU

Recall our discussion of computing correspondencesof image patches ( Lecture 7 ).

SSD - sum of squared difference measureNCC - normalized cross correlation measure

CSE486, Penn StateRobert Collins

Epipolar Constraint

Important Concept:

For stereo matching, we don’t have to search the whole 2D right image for a corresponding point.

The “epipolar constraint” reduces the search spaceto a one-dimensional line.

CSE486, Penn StateRobert Collins

Left camera

Right camera

Recall : Simple Stereo System

zz

xx

yyzz

xx

yy

TTxx

Same Y Coord!

Epipolar line

P

CSE486, Penn StateRobert Collins

Matching using Epipolar LinesLeft Image Right Image

Match Score Values

For a patch in left image

Compare with patches along same row in right image

CSE486, Penn StateRobert Collins

Matching using Epipolar LinesLeft Image Right Image

Match Score Values

Select patch with highestmatch score.

Repeat for all pixels inleft image.

CSE486, Penn StateRobert Collins Example: 5x5 windows

NCC match score

Computed disparities Ground truth

Black pixels: bad disparity values,or no matching patch in right image

CSE486, Penn StateRobert Collins

Occlusions: No matches

Left image

Right image

CSE486, Penn StateRobert Collins

Effects of Patch Size

5x5 patches 11x11patches

Smoother in some areas Loss of finer details

CSE486, Penn StateRobert Collins

Adding Inter-Scanline Consistency

So far, each left image patch has been matchedindependently along the right epipolar line.

This can lead to errors.

We would like to enforce some consistencyamong matches in the same row (scanline).

CSE486, Penn StateRobert Collins

Disparity Space Image

First we introduce the concept of DSI.The DSI for one row represents pairwise match scores between patches along that row in the left and right image.

Pixels along left scanline

Pix

els

alon

g ri

ght

scan

line

Pixel i

Pixel j

C(i,j) = Match scorefor patch centeredat left pixel i withpatch centered atright pixel j.

CSE486, Penn StateRobert Collins

Disparity Space Image (DSI)Left Image Right Image

Dissimilarity Values(1-NCC) or SSD

CSE486, Penn StateRobert Collins

Disparity Space Image (DSI)Left Image Right Image

Dissimilarity Values(1-NCC) or SSD

CSE486, Penn StateRobert Collins

Disparity Space Image (DSI)Left Image Right Image

Dissimilarity Values(1-NCC) or SSD

CSE486, Penn StateRobert Collins

Disparity Space Image (DSI)Left Image

Dissimilarity Values

DSI

Enter each vector ofmatch scores as a column in the DSI

CSE486, Penn StateRobert Collins

Disparity Space ImageLeft scanline

Rig

ht s

canl

ine

CSE486, Penn StateRobert Collins

Disparity Space Image

Invalid entries due to constraintthat disparity >= low value(0 in this case)

Invalid entries due to constraintthat disparity <= high value64 in this case)

Left scanlineR

ight

sca

nlin

e

CSE486, Penn StateRobert Collins

Disparity Space ImageN cols in left scanline

M c

ols

in r

ight

sca

nlin

e

However, I’m going to keep the full image around, including invalid values(I think it is easier to understand thepixel coordinates involved)

If we rearrange the diagonal band of valid values into a rectangular array (in this caseof size 64 x N), that is what is traditionally known as the DSI

coordinate in left scanline (e.g. N)

Disparity(e.g. 64)

Disparity Space Image

CSE486, Penn StateRobert Collins

DSI and Scanline ConsistencyAssigning disparities to all pixels in left scanline now amounts to finding a connected path through the DSI

Start

End

CSE486, Penn StateRobert Collins

Lowest Cost Path

Want one with lowest “cost” (Lowest sum ofdissimilarity scores along the path)

We would like to choose the “best” path.

?

?

?

CSE486, Penn StateRobert Collins

Constraints on Path

It is common to impose an ordering constrainton the path. Intuitively, the path is not allowedto “double back” on itself.

CSE486, Penn StateRobert Collins

Ordering Constraint

Ordering constraint…

A

B CD

A B C D

A B C D

…and its failure

A B C

D

A B C D

D A B C

CSE486, Penn StateRobert Collins

Dealing with Occlusions

… …Left scanline Right scanline

CSE486, Penn StateRobert Collins

Dealing with Occlusions

… …Left scanline Right scanline

Match

Match

MatchOccluded fromright scanline

Occluded fromleft scanline

CSE486, Penn StateRobert Collins

An Optimal Scanline Strategy

• We want to find best path, taking intoaccount ordering constraint and thepossibility of occlusions.

Algorithm we will discuss now is fromCox, Hingorani, Rao, Maggs, “A MaximumLikelihood Stereo Algorithm,” ComputerVision and Image Understanding, Vol 63(3), May 1996, pp.542-567.

CSE486, Penn StateRobert Collins

Cox et.al. Stereo Matching

matchOccludedfrom left

Occludedfrom right

Three cases:– Matching patches. Cost = dissimilarity score– Occluded from right. Cost is some constant value.– Occluded from left. Cost is some constant value.

match

Occludedfrom left

Occludedfrom right

i-1,j-1 i-1,j

i,j-1 i,j

C(i,j)= min([ C(i-1,j-1) + dissimilarity(i,j) C(i-1,j) + occlusionConstant, C(i,j-1) + occlusionConstant]);

CSE486, Penn StateRobert Collins

Cox et.al. Stereo Matching

Recap: want to find lowestcost path from upper left tolower right of DSI image.

At each point on the path wehave three choices: step left,step down, step diagonally.

Each choice has a well-definedcost associated with it.

This problem just screams out for Dynamic Programming!(which, indeed, is how Cox et.al. solve the problem)

End

Start

CSE486, Penn StateRobert Collins

DP cost matrix(cost of optimal path from each point to END)

DSIReal Scanline Example

Every pixel in left column now is marked witheither a disparity value, or an occlusion label.

Proceed for every scanline in left image.

CSE486, Penn StateRobert Collins

Example

Result of DP alg. Black pixels = occluded.

Result of DP alg Result without DP (independent pixels)

CSE486, Penn StateRobert Collins

Occlusion Filling

Simple trick for filling in gaps caused by occlusion.

= left occluded

Fill in left occluded pixels with value from the nearest valid pixel preceding it in the scanline.

Similarly, for right occluded, look for valid pixel to the right.

CSE486, Penn StateRobert Collins

Example

Result of DP alg with occlusion filling.

CSE486, Penn StateRobert Collins

ExampleResult of DP alg with occlusion filling. Result without DP (independent pixels)

CSE486, Penn StateRobert Collins

ExampleResult of DP alg with occlusion filling. Ground truth

CSE486, Penn StateRobert Collins www.middlebury.edu/stereo/

CSE486, Penn StateRobert Collins

State-of-the-Art Results

J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum.“Symmetric stereo matching for occlusion handling”. IEEE Conference on Computer Vision and Pattern Recognition, June 2005.

Algorithm Results Ground truth