Institute of Visual Computing

Computer Vision

Exercise Session 6 – Stereo matching

Institute of Visual Computing

Assignment 6

3 Tasks: Disparity computation Winner-takes-all Graph-cut

Textured 3D model

Institute of Visual Computing

Stereo Setup

Bring two views to standard stereo setup Epipoles are at infinity Epipolar lines are parallel

Institute of Visual Computing

Planar Rectification

Code provided

Institute of Visual Computing

Disparity – SSD, Winner-Takes-All

Match windows along scanline

Remember offset of best matching window as disparity value

Institute of Visual Computing

Disparity – SSD Slow Version

Find the offset d(x, y) of matching pixels

Search algorithm (convert to gray scale rgb2gray) For each pixel (x, y), for each disparity d SSD = 0 For each pixel (i, j) in window SSD = SSD + (I1(x+i, y+j) - I2(x+i+d, y+j+d)).^2

Remember disparity with smallest SSD

SLOW!

Institute of Visual Computing

Disparity – faster version for Matlab

For each disparity d Shift entire image by d (code provided (shiftImage)) Compute image difference (SSD, SAD) Convolve with box filter Use conv2(..., ‘same’) and fspecial(‘average’,…)

Remember best disparity for each pixel mask = Idiff < bestDiff

Resize images if your stereo is taking too long

Institute of Visual Computing

Disparity result

Depth should be continuous

Depth discontinuities

Institute of Visual Computing

Disparity – Graph-Cut

Stereo is a labeling problem Assign each pixel the corresponding disparity (label) Matching pixels should have similar intensities Most nearby pixels should have similar disparities

Institute of Visual Computing

Disparity – Graph-Cut

Familiarize yourself with the sample code See the gc_example() file on color segmentation compile_gc

Adapt the code to compute the disparity Change the data cost (Dc) Compute for each pixel the SSD at each disparity Store SSD values in a m x n x r matrix, where m x n is

the image size and r is the number of disparities (labels) The rest remains unchanged

You may need to change the weighting of the terms

Institute of Visual Computing

Graph-Cut - Results

Result with simple cost function

winner-takes -a ll graphcut

Institute of Visual Computing

For each pixel find the corresponding 3D point using

Disparity maps

Camera parameters, rectifying homographies

Generate textured 3D model (code provided)

.obj-file

.mtl-file

Image file

Textured 3D model

Put everything in the same folder, load .obj-file with Meshlab.

Institute of Visual Computing

3D Point Cloud

One 3D point per pixel

Two possible approaches Standard stereo geometry Triangulation

Do not forget that homographies were applied to the images in order to rectify them and maybe also scale them!

Institute of Visual Computing

3D Point Cloud

Standard stereo geometry formula given on the slides requires Equal intrinsics for both cameras Zero skew, principal point (0,0)

Do not forget that applying homographies to the images changes the camera matrices for the respective images

Institute of Visual Computing

3D Point Cloud

Triangulation For each pixel in image 1 get the corresponding pixel

in image 2 using the disparity map Triangulate using the camera matrices

Do not forget that applying homographies to the images changes the camera matrices for the respective images

Institute of Visual Computing

Textured 3D model from an image pair

Institute of Visual Computing

Framework

Functions that need to be completed/implemented (you can add functions, of course): stereoDisparity.m diffsGC.m gcDisparity.m generatePointCloudFromDisps.m exercise6.m

Institute of Visual Computing

Framework

Several functions provided for convenience linTriang: linear triangulation leftRightCheck: see if disparity calulated for left

image agrees with disparity calculated for right image Decompose: decompose camera matrix P into K, R

and COP shiftImage: shift entire image along x-axis …