www.kit.eduKIT – University of the State of Baden-Wuerttemberg andNational Laboratory of the Helmholtz Association

Andreas Geiger*, Martin Roser* and Raquel Urtasun**

Efficient Large-Scale Stereo Matching

*KARLSRUHE INSTITUTE OF TECHNOLOGY **TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO

Contents

1 Motivation and Related Work

2 Efficient Large-Scale Stereo Matching

3 Experimental Evaluation

4 Summary and Future Work

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 2/24

Contents

1 Motivation and Related Work

2 Efficient Large-Scale Stereo Matching

3 Experimental Evaluation

4 Summary and Future Work

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 3/24

Motivation

Laserscanner

Camera

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 4/24

Why is 3D from Stereo hard?

AmbiguitiesTextureless regions

Sensor saturationNon-Lambertian surfaces

∆z grows quadraticallyComputational burden

disparity error

focal length baseline

distance error

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24

Why is 3D from Stereo hard?

AmbiguitiesTextureless regions

Sensor saturationNon-Lambertian surfaces

∆z grows quadraticallyComputational burden

disparity error

focal length baseline

distance error

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24

Why is 3D from Stereo hard?

AmbiguitiesTextureless regions

Sensor saturationNon-Lambertian surfaces

∆z grows quadraticallyComputational burden

disparity error

focal length baseline

distance error

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 5/24

Related Work: Local Methods

Local MethodsWinner-takes-All

ExamplesBlock matching(Scharstein 02)Adaptive windows(Kanade 94, Yoon 06)Plane-sweep(Collins 96, Gallup 07)

ProblemsSmall matching ratiosBorder bleeding

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 6/24

Related Work: Global Methods

Global MethodsMinimize 1D/2D energy E(d) = Edata(d) + λEsmooth(d)

ExamplesGraph cuts, Belief propagation(Kolmogorov 02, Felzenszwalb 06)Variational methods(Pock 07, Zach 09)Fusion moves(Woodford 08, Bleyer 10)

ProblemsComputational and memoryrequirementsPairwise potentialscan not model planarity

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 7/24

Related Work: Seed-and-Grow

Seed-and-Grow MethodsGrow disparity components from random seeds

Examples(Cech 07)(Sara 03)

ProblemsSlanted/textureless surfacesNo dense disparity maps

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 8/24

Contents

1 Motivation and Related Work

2 Efficient Large-Scale Stereo Matching

3 Experimental Evaluation

4 Summary and Future Work

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 9/24

Idea

Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24

Idea

Easy

Hard

Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24

Idea

Robust Match

Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24

Idea

Assumption: rectified imagesImage pairs contain ’easy’ and ’hard’ correspondencesRobustly match ’easy’ correspondences on regular gridBuild prior on dense search space⇒ dense matching

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 10/24

Efficient Large-Scale Stereo

NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T

Disparity dn ∈ NObservations on = (un vn fn)T

Local image features fn

AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24

Efficient Large-Scale Stereo

NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T

Disparity dn ∈ NObservations on = (un vn fn)T

Local image features fn

AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24

Efficient Large-Scale Stereo

NotationRobust support points S = {s1, ...,sM}with sm = (um vm dm)T

Disparity dn ∈ NObservations on = (un vn fn)T

Local image features fn

AlgorithmSplit image domain into support points S and dense pixelsAssume factorization of distribution over disparity,observations and support points into ...

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 11/24

Model

Left Image Graphical ModelRight Image

Support Points

Observation in left image Observations

Support Points

p(dn,o(l)n ,o(r)

n ,S) ∝ p(dn|S,o(l)n )︸ ︷︷ ︸

Prior

p(o(r)n |o

(l)n ,dn)︸ ︷︷ ︸

Likelihood

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 12/24

Model

Left Image Graphical ModelRight Image

Support Points

Observation in left image Observations

Support Points

p(dn,o(l)n ,o(r)

n ,S) ∝ p(dn|S,o(l)n )︸ ︷︷ ︸

Prior

p(o(r)n |o

(l)n ,dn)︸ ︷︷ ︸

Likelihood

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 12/24

Prior and Likelihood

Prior p(dn|S, o(l)n )

Support pt. triangulationPiecew. linear manifoldLocal extrapolation

Likelihood p(o(r)n |o

(l)n , dn)

Laplace distribution5× 5 block window3× 3 Sobel filter

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 13/24

Prior and Likelihood

Prior p(dn|S, o(l)n )

Support pt. triangulationPiecew. linear manifoldLocal extrapolation

Likelihood p(o(r)n |o

(l)n , dn)

Laplace distribution5× 5 block window3× 3 Sobel filter

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 13/24

Sampling from the model

Left image Sample mean

Right image

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 14/24

Sampling from the model

Left image Sample mean Right image

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 14/24

Contents

1 Motivation and Related Work

2 Efficient Large-Scale Stereo Matching

3 Experimental Evaluation

4 Summary and Future Work

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 15/24

Middlebury Benchmark

900 x 750 pixels, ground truth

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24

Middlebury Benchmark

900 x 750 pixels, 0.4 seconds

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24

Middlebury Benchmark

1300 x 1100 pixels, ground truth

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24

Middlebury Benchmark

1300 x 1100 pixels, 1 second

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 16/24

Accuracy (on cones image pair)

0 0.5 1 1.5 20

5

10

15

20

Image resolution [Megapixel]

Err

or >

2px

[%]

uniform priorCech 07Felzenszwalb 06Kolmogorov 01Kostkova 03our method

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 17/24

Running times (on cones image pair)

0 0.5 1 1.5 20

500

1000

1500

2000

Image resolution [Megapixel]

Run

ning

tim

e [s

]

Kolmogorov 01Kostkova 03our method

0 0.5 1 1.5 20

10

20

30

40

50

Image resolution [Megapixel]R

unni

ng ti

me

[s]

Felzenszwalb 06Cech 07uniform priorour method

[For more details see: Geiger et al., ACCV 2010]

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 18/24

3D Reconstruction: Brussels

2 seconds[http://cvlab.epfl.ch/data/strechamvs/]

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 19/24

3D Face Reconstruction

[http://www.fujifilm.com/products/3d]

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 20/24

Urban Scene Reconstruction

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 21/24

Contents

1 Motivation and Related Work

2 Efficient Large-Scale Stereo Matching

3 Experimental Evaluation

4 Summary and Future Work

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 22/24

Summary

Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24

Summary

Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24

Summary

Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24

Summary

Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24

Summary

Simple prior based on sparse feature matchesReduced ambiguities and run-timeTakes into account slanted surfacesReal-time 3D reconstruction of static scenes on CPUC++ / MATLAB code available at http://cvlibs.net

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 23/24

Future Work

Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets

Thank you!

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24

Future Work

Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets

Thank you!

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24

Future Work

Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets

Thank you!

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24

Future Work

Develop better priorsIncorporate segmentation / global reasoning on linesGPU implementation(goal: 20 fps at 1-2 megapixels)Employ as unitary potentials on global methods⇒ smaller label sets

Thank you!

Efficient Large-Scale Stereo Matching – Code: http://cvlibs.net 24/24