stereo (part 1)

CSE 152A, Winter 2021 Introduction to Computer Vision I

Stereo(Part 1)

Introduction to Computer Vision ICSE 152ALecture 8


Announcements• Assignment 2 is due Feb 10, 11:59 PM• Quiz 2 is Feb 12• Reading:

– Szeliski• Section 11.3


Why Do We Have Two Eyes?

1. Redundancy – If we lose one, we’re not blind2. Larger field of view3. Ability to recover depth for some points


Why Do We Have Two Eyes?

Depth information islost in image formation

Binocular (stereo) visionenables depth estimation


Stereo Vision

Holmes Stereoscope


Binocular Stereopsis: MarsGiven two images of a scene where relativelocations of cameras are known, estimate depthof all common scene points.

Two images of Mars (Viking Lander)


An Application: Mobile Robot Navigation

The Stanford Cart,H. Moravec, 1979

INRIA Mobile Robot 1990

Mobi, Stanford, 1987


Mars Exploratory Rovers:Spirit and Opportunity,

2004


Curiosity Rover (2012)

• Navigation cameras (Navcams) B&W, 45o field of view• Hazard avoidance cameras (hazcams), 4 pairs, 120o field of

view


Boston Dynamics

Stereo + Lidar


Commercial Stereo Heads

Binocular stereo

Trinocularstereo


Stereo Vision Outline• Offline

– Calibrate cameras and determine epipolargeometry

• Online1. Acquire stereo images2. Rectify images to convenient epipolar geometry3. Establish correspondence 4. Estimate depthA

B

C

D


Binocular Stereo: Estimating Depth2-D world with 1-D image plane

Two measurements: XL, XRTwo unknowns: X,Z

Constants:Baseline: dFocal length: f

Disparity: (XL - XR)

Z = d f(XL - XR)

X = d XL(XL - XR)

(Adapted from Hager)

Z

X(0,0) (d,0)

Z=f

(X,Z)

XL XR

XL=f(X/Z) XR=f((X-d)/Z)


Faraway points – small disparityInfinitely far, zero disparity

Nearby points – large disparity


Reconstruction: General 3D case

• Linear Method:

• Non-Linear Method: Estimate that minimizes

Given two image measurements x and x′, estimate scene point

Solve system of linear equations


The need for corresondence


Matching complexity (naïve)

For each point in left mage, there are O(n2) possible matching points in right image.

With n2 pixels in left image, complexity of matching is O(n4)

Can we do better?

Input: two images that are n x n pixels

For a given point in the left image, where do we look in the right image?


Stereo Vision Outline• Offline

– Calibrate cameras and determine epipolargeometry

• Online1. Acquire stereo images2. Rectify images to convenient epipolar geometry3. Establish correspondence 4. Estimate depthA

B

C

D


Epipolar Geometry

• Epipolar Plane: Any plane that contains the baseline

• Epipoles (e,e’): Two intersection points of baseline with image planes

• Epipolar Lines (l, l’): Pair of lines from intersection of an epipolar plane with the two image planes

• Baseline: line connecting two centers of projection O and O’

Epipolar Geometry Terminology


Family of Epipolar Planes

• Epipolar Plane: Any plane that contains the baseline• The set of epipolar planes is a family of all planes passing through the

baseline and can be parameterized by the angle about baseline

O O’


Epipolar matching

O

p

O’

p1’p2’p’

l’l

P1

P2

P

e e’Baseline

• Potential matches for p have to lie on the corresponding epipolar line l’

• Epipolar line l’ passes through epipole e’, the intersection of the baseline with the image plane

• Potential matches for p’ have to lie on the corresponding epipolar line l


Epipolar ConstraintEpipolar matching complexity

Using epipolar matching, complexity is reduced from O(n4) to O(n3). Why?• There are n2 points in the left image• For each point in the left image, all candidate matches are on an epipolar

line in the right image, and the length of the epipolar line is O(n)• Therefore, match complexity is O(n2*n)=O(n3)

O

p

O’

p1’p2’p’

l’l

P1

P2

P

e e’Baseline


Skew Symmetric Matrix & Cross Product

• The cross product a x b of two vectors a=[a1 a2 a3]T

and b=[b1 b2 b3] T can be expressed a matrix vector product [a]xb where [a]x is the 3x3 skew symmetric matrix

• A matrix S is skew symmetric iff S = -ST

• The determinant of a skew symmetric matrix is 0


Epipolar Constraint: Calibrated Case

• Two pinhole cameras with camera frames {1} and {2}• The transformation between camera frames is given by a 3D

Euclidean transformation (rotation and translation)• Let each camera be calibrated (i.e., known internal

parameters)• P projects to p in Camera 1 and P projects to p’ in Camera 2• What is the relation of the coordinates p and p’?

{1} {2}

f f’


Epipolar Constraint: Calibrated Case

Essential Matrix(Longuet-Higgins, 1981)

The vectors , and are coplanar

skew

3D coordinates of p in camera 1 frame


Epipolar geometry example


Two ways to estimate the Essential Matrix

1. Calibration-based2. Eight-Point Algorithm


Calibration-Based Method

1. From image of known calibration fixture, determine intrinsic parameters K1, K2 and extrinsic relation of two cameras: R1, R2, t1, t2

2. Compute the relative rotation R and translation t of the two cameras from R1, R2, t1, and t2

3. Compute the Essential Matrix E


The Eight-Point Algorithm (Longuet-Higgins, 1981)

• Set E33 to 1• Use 8 points (ui,vi), i=1..8

Solve E11 to E32This are elements of the Essential Matrix

Input: 8 corresponding points in two images p=[ui,vi,1], p’=[ui’,vi’,1]


Epipolar geometry example


Example: converging cameras

courtesy of Andrew Zisserman


Example: motion parallel with image plane

(simple for stereo → rectification)courtesy of Andrew Zisserman


Example: forward motion

e

e’

courtesy of Andrew Zisserman


How do we use the Essential Matrix?

Given a point q in the left image, what is the equation of the corresponding epipolar line l in the right image?

ql


Computing the epipoles given E

• The epipole e’ in the right image is the eigenvector of Ecorresponding to the zero eigenvalue

• The epipole e in the left image is the eigenvector of ET

corresponding to the zero eigenvalue

Why?1. det(E) = det([t]x)det(R) = 0 because det([t]x=0) since [t]x is

skew symmetric. Consequently E has a zero Eigenvalue


Computing the epipoles given E

2. Now, a vector is e’ is an eigenvector of E iff Ee’ = λe’ where λ is the eigenvalue

3. Since E has a zero eigenvalue (λ=0), Ee’ =λe’= 0 4. Consider a point e’ in the right image, where Ee’ = 0, the “line

equation” in the left image using the epipolar constraint is pTEe’ = 0 = pT0

But this is not a line. Every point p in the left image satisfies this equation. Therefore e’ is the epipole in the right image.


Next Lecture• Stereo• Reading:

– Szeliski• Section 12.1.1

stereo (part 1)

Documents