lecture 4 reconstruction from two views

Lecture 4 Reconstruction from two views

Joaquim SalviUniversitat de Girona

Visual Perception

Lecture 4: Reconstruction from two views

Contents

4. Reconstruction from two views

4.1 Shape from X

4.2 Triangulation principle

4.3 Epipolar geometry – Modelling

4.4 Epipolar geometry – Calibration

4.5 Constraints in stereo vision

4.6 Experimental comparison of methods

4.7 Sample: Mobile robot performing 3D mapping

Contents

4.1 Shape from X

Techniques based on:

– Modifying the intrinsic camera parameters

i.e. Depth from Focus/Defocus and Depth from Zooming

– Considering an additional source of light onto the scene

i.e. Shape from Structured Light and Shape from Photometric

Stereo

– Considering additional surface information

i.e. Shape from Shading, Shape from Texture and Shape from

Geometric Constraints

– Multiple views

i.e. Shape from Stereo and Shape from Motion

Shape from Focus/Defocus

4.1 Shape from X

Stereo

– Multiple views

Shape from Structured Light

4.1 Shape from X

Stereo

– Multiple views

Shape from Shading

4.1 Shape from X

Stereo

– Multiple views

Shape from Stereo

Contents

4.1 Shape from X

Contents

4.1 Shape from X

coordinate

system

Camera’

coordinate

system

'CO 'C

'IX'IO

'RX 'R

Camera

coordinate

system

IXIO I

Camera

coordinate

system

Camera’

coordinate

system

'CO 'C

coordinate

system

IXIO I

'IX'IO

Pw = Pu + m u

Pw = P’u + m’ v

Steps:1 - Pu + m u = P’u + m’ v

2 - Expand to x,y,z

3 - Get m and m’

4 – Compute Pw

wP3D Object Point

Reconstructed

Camera

coordinate

system

Camera’

coordinate

system

'CO 'C

coordinate

system

IXIO I

'IX'IO

Pw = Pu + m u

Pw = P’u + m’ v

Lecture 4: Reconstruction from two viewsLecture 4: Reconstruction from two views

WP2D1u

http://astronomy.swin.edu.au/~pbourke/geometry/lineline3d/

Pa = P1 + mua (P2 - P1)

Pb = P3 + mub (P4 - P3)

Two different ways:

Minimize the distance between points:

Min || Pb - Pa ||2

Min || P1 + mua (P2 - P1) - P3 - mub (P4 - P3) ||2

Finding mua and mub once expanded to (x,y and z)

Compute the dot product between vectors:

(Pa - Pb)T (P2 - P1) = 0

(Pa - Pb)T (P4 - P3) = 0

Because they are perpendicular.

Finding mua and mub once expanded to Pa, Pb

and (x,y and z)

In practice we can use Least-Squares:

134333231

24232221

14131211

134333231

24232221

14131211

BvBBvBBvB

BuBBuBBuB

AvAAvAAvA

AuAAuAAuA

232332223221231

132331223211231

231332213221131

131331213211131

Add additional rows if we have additiional views of the same point

Contents

4.1 Shape from X

Contents

4.1 Shape from X

e e’

CKC’

4.3 Epipolar Geometry – Modelling

• Focal points, epipoles and epipolar lines

• e is defined by OC’ in {I}, e’ is defined by OC in {I’}

• m defines an epipolar line in {I’}; m’ defines an epipolar line in {I}

• All epipolar lines intersect at the epipole

Epipole

Epipolar lines

Area 1

Area 2

Correspondence

pointsZoom

Area 1

Area 2

Epipolar geometry of Camera 1 Epipolar geometry of Camera 2

epipole

e e’

CKC’

•The Epipolar Geometry concerns the problem of computing the plane .

• a plane is defined by the cross product between two vectors

• M is unknown, m and m’ are knowns

• {W} is located at {C} or {C’} and can be computed at {C} or {C’}

4 solutions

e e’

C’K’C

•The Epipolar Geometry concerns the problem of computing the plane .

• a plane is defined by the cross product between two vectors

• M is unknown, m and m’ are knowns

• {W} is located at {C} or {C’} and can be computed at {C} or {C’}

4 solutions

CKC’

e e’

lm’l’m

• Assume {W} at {C}

CKC’

tRtRRPe

• Assume {W} at {C}

OC’OC

e e’

lm’l’m

tRtRRPe

RmtRmtmPel

mtRmtRmRtRmPel

Since epipolar lines are contained in the plane , we can define the line by

a cross product of two vectors, obtaining the orthogonal vector of the line.

OC’OC

e e’

The Fundamental matrix is

defined by inner product of a

point with its epipolar line.

' Rmtl

'' Rmtmlmlm

mtRmlmlm

Orthogonal, their cosinus is 0

OC’OC

e e’

lm’l’m

'~'' '''~

Now we consider the

intrinsics. Points in pixels

instead of metrics

'~'~'~'~'0

~''~~'~''0

mRtmmRtmRmtm

mtRmmtRmmtRm

ttttttAAAABAB

FtRtRRtRtF

FRtRttRtRF

AAAAAAAA

F and F’ are related by a transpose. So,

Demonstration:

The same dissertation can be made assuming the origin at

{C’}, obtaining two more fundamental matrices that are also

equivalent to F and F’.

The Essential Matrix is the calibrated case of the Fundamental matrix.

• The Intrinsic parameters are known: A and A’ are known

The problem is reduced to estimate E or E’.

The monocular stereo is a symplified version of F where A = A’, reducing the

complexity of computing F.

Contents

4.1 Shape from X

Contents

4.1 Shape from X

4.4 Epipolar Geometry – Calibration

' ' 0T

m m F 1 ' 0

Operating, we obtain:

11 12 13 21 22 23 31 32 33, , , , , , , ,t

f F F F F F F F F F

, , , , , , , ,1i i i i i i i i i i i i iu x x y x x x y y y y x y

The epipolar geometry is defined as:

1 2, , ...,t

n nU u u u

The Eight Point Method

1n nU f

11 12 13 21 22 23 31 32, , , , , , ,t

f F F F F F F F F

, , , , , , ,i i i i i i i i i i i i iu x x y x x x y y y y x y

F is defined up to a scale factor, so we can fix one of the

component to 1. Let’s fix F33 = 1.

First solution is :

0f NOT WANTED

1 11n n n nU U f U

11n nf U

n n n nf U U U

Least-Squares

1 2, , ...,t

n nU u u u

The Eight Point Method with Least Squares

First solution is :

0f NOT WANTED

F has to be rank-2 because [tx] is rank-2.

Any system of equations:

can be solved by SVD so that f lies in the nullspace of Un= UDVT .

[U,D,V] = svd (Un)

Hence f corresponds to a multiple of the column of V that belongs to the

unique singular value of D equal to 0.

Note that f is only known up to a scaling factor.

11 12 13 21 22 23 31 32 33, , , , , , , ,t

f F F F F F F F F F0nU f

The Eight Point Method with Eigen Analysis

Contents

4.1 Shape from X

Contents

4.1 Shape from X

Extrinsics: Camera Pose

I I C W

C Ws m A K M

' '' ' ' 'I I C W

C Ws m A K M

3D Reconstruction:

'; 'I I

C CA A

'; 'C C

W WK K

Intrinsics: Optics & Internal Geometry

Constraints:

• The Correspondence Problem F/E matrix

• Stereo Configurations:

• Calibrated Stereo: Intrinsics and Extrinsics known Triangulation!

• Uncalibrated Stereo: Intrinsics and Extrinsics unknown F matrix

• Calibrated Monocular: Intrinsics known, Extrinsics unknown E matrix

• Uncalibrated Monocular: Intrinsics and Extrinsics unknown F matrix

Contents

4.1 Shape from X

Contents

4.1 Shape from X

4.6 Experimental comparison – methods

Linear Iterative Robust Optimisation Rank-2

Seven point (7p) X — yes

Eight point (8p) X LS or Eig. no

Rank-2 constraint X LS yes

Iterative Newton-

RaphsonX LS no

Linear iterative X LS no

Non-linear

minimization in

parameter space

X Eig. yes

Gradient technique X LS or Eig. no

FNS X AML no

CFNS X AML no

M-Estimator X LS or Eig. no / yes

LMedS X 7p / LS or Eig. no

RANSAC X 7p / Eig no

MLESAC X AML no

MAPSAC X AML no

LS: Least-Squares Eig: Eigen Analysis AML: Approximate Maximum Likelihood

Least-squares Eigen AnalysisApproximate Maximum

Likelihood

Image plane camera 1 Image plane camera 2

4.6 Experimental comparison – Methodology

Methods: 1.- 7-Point; 2.- 8-Point with Least-Squares;

3.- 8-Point with Eigen Analysis 4.- Rank-2 Constraint* Mean and Std. in pixels

Linear methods: Good results if the points are well located and no outilers

meanstd

4.6 Experimental comparison – Synthetic images

Lecture 4: Reconstruction from two viewsLecture 4: Reconstruction from two views* Mean and Std. in pixels

Iterative methods: Can cope with noise but inefficient in the presence of outliers

Methods: 5.- Iterative Linear; 6.- Iterative Newton-Raphson;

7.- Minimization in parameter space;

8.- Gradient using LS; 9.- Gradient using Eigen;

10.- FNS; 11.- CFNS

Lecture 4: Reconstruction from two viewsLecture 4: Reconstruction from two views* Mean and Std. in pixels

Robust methods: Cope with both noise and outliers

Methods: 12.- M-Estimator using LS; 13.- M-Estimator using Eigen;

14.- M-Estimator proposed by Torr;

15.- LMedS using LS; 16.- LMedS using Eigen;

17.- RANSAC; 18.- MLESAC; 19.- MAPSAC.

1.- 7-Point; 2.- 8-Point with Least-Squares; 3.- 8-Point with Eigen Analysis; 4.- Rank-2 Constraint;

5.- Iterative Linear; 6.- Iterative Newton-Raphson; 7.- Minimization in parameter space; 8.- Gradient using LS;

9.- Gradient using Eigen; 10.- FNS; 11.- CFNS; 12.- M-Estimator using LS; 13.- M-Estimator using Eigen;

14.- M-Estimator proposed by Torr; 15.- LMedS using LS; 16.- LMedS using Eigen; 17.- RANSAC;

18.- MLESAC; 19.- MAPSAC.

Linear Iterative Robust

Computing Time

4.6 Experimental comparison – Real images

Methods: 1.- 7-Point; 2.- 8-Point with Least-Squares;

3.- 8-Point with Eigen Analysis 4.- Rank-2 Constraint

Methods: 5.- Iterative Linear; 6.- Iterative Newton-Raphson;

7.- Minimization in parameter space;

8.- Gradient using LS; 9.- Gradient using Eigen;

10.- FNS; 11.- CFNS

Methods: 12.- M-Estimator using LS; 13.- M-Estimator using Eigen;

14.- M-Estimator proposed by Torr;

15.- LMedS using LS; 16.- LMedS using Eigen;

17.- RANSAC; 18.- MLESAC; 19.- MAPSAC.* Mean and Std. in pixels

4.6 Experimental comparison – Real images

• Survey of 15 methods of computing F and up to 19 different

implementations

• Description of the estimators from an algorithmic point of view

• Conditions: Gaussian noise, outliers and real images

– Linear methods: Good results if the points are well located and

the correspondence problem previously solved (without outliers)

– Iterative methods: Can cope with noise but inefficient in the

presence of outliers

– Robust methods: Cope with both noise and outliers

• Least-squares is worse than eigen analysis and approximate

maximum likelihood

• Rank-2 matrices are preferred if a good geometry is required

• Better results when data are previously normalized

4.6 Experimental comparison – Conclusions

Publications– X. Armangué and J. Salvi. Overall View Regarding Fundamental Matrix

Estimation. Image and Vision Computing, IVC, pp. 205-220, Vol. 21, Issue 2, February 2003.

– J. Salvi. An approach to coded structured light to obtain three dimensional information. PhD Thesis. University of Girona, 1997. Chapter 3.

– J. Salvi, X. Armangué, J. Pagès. A survey addressing the fundamental matrix estimation problem. IEEE International Conference on Image Processing, ICIP 2001, Thessaloniki, Greece, October 2001.

More Information: http://eia.udg.es/~qsalvi/

4.6 Experimental comparison – Conclusions

Contents

4.1 Shape from X

Contents

4.1 Shape from X

• Building a 3D map from an

unknown environment

using a stereo camera

system

• Localization of the robot in

the map

• Providing a new useful

sensor for the robot control

architecture GRILL Mobile robot with a

stereo camera system

Onboard Control Computer

Microcontroller

Onboard Vision Computer

Frame Grabber A Frame Grabber B

Motor Encoder Sonar

Camera A Camera B

Wireless Ethernet

Radio video emitter

Ethernet Card

PCI Bus

Ethernet Card

PCI Bus

Ethernet Link

PC104+PC104+

RS-232

Control system

Vision system

Pioneer 2

Outside stereo vision systemInside stereo vision system

GRILL Mobile Robot

4.7 3D mapping – Robot components

Image Acquisition

Image Processing

Low Level

Image Processing

High Level

Description Level

Camera

LocalizationMap Building

3D Information Motion

Estimation

Correspondence

Problem

Feature

Extraction

Gradients

Filtering

CalibrationRemove

Distortion

Tracking

Camera Modelling

and Calibration

Localization and

Map Building

Stereo Vision and

Reconstruction

LocalizationMap Building

CalibrationRemove

Distortion

3D Information

Correspondence

Problem

Motion

Estimation

4.7 3D mapping – Data flow diagram

2D Image Processing

3D Image Processing

Map Building and

Localization

Camera A Camera B

Sequence A Sequence B

2D Points

3D Points

3D Points Position

3D Map Trajectory

Image Flow

2D Points Flow

3D Points Flow

Position Flow

2D Image Processing

3D Image Processing

Map Building and

Localization

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

• Cameras are calibrated

• Both stereo images are

obtained simultaneously

4.7 3D mapping – Input sequence

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – RGB to I

• Description

– Converting a color image

to an intensity image

• Input

– Color image (RGB)

• Output

– Intensity image

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

• Description

– Removing distortion of an

image using camera

calibration parameters

• Input

– Distorted image

• Output

– Undistorted image

4.7 3D mapping – Remove Distortion

Intensity ImagesUndistorted Images

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Corners

• Description

– Detection of corners using

a variant of Harris corners

detector

• Input

– Undistorted image

• Output

– Corners list

Undistorted ImagesCorners Detected

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Spatial Cross Correlation

• Description

– Spatial cross correlation

using fundamental matrix

obtained from camera

• Input

– Undistorted image A

– Corners list A

– Undistorted image B

– Corners list B

• Output

– Spatial points list

– Spatial matches list

Corners Detected Points and matches list

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Temporal Cross Correlation

• Description

– Temporal cross correlation

using small windows

search

• Input

– Previous undistorted

– Previous corners list

– Current undistorted image

– Current corners list

• Output

– Temporal points list

– Temporal matches list

Corners Detected Points and matches list

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Stereo Reconstruction

• Description

– Stereo reconstruction by

triangulation using camera

• Input

– Spatial points list

– Spatial matches list

• Output

– 3D points list

Points and matches list

3D points list

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – 3D Tracker

• Description

– Tracking 3D points using

temporal cross correlation

• Input

– 3D points list

– Temporal points list A

– Temporal matches list A

– Temporal point list B

– Temporal matches list B

• Output

– 3D points history

– Points history A

– Matches history A

– Points history B

– Matches history B

Points and matches tracker

3D points

tracker

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Outliers Detection

• Description

– Detection of outliers

comparing distance

between current and

previous 3D points list

• Input

– Odometry position

– Current 3D points list

– Previous 3D points list

• Output

– Outliers list

Current and previous 3D points with outliers

Outlier Example

Current and Previous 3D points without outliers

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Local Localization

• Description

– Computing the absolute

position from the map by

minimizing distance

between the projection of

3D map points in cameras

and current 2D points and

matches.

• Input

– Odometry position as

initial guest

from the map

– Current 2D matches list

• Output

– Absolute position

Map absolute position

4.7 3D mapping – Global Localization

• Description

– Computing the trajectory

effect by the robot

• Input

– Local position

• Output

– Global position

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

RGB to I RGB to I

Remove

Distortion

Remove

Distortion

Corners Corners

Spatial Cross

Correlation

Temp. Cross

Correlation

Temp. Cross

Correlation

Image Points

Buffer A

Image Points

Buffer B

Camera A Camera B

Buffer B

Buffer A

Stereo

Reconstruction

3D TrackerOutliers

Detection

Localization

Global

Localization

Build 3D Map

3D Map Trajectory

4.7 3D mapping – Building 3D Map

• Description

– Building the 3D map from

the 3D points with a

history longer than n times

• Input

– Global position

• Output

– Global position

3D Map

4.7 3D mapping – Video

lecture 4 reconstruction from two views

Education

initializing 3-d reconstruction from three views using three

high resolution surface reconstruction from overlapping...

lecture 41 practical sampling and reconstruction

us history since reconstruction ~ week four lecture

3d model reconstruction from two orthographic views using...

fan beam reconstruction for limited views & sparse data

jet reconstruction at the lhc (lecture 1)

lecture 5: sql schema & views

lose the views: limited angle ct reconstruction via...

us history since reconstruction ~ week three lecture

games 3d reconstruction & understanding lecture i

projective reconstruction from multiple views with...

reconstruction from two views using approximate...

isometric views lecture 7

lecture #14: the reconstruction era (1865-1877)

reconstruction lecture notes

lecture 17 on reconstruction

views reconstruction

lecture 11 image restoration and reconstruction ii › hfan...

track reconstruction lecture notes