Mixing Catadioptric and Perspective Cameras

Peter Sturm

INRIA Rhône-AlpesFrance

Introduction

Introduction

Introduction• Existing results

- epipolar geometry between omnidirectional cameras- motion estimation- (self-) calibration- ...

Our Goals• Study the geometry of hybrid stereo systems (omnidirectional and perspective cameras)

• Applications

- motion estimation

- epipolar geometry- trifocal tensors- plane homographies

- calibration- self-calibration, calibration transfer- 3D reconstruction- ...

Plan Camera models

• Epipolar geometry of hybrid systems

• Applications

• Conclusions

• Derivation of matching tensors

Camera ModelsPerspective and affine cameras

Camera ModelsCentral catadioptric cameras

• mirror (surface of revolution of a conic)

• camera

• virtual optical center

Camera ModelsCentral catadioptric cameras

• calibration

• mirror (surface of revolution of a conic)

• camera

• virtual optical center

Types of central catadioptric cameras• hyperbola + perspective camera

• parabola + affine camera

• ...

Camera Models

Plan• Camera models

Epipolar geometry of hybrid systems

• Applications

• Conclusions

• Derivation of matching tensors

Epipolar Geometry

epipolar line

epipolar plane

epipolar line

q

q'Fd 3

T

333~

x

d

qFd' 3333~

x

d’q’

epipole

Epipolar Geometry

epipolar conic

Epipolar Geometry

epipolar conic

epipolar line

epipole

epipoleepipoles

Epipolar Geometry Example

Epipolar Geometry

q

qFd' 3333~

x

q'Fd 3

T

333~

x

d d’q’

purely perspective case

0Fqq'T

epipolar conic

epipolar line

epipole

epipoles

“ conic ~ F q’ “

There exists withF 36x

q'Fc 3366~

xconcretely:

Interpretation:

0q2

22

q365

624

541T

ccccccccc

Epipolar Geometry

epipolar conic

epipolar line

epipole

epipoles

06325314213

2

32

2

21

2

1 cqqcqqcqqcqcqcq

0

6

5

4

3

2

1

323121

2

3

2

2

2

1

cccccc

qqqqqqqqq

“lifted coordinates”q̂

0Fq'q̂T

“ conic ~ F q’ “

There exists withF 36x

q'Fc 3366~

xconcretely:

Interpretation:

0q2

22

q365

624

541T

ccccccccc

Epipolar Geometry Example

Epipolar Geometry

BUT...

Until now, linear epipolar relation only found for:• any combination of perspective, affine or para-catadioptric cameras (parabolic mirrors)

Not yet for:• other omnidirectional cameras than para-catadioptric ones (e.g. based on hyperbolic mirrors)

Epipolar Geometry

Special case:

• combination of perspective and para-catadioptric cameras

• epipolar conics are circles

• F is of dimension 4x3

qqqqqqq

32

31

2

3

2

2

2

1

• the « lifted coordinates » are:

Epipolar Geometry

Epipoles:

• is of rank 2

• The epipole of the perspective camera is the right null-vector of F

• F has a one-dimensional left null-space the two epipoles of the catadioptric camera are the left null-vectors that are valid lifted coordinates (quadratic constraint):

0~ q̂q̂q̂q̂q̂ 2

4

2

321

32

31

2

3

2

2

2

1

qqqqqqq

F 34x

Plan• Camera models

• Epipolar geometry of hybrid systems

• Applications

• Conclusions

Derivation of matching tensors

Matching Tensors

• Multi-linear relations between coordinates of correponding image points

• Purely perspective case: derivation based on linear equations representing projections (3D 2D)

• Here: equations for back-projection (2D 3D directions)

- perspective cameras

1

-Q

tIRKq PPPPP

DtQ PPP

qKRD P

-1

P

T

PPwith

qP

tP

Q

Matching Tensors• Linear equations representing back-projections

- perspective cameras

qKRtQP

-1

P

T

PPP

- para-catadioptric cameras

q̂BRtQCC

T

CCC

• is of dimension 3x4• it depends - on the mirror’s intrinsic parameters - on the affine camera’s intrinsic parameters

BC

tC

Q

qC

Matching Tensors• Putting the equations together

qKRtQP

-1

P

T

PPP

q̂BRtQCC

T

CCC

0

0

Q

1

Iq̂BR0t

I0qKRt

6

6

66C

P

33CCTCC

33P1-P

TPP

x

x

x

0QqKRt P

-1

P

T

PPP

0Qq̂BRt CC

T

CCC

Matching Tensors• Putting the equations together

0

0

Q

1

Iq̂BR0t

I0qKRt

6

6

66C

P

33CCTCC

33P1-P

TPP

x

x

x

This matrix has a kernel

its rank is lower than 6

its determinant is zero

bilinear equation on the coefficients of andqP

q̂C

0qFq̂PC

T

Matching Tensors• Straightforward extension to more than 2 views ...

0

0

0

Q

'

1

Iq̂BR00t

I0q'K'R'0t'

I00qKRt

9

7

79C

P

P

33CCTCC

33P1-P

TPP

33P1-P

TPP

x

x

x

x

Plan• Camera models

• Epipolar geometry of hybrid systems

• Derivation of matching tensors

Applications

• Conclusions

- Self-calibration of omnidirectional cameras from fundamental matrices- Calibration transfer from an omnidirectional to a perspective camera- Self-calibration of omnidirectional cameras from a plane homography

Applications

Self-calibration of omnidirectional cameras fromfundamental matrices:

• para-catadioptric camera has 3 intrinsic parameters

self-calibration is possible from two or more fundamental matrices

• representation as a 4-vector of homogeneous coordinates

• this vector is in the left null-space of the fundamental matrix of this camera, defined with respect to any other camera (perspective, affine, catadioptric)

Applications

Calibration transfer from an omnidirectional to aperspective camera:

• input: - calibration of a para-catadioptric camera - fundamental matrix with a perspective camera

closed-form solution for the focal length of the perspective camera

Applications

Self-calibration of omnidirectional cameras froma plane homography:

• input: - plane homography H with a perspective camera recovery of intrinsic parameters of para-catadioptric camera (given by null-vector of H)

H 43x

Plan• Camera models

• Epipolar geometry of hybrid systems

• Applications

Conclusions

• Derivation of matching tensors

• Multi-linear matching relations between perspective, affine and para-catadioptric cameras

• Applications in calibration, self-calibration, motion estimation, 3D reconstruction, ...

• Plane homographies « for the inverse direction » ?

Conclusions

Open questions• Fundamental matrix etc. for hyper-catadioptric cameras ?

• Multi-view 3D reconstruction for hybrid systems

Perspectives• Hybrid trifocal tensors for line images

Mixing Catadioptric and Perspective Cameras

Peter Sturm

INRIA Rhône-AlpesFrance