Mixing Catadioptric and Perspective Cameras
Peter Sturm
INRIA Rhône-AlpesFrance
Introduction
Introduction
Introduction• Existing results
- epipolar geometry between omnidirectional cameras- motion estimation- (self-) calibration- ...
Our Goals• Study the geometry of hybrid stereo systems (omnidirectional and perspective cameras)
• Applications
- motion estimation
- epipolar geometry- trifocal tensors- plane homographies
- calibration- self-calibration, calibration transfer- 3D reconstruction- ...
Plan Camera models
• Epipolar geometry of hybrid systems
• Applications
• Conclusions
• Derivation of matching tensors
Camera ModelsPerspective and affine cameras
Camera ModelsCentral catadioptric cameras
• mirror (surface of revolution of a conic)
• camera
• virtual optical center
Camera ModelsCentral catadioptric cameras
• calibration
• mirror (surface of revolution of a conic)
• camera
• virtual optical center
Types of central catadioptric cameras• hyperbola + perspective camera
• parabola + affine camera
• ...
Camera Models
Plan• Camera models
Epipolar geometry of hybrid systems
• Applications
• Conclusions
• Derivation of matching tensors
Epipolar Geometry
epipolar line
epipolar plane
epipolar line
q
q'Fd 3
T
333~
x
d
qFd' 3333~
x
d’q’
epipole
Epipolar Geometry
epipolar conic
Epipolar Geometry
epipolar conic
epipolar line
epipole
epipoleepipoles
Epipolar Geometry Example
Epipolar Geometry
q
qFd' 3333~
x
q'Fd 3
T
333~
x
d d’q’
purely perspective case
0Fqq'T
epipolar conic
epipolar line
epipole
epipoles
“ conic ~ F q’ “
There exists withF 36x
q'Fc 3366~
xconcretely:
Interpretation:
0q2
22
q365
624
541T
ccccccccc
Epipolar Geometry
epipolar conic
epipolar line
epipole
epipoles
06325314213
2
32
2
21
2
1 cqqcqqcqqcqcqcq
0
6
5
4
3
2
1
323121
2
3
2
2
2
1
cccccc
qqqqqqqqq
“lifted coordinates”q̂
0Fq'q̂T
“ conic ~ F q’ “
There exists withF 36x
q'Fc 3366~
xconcretely:
Interpretation:
0q2
22
q365
624
541T
ccccccccc
Epipolar Geometry Example
Epipolar Geometry
BUT...
Until now, linear epipolar relation only found for:• any combination of perspective, affine or para-catadioptric cameras (parabolic mirrors)
Not yet for:• other omnidirectional cameras than para-catadioptric ones (e.g. based on hyperbolic mirrors)
Epipolar Geometry
Special case:
• combination of perspective and para-catadioptric cameras
• epipolar conics are circles
• F is of dimension 4x3
qqqqqqq
32
31
2
3
2
2
2
1
• the « lifted coordinates » are:
Epipolar Geometry
Epipoles:
• is of rank 2
• The epipole of the perspective camera is the right null-vector of F
• F has a one-dimensional left null-space the two epipoles of the catadioptric camera are the left null-vectors that are valid lifted coordinates (quadratic constraint):
0~ q̂q̂q̂q̂q̂ 2
4
2
321
32
31
2
3
2
2
2
1
qqqqqqq
F 34x
Plan• Camera models
• Epipolar geometry of hybrid systems
• Applications
• Conclusions
Derivation of matching tensors
Matching Tensors
• Multi-linear relations between coordinates of correponding image points
• Purely perspective case: derivation based on linear equations representing projections (3D 2D)
• Here: equations for back-projection (2D 3D directions)
- perspective cameras
1
-Q
tIRKq PPPPP
DtQ PPP
qKRD P
-1
P
T
PPwith
qP
tP
Q
Matching Tensors• Linear equations representing back-projections
- perspective cameras
qKRtQP
-1
P
T
PPP
- para-catadioptric cameras
q̂BRtQCC
T
CCC
• is of dimension 3x4• it depends - on the mirror’s intrinsic parameters - on the affine camera’s intrinsic parameters
BC
tC
Q
qC
Matching Tensors• Putting the equations together
qKRtQP
-1
P
T
PPP
q̂BRtQCC
T
CCC
0
0
Q
1
Iq̂BR0t
I0qKRt
6
6
66C
P
33CCTCC
33P1-P
TPP
x
x
x
0QqKRt P
-1
P
T
PPP
0Qq̂BRt CC
T
CCC
Matching Tensors• Putting the equations together
0
0
Q
1
Iq̂BR0t
I0qKRt
6
6
66C
P
33CCTCC
33P1-P
TPP
x
x
x
This matrix has a kernel
its rank is lower than 6
its determinant is zero
bilinear equation on the coefficients of andqP
q̂C
0qFq̂PC
T
Matching Tensors• Straightforward extension to more than 2 views ...
0
0
0
Q
'
1
Iq̂BR00t
I0q'K'R'0t'
I00qKRt
9
7
79C
P
P
33CCTCC
33P1-P
TPP
33P1-P
TPP
x
x
x
x
Plan• Camera models
• Epipolar geometry of hybrid systems
• Derivation of matching tensors
Applications
• Conclusions
- Self-calibration of omnidirectional cameras from fundamental matrices- Calibration transfer from an omnidirectional to a perspective camera- Self-calibration of omnidirectional cameras from a plane homography
Applications
Self-calibration of omnidirectional cameras fromfundamental matrices:
• para-catadioptric camera has 3 intrinsic parameters
self-calibration is possible from two or more fundamental matrices
• representation as a 4-vector of homogeneous coordinates
• this vector is in the left null-space of the fundamental matrix of this camera, defined with respect to any other camera (perspective, affine, catadioptric)
Applications
Calibration transfer from an omnidirectional to aperspective camera:
• input: - calibration of a para-catadioptric camera - fundamental matrix with a perspective camera
closed-form solution for the focal length of the perspective camera
Applications
Self-calibration of omnidirectional cameras froma plane homography:
• input: - plane homography H with a perspective camera recovery of intrinsic parameters of para-catadioptric camera (given by null-vector of H)
H 43x
Plan• Camera models
• Epipolar geometry of hybrid systems
• Applications
Conclusions
• Derivation of matching tensors
• Multi-linear matching relations between perspective, affine and para-catadioptric cameras
• Applications in calibration, self-calibration, motion estimation, 3D reconstruction, ...
• Plane homographies « for the inverse direction » ?
Conclusions
Open questions• Fundamental matrix etc. for hyper-catadioptric cameras ?
• Multi-view 3D reconstruction for hybrid systems
Perspectives• Hybrid trifocal tensors for line images
Mixing Catadioptric and Perspective Cameras
Peter Sturm
INRIA Rhône-AlpesFrance