lecture 03 image formation 2 - davide...
TRANSCRIPT
![Page 1: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/1.jpg)
Lecture 03Image Formation 2
Davide Scaramuzza
http://rpg.ifi.uzh.ch
Institute of Informatics – Institute of Neuroinformatics
1
![Page 2: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/2.jpg)
Lab Exercise 2 - Today afternoon
Room ETH HG E 1.1 from 13:15 to 15:00
Work description: your first camera motion estimator using DLT
2
![Page 3: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/3.jpg)
Goal of today’s lecture• Study the algorithms behind robot-position control and augmented reality
3
![Page 4: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/4.jpg)
Outline of this lecture
• (Geometric) Camera calibration
– PnP problem
• P3P for calibrated cameras
• DLT for uncalibrated cameras
• Omnidirectional cameras
4
![Page 5: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/5.jpg)
Camera
world
3D Landmarks
Image
Perspective from n Points (aka PnP Problem)
• Given known 3D landmarks positions in the world frame and given their image correspondences in the camera frame, determine the 6DOF pose of the camera in the world frame (including the intrinsinc parameters if uncalibrated)
5
![Page 6: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/6.jpg)
Perspective from n Points (aka PnP Problem)
6
Calibrated camera (i.e., instrinc parameters are known)
Uncalibrated camera (i.e., intrinsic parameters unknown)
Works for any 3D point configurations Direct Linear Transform (DLT)
Minimum number of points: 3P3P (Perspective from Three Points)
Minimum number of points: 4 if coplanar
6 if non coplanar
PnP Problem
![Page 7: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/7.jpg)
How Many Points are Enough?
• 1 Point: infinitely many solutions.
• 2 Points: infinitely many solutions, but bounded.
• 3 Points: – (no 3 collinear) finitely many solutions (up to 4).
• 4 Points:– Unique solution
7
![Page 8: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/8.jpg)
1 Point
C
8
![Page 9: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/9.jpg)
2 Points
C
A
B
9
![Page 10: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/10.jpg)
Inscribed Angles are Equal
q
q
q
C
C
C
A B10
![Page 11: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/11.jpg)
C
A
B
CP
LC
LB
LA
qBCqAB
qCA
s1
s3s2
ABBABA LLLLs qcos222
1
A’
B’
C’
Image Plane
BCCBCB LLLLs qcos222
2
ACCACA LLLLs qcos222
3
𝑠1
𝑠2
𝑠3
𝐿𝐴
𝐴
𝐵
𝐶
𝐿𝐵
𝐿𝐶
𝜃𝐴𝐵
𝜃𝐴𝐶
𝜃𝐵𝐶
3 Points
C
𝑠12 = 𝐿𝐵
2 + 𝐿𝐶2 − 2𝐿𝐵𝐿𝐶 cos 𝜃𝐵𝐶
𝑠22 = 𝐿𝐴
2 + 𝐿𝐶2 − 2𝐿𝐴𝐿𝐶 cos 𝜃𝐴𝐶
𝑠32 = 𝐿𝐴
2 + 𝐿𝐵2 − 2𝐿𝐴𝐿𝐵 cos 𝜃𝐴𝐵
From Carnot’s Theorem:
11
![Page 12: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/12.jpg)
Algebraic Approach: reduce to 4th order equation(Fischler and Bolles, 1981)
• It is known that 𝑛 independent polynomial equations, in 𝑛 unknowns, can have no more solutions than the product of their respective degrees. Thus, the system can have a maximum of 8 solutions. However, because every term in the system is either a constant or of second degree, for every real positive solution there is a negative solution.
• Thus, with 3 points, there are at most 4 valid (positive) solutions.
• A 4th point can be used to disambiguate the solutions.
𝑠12 = 𝐿𝐵
2 + 𝐿𝐶2 − 2𝐿𝐵𝐿𝐶 cos 𝜃𝐵𝐶
𝑠22 = 𝐿𝐴
2 + 𝐿𝐶2 − 2𝐿𝐴𝐿𝐶 cos 𝜃𝐴𝐶
𝑠32 = 𝐿𝐴
2 + 𝐿𝐵2 − 2𝐿𝐴𝐿𝐵 cos 𝜃𝐴𝐵
04
4
3
3
2
210 xGxGxGxGG
By defining 𝑥 = 𝐿𝐵/𝐿𝐴, it can be shown that the system can be reduced to a 4th order equation:
12
![Page 13: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/13.jpg)
Application to Monocular Visual Odometry: camera pose estimation from known 3D-2D correspondences
Keyframe 1 Keyframe 2
Initial point cloud New triangulated points
Current frameNew keyframe
13
![Page 14: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/14.jpg)
AR Application: Microsoft HoloLens
14
![Page 15: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/15.jpg)
Outline of this lecture
• (Geometric) Camera calibration
– PnP problem
• P3P for calibrated cameras
• DLT for uncalibrated cameras– From general 3D objects
– From planar grids
• Omnidirectional cameras
15
![Page 16: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/16.jpg)
Camera calibration• Calibration is the process to determine the intrinsic and extrinsic parameters of the
camera model
• A method proposed in 1987 by Tsai consists of measuring the 3D position of 𝑛 ≥ 6 control points on a three-dimensional calibration target and the 2D coordinates of their projection in the image. This problem is also called “Resection”, or “Perspective from 𝒏 Points (PnP)”, or “Camera pose from 3D-to-2D correspondences”, and is one of the most widely used algorithms in Computer Vision and Robotics
• Solution: The intrinsic and extrinsic parameters are computed directly from the perspective projection equation; let’s see how!
Pc
O
u
v
p
XcCZc
Yc
W
Zw
Yw
Xw
= Pw
3D position of control points is assignedin a reference frame specified by the user
16
![Page 17: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/17.jpg)
Camera calibration: Direct Linear Transform (DLT)
Our goal is to compute K, R, and T that satisfy the perspective projection equation (we neglect the radial distortion)
1
1~
~
~
~
w
w
w
Z
Y
X
TRKv
u
w
v
u
p
1100
0
0
~
~
~
3333231
2232221
1131211
0
0
w
w
w
v
u
Z
Y
X
trrr
trrr
trrr
v
u
w
v
u
1
~
~
~
3333231
302330233202231021
301330133201231011
w
w
w
vvvv
uuuu
Z
Y
X
trrr
tvtrvrrvrrvr
tutrurrurrur
w
v
u
17
![Page 18: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/18.jpg)
Camera calibration: Direct Linear Transform (DLT)
Our goal is to compute K, R, and T that satisfy the perspective projection equation (we neglect the radial distortion)
1
1~
~
~
~
w
w
w
Z
Y
X
TRKv
u
w
v
u
p
1100
0
0
~
~
~
3333231
2232221
1131211
0
0
w
w
w
v
u
Z
Y
X
trrr
trrr
trrr
v
u
w
v
u
18
1
~
~
~
34333231
24232221
14131211
w
w
w
Z
Y
X
mmmm
mmmm
mmmm
w
v
u
![Page 19: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/19.jpg)
Camera calibration: Direct Linear Transform (DLT)
Our goal is to compute K, R, and T that satisfy the perspective projection equation (we neglect the radial distortion)
where is the i-th row of M
1
~
~
~
34333231
24232221
14131211
w
w
w
Z
Y
X
mmmm
mmmm
mmmm
w
v
u
1
~
~
~
w
w
w
Z
Y
X
M
w
v
u
T
im19
1
~
~
~
3
2
1
w
w
w
T
T
T
Z
Y
X
m
m
m
w
v
u
![Page 20: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/20.jpg)
Camera calibration: Direct Linear Transform (DLT)
~
~
~
~
3
2
3
1
Pm
Pm
w
vv
Pm
Pm
w
uu
T
T
T
T
Conversion back from homogeneous coordinates to pixel coordinates leads to:
P
20
1
~
~
~
3
2
1
w
w
w
T
T
T
Z
Y
X
m
m
m
w
v
u
0)(
0)(
32
31
i
T
i
T
i
T
i
T
Pmvm
Pmum
![Page 21: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/21.jpg)
By re-arranging the terms, we obtain
For 𝑛 points, we can stack all these equations into a big matrix:
0
0
0
0
0
0
0
0
3
2
1111
111
m
m
m
PvP
PuP
PvP
PuP
T
nn
T
n
T
T
nn
TT
n
TTT
TTT
Camera calibration: Direct Linear Transform (DLT)
0
0
0
0
0)(
0)(
3
2
1
111
111
32
31
m
m
m
PvP
PuP
Pmvm
PmumTTT
TTT
i
T
i
T
i
T
i
T
21
![Page 22: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/22.jpg)
By re-arranging the terms, we obtain
For 𝑛 points, we can stack all these equations into a big matrix:
Camera calibration: Direct Linear Transform (DLT)
0
0
0
0
10000
00001
10000
00001
34
33
32
31
24
23
22
21
14
13
12
11
1
1
1
1
1
1
1
111
1
1
1
1
1
1
1
111
m
m
m
m
m
m
m
m
m
m
m
m
vZvYvXvZYX
uZuYuXuZYX
vZvYvXvZYX
uZuYuXuZYX
n
n
wn
n
wn
n
wn
n
w
n
w
n
w
n
n
wn
n
wn
n
wn
n
w
n
w
n
w
wwwwww
wwwwww
Q (this matrix is known) M (this matrix is unknown)
0MQ
0
0
0
0
0)(
0)(
3
2
1
111
111
32
31
m
m
m
PvP
PuP
Pmvm
PmumTTT
TTT
i
T
i
T
i
T
i
T
22
![Page 23: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/23.jpg)
Minimal solution
• 𝑄(2𝑛×12) should have rank 11 to have a unique (up to a scale) non-zero solution 𝑀
• Each 3D-to-2D point correspondence provides 2 independent equations
• Thus, 5+1
2point correspondences are needed (in practice 6 point correspondences!)
Over-determined solution
• n ≥ 6 points
• A solution is to minimize | 𝑄𝑀 |2 subject to the constraint | 𝑀 |2 = 1.It can be solved through Singular Value Decomposition (SVD). The solution is the eigenvector corresponding to the smallest eigenvalue of the matrix 𝑄𝑇𝑄(because it is the unit vector 𝑥 that minimizes | 𝑄𝑥 |2 = 𝑥𝑇𝑄𝑇𝑄𝑥).
• Matlab instructions:
• [U,S,V] = svd(Q);
• M = V(:,12);
Camera calibration: Direct Linear Transform (DLT)
0MQ
23
![Page 24: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/24.jpg)
Degenerate configurations
1. Points lying on a plane and/or along a single line passing through the center of projection
2. Camera and points on a twisted cubic (i.e., smooth curve in 3D space of degree 3)
Camera calibration: Direct Linear Transform (DLT)
0MQ
𝐶
𝐶
24
![Page 25: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/25.jpg)
Camera calibration: Direct Linear Transform (DLT)
T)|K(RM
3
2
1
333231
232221
131211
0
0
34
24
14
333231
232221
131211
100
0
0
t
t
t
rrr
rrr
rrr
v
u
m
m
m
mmm
mmm
mmm
v
u
25
• Once we have determined M, we can recover the intrinsic and extrinsic parameters by remembering that:
![Page 26: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/26.jpg)
• Once we have determined M, we can recover the intrinsic and extrinsic parameters by remembering that
• However, notice that we are not enforcing the constraint that 𝑅 is orthogonal, i.e., 𝑅 ∙ 𝑅𝑇= 𝐼
• To do this, we can use the so-called QR factorization of 𝑀, which decomposes 𝑀 into a 𝑅 (orthogonal), T, and an upper triangular matrix (i.e., 𝐾)
Camera calibration: Direct Linear Transform (DLT)
T)|K(RM
3333231
302330233202231021
301330133201231011
34
24
14
333231
232221
131211
trrr
tvtrvrrvrrvr
tutrurrurrur
m
m
m
mmm
mmm
mmm
26
![Page 27: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/27.jpg)
Tsai’s (1987) Calibration example1. Edge detection
2. Straight line fitting to the detected edges
3. Intersecting the lines to obtain the image corners (corner accuracy < 0.1 pixels)
4. Use more than 6 points (ideally more than 20) and not all lying on the same plane
27
𝛼𝑢/𝛼𝑣𝛼𝑢 𝑢0 𝑣0
What are the «skew» and «residuals»?
Why is this ratio not 1?
![Page 28: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/28.jpg)
Tsai’s (1987) Calibration example• The original Tsai calibration (1987) used to consider two different focal lengths 𝛼𝑢, 𝛼𝑣
(which means that the pixels are not squared) and a skew factor (𝐾12 ≠ 0, which means the pixels are parallelograms instead of rectangles) to account for possible misalignments (small 𝑥, 𝑦 rotation) between image plane and lens
• Most today’s cameras are well manufactured, thus, we can assume 𝛼𝑢
𝛼𝑣= 1 and 𝐾12 = 0
• What is the residual? The residual is the average “reprojection error” (see Lecture 8). The reprojection error is computed as the distance (in pixels) between the observed pixel point and the camera-reprojected 3D point. The reprojection error gives as a quantitative measure of the accuracy of the calibration (ideally it should be zero).
28
![Page 29: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/29.jpg)
DLT algorithm applied to mutual robot localization
In this case, the camera has been pre-calibrated (i.e., K is known). Can you think of how the DLT algorithm could be modified so that only R and T need to be determined and not K? 29
http://youtu.be/8Ui3MoOxcPQ
![Page 30: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/30.jpg)
Outline of this lecture
• (Geometric) Camera calibration
– PnP problem
• P3P for calibrated cameras
• DLT for uncalibrated cameras– From general 3D objects
– From planar grids
• Omnidirectional cameras
30
![Page 31: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/31.jpg)
Camera calibration from planar grids: homographies
• Tsai’s calibration is based on DLT algorithm, which requires points not to lie on the same plane
• An alternative method (today’s standard camera calibration method) consists of using a planar grid (e.g., a chessboard) and a few images of it shown at different orientations
• This method was invented by Zhang (1999) @Microsoft Research and is implemented in both Matlab and the OpenCV C/C++ library
31
![Page 32: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/32.jpg)
Camera calibration from planar grids: homographies
• Tsai’s calibration is based on DLT algorithm, which requires points not to lie on the same plane
• An alternative method (today’s standard camera calibration method) consists of using a planar grid (e.g., a chessboard) and a few images of it shown at different orientations
• This method was invented by Zhang (1999) @Microsoft Research and is implemented in both Matlab and the OpenCV C/C++ library
32
![Page 33: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/33.jpg)
• Our goal is to compute K, R, and T, that satisfy the perspective projection equation (we neglect the radial distortion)
• Since the points lie on a plane, we have 𝑍𝑤 = 0
1
0
~
~
~
w
w
Y
X
TRK
w
v
u
1
0100
0
0
~
~
~
3333231
2232221
1131211
0
0
w
w
v
uY
X
trrr
trrr
trrr
v
u
w
v
u
Camera calibration from planar grids: homographies
1100
0
0
~
~
~
33231
22221
11211
0
0
w
w
v
u
Y
X
trr
trr
trr
v
u
w
v
u
33
![Page 34: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/34.jpg)
• Our goal is to compute K, R, and T, that satisfy the perspective projection equation (we neglect the radial distortion)
• Since the points lie on a plane, we have 𝑍𝑤 = 0
where is the i-th row of 𝐻
1~
~
~
333231
232221
131211
w
w
Y
X
hhh
hhh
hhh
w
v
u
1~
~
~
w
w
Y
X
H
w
v
u
1~
~
~
3
2
1
w
w
T
T
T
Y
X
h
h
h
w
v
u
T
ih
Camera calibration from planar grids: homographies
This matrix is called Homography
34
![Page 35: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/35.jpg)
~
~
~
~
3
2
3
1
Ph
Ph
w
vv
Ph
Ph
w
uu
T
T
T
T
Conversion back from homogeneous coordinates to pixel coordinates leads to:
where P = (𝑋𝑤, 𝑌𝑤 , 1)𝑇
0)(
0)(
32
31
i
T
i
T
i
T
i
T
Phvh
Phuh
Camera calibration from planar grids: homographies
1~
~
~
3
2
1
w
w
T
T
T
Y
X
h
h
h
w
v
u
35
![Page 36: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/36.jpg)
By re-arranging the terms, we obtain
For 𝑛 points, we can stack all these equations into a big matrix:
0
0
0
0
0
0
0
0
3
2
1111
111
h
h
h
PvP
PuP
PvP
PuP
T
nn
T
n
T
T
nn
TT
n
TTT
TTT
0)(
0)(
32
31
i
T
i
T
i
T
i
T
Phvh
Phuh
Camera calibration from planar grids: homographies
0HQ
H (this matrix is unknown)Q (this matrix is known)
0
0
0
0
3
2
1
1
1
h
h
h
PvP
PuPT
i
T
i
T
T
i
TT
i
00
00
321
321
TT
ii
TT
i
T
TT
ii
TT
i
hPvhPh
hPuhhP
36
![Page 37: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/37.jpg)
Minimal solution
• 𝑄(2𝑛×9) should have rank 8 to have a unique (up to a scale) non-trivial solution 𝐻
• Each point correspondence provides 2 independent equations
• Thus, a minimum of 4 non-collinear points is required
Over-determined solution
• n ≥ 4 points
• It can be solved through Singular Value Decomposition (SVD) (same considerations as before)
Solving for K, R and T• H can be decomposed by recalling that
0HQ
Camera calibration from planar grids: homographies
33231
22221
11211
0
0
333231
232221
131211
100
0
0
trr
trr
trr
v
u
hhh
hhh
hhh
v
u
37
![Page 38: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/38.jpg)
How to recover K, R, T from H?
1. Estimate the homography 𝐻𝑖 for each view, using the DLT algorithm.
2. Determine the intrinsics K of the camera from a set of homographies:
1. Each homography 𝐻𝑖 ∼ 𝐾 𝒓1, 𝒓2, 𝒕 provides two linear equations in the 6 entries of the matrix B ≔ 𝐾−⊤𝐾−1. Letting 𝒘1 ≔ 𝐾𝒓1, 𝒘2 ≔ 𝐾𝒓2, the rotation constraints 𝒓1⊤𝒓1 = 𝒓2
⊤𝒓2 = 1 and 𝒓1⊤𝒓2 = 0 become
𝒘1⊤𝐵𝒘1 −𝒘2
⊤𝐵𝒘2 = 0 and 𝒘1⊤𝐵𝒘2 = 0.
2. Stack 2N equations from N views, to yield a linear system 𝐴𝒃 = 𝟎. Solve for b (i.e., B) using the Singular Value Decomposition (SVD).
3. Use Cholesky decomposition to obtain K from B.
3. The extrinsic parameters for each view can be computed using K:𝒓1 ∼ 𝜆𝐾−1𝐻𝑖 : , 1 , 𝒓2 ∼ 𝜆𝐾−1𝐻𝑖 : , 2 , 𝒓3 = 𝒓1 × 𝒓2 and 𝑇𝑖 = 𝜆𝐾−1𝐻𝑖 : , 3 , with 𝜆 = 1/𝐾−1𝐻𝑖 : , 1 . Finally, build 𝑅𝑖 = (𝒓1, 𝒓2, 𝒓3) and enforce rotation matrix constraints.
Won’t be asked at the exam
38
![Page 39: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/39.jpg)
Types of 2D Transformations
This transformation is called Homography
39
![Page 40: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/40.jpg)
Application of calibration from planar grids• Today, there are thousands of application of this algorithm:
– Augmented reality
Most AR apps use AprilTag or ARuco Markers40
![Page 41: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/41.jpg)
Application of calibration from planar grids
Marc Pollefeys’ labMy lab
• Today, there are thousands of application of this algorithm:
– Augmented reality
– Robotics (beacon-based localization)
• Do we need to know the metric size of the tag? – For Augmented Reality?
– For Robotics?
41Most AR apps use AprilTag or ARuco Markers
![Page 42: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/42.jpg)
If the camera is calibrated, only R and T need to be determined. In this case, should we use DLT (linear system of equations) or PnP (non linear)?
Lepetit, Moreno Noguer, Fua, EPnP: An Accurate O(n) Solution to the PnP Problem, IJCV’09
DLT vs PnP: Accuracy vs noise
42
![Page 43: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/43.jpg)
DLT vs PnP: Accuracy vs number of points
If the camera is calibrated, only R and T need to be determined. In this case, should we use DLT (linear system of equations) or PnP (non linear)?
Lepetit, Moreno Noguer, Fua, EPnP: An Accurate O(n) Solution to the PnP Problem, IJCV’0943
![Page 44: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/44.jpg)
DLT vs PnP: Timing
Lepetit, Moreno Noguer, Fua, EPnP: An Accurate O(n) Solution to the PnP Problem, IJCV’0944
![Page 45: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/45.jpg)
An Efficient Algebraic Solution to P3P
Similarly to Kneip (CVPR’11) and Maselli (ICPR’14), directly solve for the camera’s pose (not the distances).
1. Eliminate the camera’s position and the features’ distances to yield a system of 3 equations in the camera’s orientation alone.
2. Successively eliminate two of the unknown 3-DOFs (angles) algebraically and arrive at a quartic polynomial equation.
• Results: outperforms previous methods in terms of speed, accuracy, and robustness to close-to-singular cases.
• Available in OpenCV > 3.3. solvePnP() with SOLVEPNP_AP3P
3 points Pairwise subtraction and dot product
Ke and Roumeliotis (CVPR’17)
45
Won’t be asked at the exam
![Page 46: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/46.jpg)
Recap
46
Calibrated camera (i.e., instrinc parameters are known)
Uncalibrated camera (i.e., intrinsic parameters unknown)
Works for any 3D point configuration DLT (Direct Linear Transform)
Minimum number of points: 4 (i.e., 3+1)P3P (Perspective from Three Points (plus 1))
Minimum number of points: 4 if coplanar
6 if non coplanar
PnP Problem
![Page 47: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/47.jpg)
Outline of this lecture
• (Geometric) Camera calibration
– PnP problem
• P3P for calibrated cameras
• DLT for uncalibrated cameras– From general 3D objects
– From planar grids
• Omnidirectional cameras
47
![Page 48: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/48.jpg)
Overview on Omnidirectional Cameras
![Page 49: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/49.jpg)
Example scene viewed by three different camera models
Zhang et al., Benefit of Large Field-of-View Cameras for Visual Odometry, ICRA’16http://rpg.ifi.uzh.ch/fov.html 49
Perspective Fisheye Catadioptric
![Page 50: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/50.jpg)
Catadioptric Cameras
50
![Page 51: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/51.jpg)
Catadioptric Cameras
51
![Page 52: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/52.jpg)
Non Central Catadioptric cameras
52
Rays do not intersect in a single point
![Page 53: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/53.jpg)
Camera Models
single effective viewpoint
Central Catadioptric cameras
53
• Rays do not intersect in a single point• Mirror must be surface of revolution of a conic
![Page 54: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/54.jpg)
Hyperbola
+
Perspective camera
Parabola
+
Orthographic lens
Types of central catadioptric cameras
F1
F2
F1
Central Catadioptric cameras
54
NB: one of the foci of the hyperbola must lie in the camera’s center of projection
![Page 55: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/55.jpg)
55
Because we can:
• Apply standard algorithms valid for perspective geometry.
• Unwarp parts of an image into a perspective one
• Transform image points into normalized vectors on the unit sphere
Why do we prefer central cameras?
![Page 56: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/56.jpg)
𝛼
p
• We describe the image projection function by means of a polynomial, whose coefficients are the parameters to be estimated
• The coefficients, intrinsics, and extrinsics are then found via DLT
N
Naaag ... 2
21
1
T |
)(
0
0
w
w
w
Z
Y
X
R
g
vv
uu
p
2
0
2
0 )()( vvuu
Scaramuzza, Martinelli, Siegwart, A toolbox for easily calibrating omnidirectional cameras, IROS’06. PDFScaramuzza, Omnidirectional Camera, chapter of Encyclopedia of Computer Vision, Springer’14. PDF
Unified Omnidirectional Camera Model
When 𝑎𝑖 = 0 then we get a pinhole camera
![Page 57: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/57.jpg)
OcamCalib: Omnidirectional Camera Calibration
• Released in 2006, it is the standard toolbox for calibrating wide angle cameras
• Original url: https://sites.google.com/site/scarabotix/ocamcalib-toolbox
• Since 2015, included in the Matlab Computer Vision Toolbox: https://ch.mathworks.com/help/vision/ug/fisheye-calibration-basics.html
D. Scaramuzza, A. Martinelli, R. Siegwart, A toolbox for easily calibrating omnidirectional cameras, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006. PDF. 57
Example calibration images of a catadioptric camera Example calibration images of a fisheye camera
![Page 58: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/58.jpg)
Equivalence between Perspective and Omnidirectional model
58
![Page 59: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/59.jpg)
Measures the ray direction (angles).
59
Equivalence between Perspective and Omnidirectional model
![Page 60: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/60.jpg)
Measures the ray direction (angles).
60
Equivalence between Perspective and Omnidirectional model
![Page 61: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/61.jpg)
61
Pc
C
u
v
p
Zc
f
Image plane
C = optical center = center of the lens
Image plane
O = principal point
Zc = optical axis
Equivalence between Perspective and Omnidirectional model
![Page 62: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/62.jpg)
Representation of image points on the unit sphereAlways possible after the camera has been calibrated!
62
![Page 63: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/63.jpg)
Summary (things to remember)
• P3P and PnP problems
• DLT algorithm
• Calibration from planar grid (Homography algorithm)
• Readings: Chapter 2.1 of Szeliski book
• Omnidirectional cameras
– Central and non central projection
– Dioptric
– Catadioptric (working principle of conic mirrors)
• Unified (spherical) model for perspective and omnidirectional cameras
• Reading: Chapter 4 of Autonomous Mobile Robots book: link
63
![Page 64: Lecture 03 Image Formation 2 - Davide Scaramuzzarpg.ifi.uzh.ch/docs/teaching/2019/03_image_formation_2.pdf · Lecture 03 Image Formation 2 ... • Given known 3D landmarks positions](https://reader033.vdocuments.us/reader033/viewer/2022042311/5eda195cb3745412b570c300/html5/thumbnails/64.jpg)
Understanding Check
Are you able to:
• Describe the general PnP problem and derive the behavior of its solutions?
• Explain the working principle of the P3P algorithm?
• Explain and derive the DLT? What is the minimum number of point correspondences it requires?
• Define central and non central omnidirectional cameras?
• What kind of mirrors ensure central projection?
64