geometric camera calibrationjjcorso/t/598f14/files/... · 2016-09-07 · plan • review...
TRANSCRIPT
Geometric Camera Calibration
Instructor: Jason Corso (jjcorso)!web.eecs.umich.edu/~jjcorso/t/598F14!
!
Materials on these slides have come from many sources in addition to myself; I am infinitely grateful to these, especially Greg Hager, Silvio Savarese, and Steve Seitz.!
EECS 598-08 Fall 2014!Foundations of Computer Vision!!
Readings: FP 1.3; SZ 6.3 (FL 4.6; extra notes) !Date: 9/17/14!!
Plan
• Review Perspective Projection!• Geometric Camera Calibration!
– Indirect camera calibration!• Solve for projection matrix then the parameters!
– Direct camera calibration!– Multi-planes method!
• Example with the Matlab Toolbox!• Catadioptric Sensing!
– Different slide-deck. (See Chris Geyer’s CVPR 2003 Tutorial)!
• Other calibration methods not covered!– Vanishing points-based method (see SZ)!– Self-calibration!
2
Projection equation
• The projection matrix models the cumulative effect of all parameters • Useful to decompose into a series of operations
ΠXx =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
1************
ZYX
ssysx
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−
=110100
00100001
100'0'0
31
1333
31
1333
x
xx
x
xxcy
cx
yfsxfs
000 TIRΠ
projection intrinsics rotation translation
identity matrix
Camera parameters A camera is described by several parameters
• Translation T of the optical center from the origin of world coords • Rotation R of the image plane • focal length , principle point (x’c, y’c), pixel size (sx, sy) • blue parameters are called “extrinsics,” red are “intrinsics”
• The definitions of these parameters are not completely standardized – especially intrinsics—varies from one book to another Source: S Seitz slides.!
3
Oc
focal length:
Source: S Savarese slides.!
Projective Camera 4
Projective Camera: The Normalized Image Plane 5
• The normalized image plane is parallel to the physical retina (e.g., ccd) but located at unit distance ( = 1) from the pinhole.!
Image Source: Forsyth and Ponce Book.!
Projective Camera: The Normalized Image Plane
• Physical pixels in the retina (e.g. ccd) may not be square, so we have two additional scale parameters.!
• Units:!– is a distance expressed in meters!– A pixel will have dimensions where and are in!
• Can replace dependent pixel parameters !
6
Source: S Savarese slides.!
Projective Camera 7
focal length: image center-point:
Source: S Savarese slides.!
Projective Camera 8
focal length: image center-point: non-square pixels: skew angle:
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
10100000
zyx
vus
P' o
o
β
α
K has 5 degrees of freedom! Source: S Savarese slides.!
Projective Camera 9
focal length: image center-point: non-square pixels: skew angle:
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=ʹ′
1zyx
01000v0
0ucot
P o
o
sinθβ
θαα
K has 5 degrees of freedom! Source: S Savarese slides.!
Projective Camera 10
focal length: image center-point: non-square pixels: skew angle:
Ow
iw
kw
jw
Source: S Savarese slides.!
Projective Camera 11
focal length: image center-point: non-square pixels: skew angle: rotation, translation:
Ow
iw
kw
jw
Internal parameters External parameters
Source: S Savarese slides.!
Projective Camera 12
focal length: image center-point: non-square pixels: skew angle: rotation, translation:
Properties of Pinhole Perspective Projection
• Distant objects appear smaller!• Points project to points!• Lines project to lines!
• Angles are not preserved.!• Parallel lines meet!!
13
Source: S. Savarese slides.!
Vanishing Point!
wPMP =ʹ′ [ ] wPTRK=Internal parameters
External parameters
Source: S Savarese slides.!
Projective Camera 14
Projective Camera 15
Internal parameters External parameters
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=
100v0
ucot
K o
o
sinθβ
θαα
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=T3
T2
T1
Rrrr
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
z
y
x
ttt
T
43×
Projective Camera 16
Estimate intrinsic and extrinsic parameters from 1 or multiple images
Change notation: P = Pw
p = P’ ⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=
100v0
ucot
K o
o
sinθβ
θαα
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=T3
T2
T1
Rrrr
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
z
y
x
ttt
T
43×
Source: S Savarese slides.!
Goal of Calibration 17
• P1… Pn with known positions in [Ow,iw,jw,kw] • p1, … pn known positions in the image Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
Source: S Savarese slides.!
The Calibration Problem 18
• P1… Pn with known positions in [Ow,iw,jw,kw] • p1, … pn known positions in the image Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
image!
Source: S Savarese slides.!
The Calibration Problem 19
jC
Calibration rig
How many correspondences do we need? • M has 11 unknown • We need 11 equations • 6 correspondences would do it
image!
Source: S Savarese slides.!
The Calibration Problem 20
image!jC
Calibration rig
In practice, using more than 6 correspondences enables more robust results
Source: S Savarese slides.!
The Calibration Problem 21
jC
ii PMP → ⎥⎦
⎤⎢⎣
⎡=→
i
ii v
up
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
3
2
1
Mmmm
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
mmmm
in pixels Source: S Savarese slides.!
The Calibration Problem 22
i3
i1i P
Pumm
=
i2i3i P)P(v mm =→
i1i3i P)P(u mm =→
i3
i2i P
Pvmm
=
⎥⎦
⎤⎢⎣
⎡
i
i
vu
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
mmmm
0P)P(v 23 =−→ iii mm
0P)P(u 13 =−→ iii mm
Source: S Savarese slides.!
The Calibration Problem 23
…
…
0)( 12131 =− PPv mm
0)( 11131 =− PPu mm
0)( 23 =− iii PPv mm
0)( 13 =− iii PPu mm
0)( 23 =− nnn PPv mm
0)( 13 =− nnn PPu mm
Source: S Savarese slides.!
The Calibration Problem 24
⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡=
2221
1211
2221
1211
BBBB
BAAAA
A
What is AB ?
⎥⎦
⎤⎢⎣
⎡
++
++=
2222122121221121
2212121121121111
BABABABABABABABA
AB
Source: S Savarese slides.!
Block Matrix Multiplication 25
2n x 12 12x1
1x4
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
=T3
T2
T1def
mmm
m
4x1
…
Homogenous linear system
known unknown
0)( 12131 =+− PPv mm
0)( 11131 =+− PPu mm
0)( 23 =+− nnn PPv mm
0)( 13 =+− nnn PPu mm
Source: S Savarese slides.!
The Calibration Problem 26
P! m 0 =
Rectangular system (M>N)
• 0 is always a solution
Minimize |P m|2
under the constraint |m|2 =1
M=number of equations = 2n N=number of unknown = 11
• To find non-zero solution
Source: S Savarese slides.!
Homogeneous M x N Linear Systems 27
• How do we solve this homogenous linear system?
• Using DLT (Direct Linear Transformation) algorithm via SVD decomposition!
Source: S Savarese slides.!
The Calibration Problem 28
12
1
1
.−−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=Λ= SSSSA
Nλ
λ
λ
Eigenvectors of A are columns of S
[ ]NS vv1=
Eigendecomposition
Source: S Savarese slides.!
Eigenvalues and Eigenvectors 29
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=Σ
Nσ
σ
σ
.2
1
1−Σ= VUA
ii λσ = σ = singular value λ = eigenvalue of At A
U, V = orthogonal matrix
Source: S Savarese slides.!
Singular Value Decomposition 30
Properties of SVD
• Recall the singular values of a matrix are related to its rank.!• Recall that Ax = 0 can have a nonzero x as solution only if A
is singular.!• Finally, note that the matrix V of the SVD is an orthogonal
basis for the domain of A; in particular the zero singular values are the basis vectors for the null space.!
• Putting all this together, we see that A must have rank 7 (in this particular case) and thus x must be a vector in this subspace. !
• Clearly, x is defined only up to scale.!
31
Source: G Hager slides.!
ix ixʹ′
H
0Ai =hFunction of measurements
unknown
ii xHx =ʹ′Source: S Savarese slides.!
DLT algorithm (Direct Linear Transformation) 32
1212T
121212n2 VDU ×××
Last column of V gives m
M iPM ip→
SVD decomposition of P
Why? See pag 593 of AZ
Source: S Savarese slides.!
The Calibration Problem 33
Tnnnnnmnm VDUP ×××× =
Tnnnmmmnm VDUP ×××× =
Has n orthogonal columns
Orthogonal matrix
• This is one of the possible SVD decompositions • This is typically used for efficiency • The classic SVD is actually:
• Thanks to Pat O’ Keefe!!
orthogonal Orthogonal Source: S Savarese slides.!
Clarification about SVD 34
Estimated values
Intrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 35
Source: S Savarese slides.!
Theorem (Faugeras, 1993) 36
Estimated values
Intrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 37
Estimated values
Extrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 38
Degenerate cases
• Pi’s cannot lie on the same plane! • Points cannot lie on the intersection curve of two quadric surfaces
Source: S Savarese slides.!
39
Taking lens distortions into account • Chromatic Aberration!• Spherical aberration !• Radial Distortion!
Source: S Savarese slides.!
40
No distortion
Pin cushion
Barrel
Dealing with Radial Distortion As Well
– Caused by imperfect lenses!– Deviations are most noticeable for rays that pass through
the edge of the lens!
Source: S Savarese slides.!
41
Issues with lenses: Radial Distortion
Pin cushion
Barrel (fisheye lens) Source: S Savarese slides.!
42
Source: S Savarese slides.!
Radial Distortion 43
Image magnification in(de)creases with distance from the optical center
ii
ii p
vu
PM
100
00
00
1
1
=⎥⎦
⎤⎢⎣
⎡→
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
λ
λd
v
vucvbuad 222 ++=
u
∑±==
3
1p
2ppdκ1λ
Polynomial function
Distortion coefficient
To model radial behavior
Source: S Savarese slides.!
Radial Distortion 44
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
3
2
1
Qqqq
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
ii
ii p
vu
PM
100
00
00
1
1
=⎥⎦
⎤⎢⎣
⎡→
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
λ
λ
Q
⎩⎨⎧
=
=
PPvPPu
2i3i
i1i3i
qqqq
Is this a linear system of equations?
No! why? Source: S Savarese slides.!
Radial Distortion 45
)(PfX =
measurement parameter
f( ) is nonlinear
- Newton Method - Levenberg-Marquardt Algorithm
• Iterative, starts from initial solution • May be slow if initial solution far from real solution • Estimated solution may be function of the initial solution • Newton requires the computation of J, H • Levenberg-Marquardt doesn’t require the computation of H
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 46
A possible algorithm 1. Solve linear part of the system to find approximated solution 2. Use this solution as initial condition for the full system 3. Solve full system using Newton or L.M.
)(PfX =
measurement parameter
f( ) is nonlinear
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 47
Typical assumptions:
- zero-skew, square pixel - uo, vo = known center of the image - no distortion
Just estimate f and R, T
)(PfX =
measurement parameter
f( ) is nonlinear
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 48
Can estimate m1 and m2 and ignore the radial distortion?!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
d
v
u Hint: slopevu
i
i =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
Source: S Savarese slides.!
Radial Distortion 49
Estimating m1 and m2…!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
0)()( 121111 =− PuPv mm0)()( 21 =− iiii PuPv mm
0)()( 21 =− nnnn PuPv mm…
0Q =n ⎥⎦
⎤⎢⎣
⎡=
2
1
mm
n
Tsai technique [87]
i
i
i
i
i
i
i
i
PP
PPPP
vu
2
1
3
2
3
1
)()()()(
mm
mmmm
==
Source: S Savarese slides.!
Radial Distortion 50
Once that m1 and m2 are estimated…!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
3m is non linear function of 1m 2m λ
There are some degenerate configurations for which m1 and m2 cannot be computed
, ,
Source: S Savarese slides.!
Radial Distortion 51
Direct Calibration: The Algorithm
1. Compute image center from orthocenter!2. Compute the Intrinsic matrix (6.8)!3. Compute solution with SVD!4. Compute gamma and alpha!5. Compute R (and normalize)!6. Compute fx and and Tz!
Source: G Hager slides.!
52
Basic Equations
Source: G Hager slides.!
53
Basic Equations
Source: G Hager slides.!
54
Basic Equations
one of these for each point!
Source: G Hager slides.!
55
Basic Equations
Source: G Hager slides.!
56
Properties of SVD Again
• Recall the singular values of a matrix are related to its rank.!• Recall that Ax = 0 can have a nonzero x as solution only if A is singular.!• Finally, note that the matrix V of the SVD is an orthogonal basis for the domain of
A; in particular the zero singular values are the basis vectors for the null space.!
• Putting all this together, we see that A must have rank 7 (in this particular case) and thus x must be a vector in this subspace. !
• Clearly, x is defined only up to scale.!
Source: G Hager slides.!
57
Basic Equations
We now know Rx and Ry up to a sign and gamma.!Rz = Rx x Ry!!We will probably use another SVD to orthogonalize!this system (R = U D V’; set D to I and multiply).!
Source: G Hager slides.!
58
Last Details about Direct Calibration
• We still need to compute the correct sign.!– note that the denominator of the original equations must be
positive (points must be in front of the cameras)!– Thus, the numerator and the projection must disagree in sign.!– We know everything in numerator and we know the projection,
hence we can determine the sign.!
• We still need to compute Tz and fx!– we can formulate this as a least squares problem on those two
values using the first equation.!
Source: G Hager slides.!
59
Self-Calibration
• Calculate the intrinsic parameters solely from point correspondences from multiple images.!
• Static scene and intrinsics are assumed.!• No expensive apparatus.!• Highly flexible but not well-established.!• Projective Geometry – image of the absolute conic.!
Source: G Hager slides.!
Multi-Plane Calibration
• Hybrid method: Photogrammetric and Self-Calibration.!• Uses a planar pattern imaged multiple times (inexpensive).!• Used widely in practice and there are many
implementations.!• Based on a group of projective transformations called
homographies.!
• m be a 2d point [u v 1]’ and M be a 3d point [x y z 1]’.!
• Projection is !
Source: G Hager slides.!
Planar Homographies
• First Fundamental Theorem of Projective Geometry:!– There exists a unique homography that performs a change of basis
between two projective spaces of the same dimension.!
– Projection Becomes !
– Notice that the homography is defined up to scale (s).!
Source: G Hager slides.!
Computing the Intrinsics
• We know that!
• From one homography, how many constraints on the intrinsic parameters can we obtain? !– Extrinsics have 6 degrees of freedom.!– The homography has 8 degrees of freedom.!– Thus, we should be able to obtain 2 constraints per
homography.!
• Use the constraints on the rotation matrix columns…!
Source: G Hager slides.!
Computing Intrinsics
• Rotation Matrix is orthonormal:!
• Write the homography in terms of its columns…!
Source: G Hager slides.!
Computing Intrinsics
• Derive the two constraints:!
Source: G Hager slides.!
Closed-Form Solution
• Notice is symmetric, 6 parameters can be written as a vector b.!• From the two constraints, we have!
• Stack up n of these for n images and build a 2n*6 system.!• Solve with SVD (yet again). !• Extrinsics “fall-out” of the result easily.!
Source: G Hager slides.!
Non-linear Refinement
• Closed-form solution minimized algebraic distance.!• Since full-perspective is a non-linear model!
– Can include distortion parameters (radial, tangential)!– Minimize squared distance with a non-linear method.!
Source: G Hager slides.!
Camera Calibration Toolbox for Matlab!J. Bouguet – [1998-2000]! !http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples!
Source: S Savarese slides.!
Example Calibration Procedure 68
Source: S Savarese slides.!
Example Calibration Procedure 69
Source: S Savarese slides.!
Example Calibration Procedure 70
Source: S Savarese slides.!
Example Calibration Procedure 71
Source: S Savarese slides.!
Example Calibration Procedure 72
Source: S Savarese slides.!
Example Calibration Procedure 73
Source: S Savarese slides.!
Example Calibration Procedure 74
Source: S Savarese slides.!
Example Calibration Procedure 75
Next Lecture: Photometric and Radiometric Aspects
• Reading: FP 2, 3; SZ 2.2, 2.3!
76