camera calibrationcis580/spring2016/... · intrinsic bias • a scale factor that converts physical...
TRANSCRIPT
![Page 1: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/1.jpg)
Camera Calibration
Machine Vision (CIS 580)
![Page 2: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/2.jpg)
![Page 3: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/3.jpg)
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
![Page 4: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/4.jpg)
Why Do We Calibrate?
• To find an intrinsic bias produced by lens configuration and manufacture imprecision. • The intrinsic bias is often fixed as long as the focal length
of the camera does not change. • Pre-calibration of the intrinsic bias reduces the number of
unknowns in estimation of camera parameters, e.g., structure from motion.
• The intrinsic bias encoded in an image can serve as a fingerprint of the camera.
![Page 5: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/5.jpg)
Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e.,
f (mm) fx (pixel).
• Position of image center (principal point), i.e., px, py
• A skew factor between x and y axis of the image, i.e., uimg=fx x+sy+px.
• Lens distortion
f (mm)
fx (pixel)
![Page 6: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/6.jpg)
Single View Calibration
![Page 7: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/7.jpg)
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
![Page 8: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/8.jpg)
Ideal Perspective Projection
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
• Estimating K from 2D measurements, x, without assumption is an ill-posed problem because the number of unknowns are greater than the number of equations, i.e.,
• # of unknowns: 5 (K) + 3 (R) + 3 (t) + 3 (X) • # of equations: 2
![Page 9: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/9.jpg)
Single View
[ ]×
=
= I 0x x x x
y y y y
Xu f s p f s p X
Yz v f p f p Y
ZZ
camimg cam
camimg 3 3 cam
camcam1 1 1
1
• Without loss of generality, we can reduce the number of unknowns by setting . If Xcam is known, the intrinsic parameters can be estimated using multiple Xcam.
• # of unknowns: 5 (K) • # of equations: 2n
[ ]×P = K I 03 3
![Page 10: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/10.jpg)
Lines of Projections
u
v x
X−K x1
[ ]×
=
Xx = K I 0 KXz 3 3 1
−K x = Xz 1
−K x1 defines a ray emitting from the origin (the camera center) towards X.
[ ]×P = K I 03 3
![Page 11: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/11.jpg)
Lines of Projections
u
v x1
[ ]×P = K I 03 3
X−K x1
x2θ
2
− −
− − − −
− −
− − − −=
K x K x
K x K x K x K x
x K K x
x K K x x K K x
1 T 11 2
1 T 1 1 T 11 1 2 2
T T 11 2
T T 1 T T 11 1 2
( ) ( )cos
( ) ( ) ( ) ( )
( )
( ) ( )
θ =
[ ]×
=
Xx = K I 0 KXz 3 3 1
−K x = Xz 1
−K x1 defines a ray emitting from the origin (the camera center) towards X.
Angle between two rays:
![Page 12: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/12.jpg)
Angle Between Vanishing Points
[ ] ∞
∞
=
=
x K 0 X
KX1 1
1
∞X2
∞=x KX2 2θ
′θ
∞X1
′θ = θKey observation: the angle between two lines in 3D is the same as the angle between two rays pointing at the vanishing points of the two lines, i.e., .
2
− −
− − − −=
x K K x
x K K x x K K x
T T 11 2
T T 1 T T 11 1 2
( )cos
( ) ( )θ =
Result: if we know angle between two lines in 3D, we can estimate K via
Projection of ∞X1
X is inhomogeneous coordinate of . X
![Page 13: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/13.jpg)
2D Case
∞X2
∞=x KX2 2
θ′θ
∞X1
′θ = θ
Camera center at O
f
∞=x KX1 1
p
Focal plane
L1 ′L1
L2
′L2
L1 ′L1 L2′L2because and
![Page 14: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/14.jpg)
2D Case Wrong Focal Length
∞X2
∞=x KX2 2
θ
′θ
∞X1
′≠θ θ
Camera center at O
f
∞=x KX1 1
p
![Page 15: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/15.jpg)
2D Case Wrong Principal Point
∞X2
∞=x KX2 2
θ
′θ
∞X1
Camera center at O
f
∞=x KX1 1
p
′≠θ θ in general.
![Page 16: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/16.jpg)
2D Case
∞X2
∞=x KX2 2
θ′θ
∞X1
′θ = θ
Camera center at O
f
∞=x KX1 1
p
Focal plane
L1 ′L1
L2
′L2
L1 ′L1 L2′L2because and
Given two vanishing points, the focal length and principal point are uniquely defined. For the 3D case, another vanishing point is needed to uniquely define f, px, and py.
![Page 17: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/17.jpg)
Calibration via Vanishing Points
![Page 18: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/18.jpg)
Calibration via Vanishing Points
![Page 19: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/19.jpg)
Calibration via 3 Vanishing Points
− − =x K K xT T 11 2( ) 0
x1 x2
x3
When vanishing points of lines are perpendicular,
x1
x2
O
2
− −
− − − −=
x K K x
x K K x x K K x
T T 11 2
T T 1 T T 11 1 2
( )cos 0
( ) ( )θ =
−K x1 2
−K x1 1
![Page 20: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/20.jpg)
Calibration via 3 Vanishing Points − − =x K K xT T 1
1 2( ) 0− − =x K K xT T 1
3 2( ) 0− − =x K K xT T 1
1 3( ) 0x1 x2
x3
Assume zero skew and square pixel:
K =x
y
f p
f p
0
13 unknowns and 3 equations.
− −
= =
K K B
B B
B B
B B B
1 2T 1
1 3
2 3 4
=x BxT3 2 0
=x BxT1 2 0
=x BxT1 3 0
Substitution to formulate linear equations:
+ + +
+ + + = = + + +
A 0b
Bu u v v u u v v
Bu u v v u u v v
Bu u v v u u v v
B
11 2 1 2 1 2 1 2
23 2 3 2 3 2 3 2
31 3 1 3 1 3 1 3
4
1
1
1
![Page 21: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/21.jpg)
Calibration via 3 Vanishing Points
x1 x2
x3
+ + +
+ + + = = + + +
A 0b
Bu u v v u u v v
Bu u v v u u v v
Bu u v v u u v v
B
11 2 1 2 1 2 1 2
23 2 3 2 3 2 3 2
31 3 1 3 1 3 1 3
4
1
1
1
= Ab null( )
b has to satisfy Ab=0 while avoiding a trivial solution, i.e., b=0.
Such b is the nullspace of A,i.e, .
The nullspace of A is can be computed by Singular Value Decomposition (SVD):
A = UDV T
b = V:,4 (the fourth or last column of V) where
Once B is estimated, it can be decomposed by Cholesky decomposition that decomposes a positive definite matrix to lower triangular matrix, i.e., where . B = LLT −=L KT 1
![Page 22: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/22.jpg)
Calibration via 3 Vanishing Points
−
= − = − +
K K
x
y
y yx x
pf f
B Bp
B Bf f
B B Bp p-p p
f f f f
2 2
1 2-T -1
1 32 2
2 3 422
2 2 2 2
1
1
+1
( )
/
/
= −
= −
= − +
y
x
x y
p B B
p B B
Bf p p
B
3 1
2 1
2 24
1
Given B, solve for intrinsic parameters:
Or
![Page 23: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/23.jpg)
Multiple View Calibration
![Page 24: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/24.jpg)
Counting Argument
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
u
v
: knowns
# of unknowns: 5 (K) + 3 (R) + 3 (t) # of equations: 2m where m is the number of known points.
![Page 25: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/25.jpg)
Multiple View Calibration
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
# of unknowns: 5 (K) + 3n (R) + 3n (t) # of equations: 2nm where m is the number of known points and n is the number of images.
![Page 26: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/26.jpg)
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
x X×∈R 3 3 tK
: knowns
: unknowns
![Page 27: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/27.jpg)
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
r r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 3
01 1
1
x X×∈R 3 3 tK
: knowns
: unknowns
![Page 28: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/28.jpg)
Known 3D Points
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
u f s p r r t X
z v f p r r t Y
r r t
img 11 12 1
img 21 22 2
31 32 31 1 1
u
v
: knowns
X Y( , ,0)
: unknowns
![Page 29: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/29.jpg)
Homography btw Two Planes
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
=
u h h h X
z v h h h Y
h h h
11 12 13
21 22 23
31 32 331 1H Xx =: estimate
: knowns
: homography between 3D plane and 2D image plane. H
![Page 30: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/30.jpg)
Homography and Projection
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1h2 h3
: unknowns
![Page 31: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/31.jpg)
Orthogonality
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1
Recall orthogonality:
h2 h3
: unknowns
r rT1 2 = 0r rT2 2 = 1r rT1 1 = 1
![Page 32: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/32.jpg)
Orthogonality
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
′=
x x
y y
h h h f s p r r t
h h h z f p r r t
h h h r r t
11 12 13 11 12 1
21 22 23 21 22 2
31 32 33 31 32 31
h1
: estimate
K r1 r2 t1
Recall orthogonality: r rT1 1 = 1 r rT2 2 = 1 r rT1 2 = 0
h2 h3
: unknowns
where and ′=h rK z11
-1 ′=h rK z1
2-
2
=h h hK K K K hT T1 1 2
-T -1 -T -12 =K Kh h-T -1T
1 2 0 2 equations and 5 unknowns
![Page 33: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/33.jpg)
Substitution
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
= − + − − =
− − +
K K
y x y
x x y x y
y x y y
x y x y y x y y
y x y y x y y y x y y
x y x y y x y y
p s - p fsf f f f f
B B Bs p s - p f ps s
B B Bf f f f f f f f
B B Bp s - p f s p s - p f p p s - p f p
f f f f f f f f
2 2 2
11 12 132-T -1
21 22 232 2 2 2 2 2 2
31 32 332 2
2 2 2 2 2 2 2
1
( )1
( ) ( )+1
=h h hK K K K hT T1 1 2
-T -1 -T -12 =K Kh h-T -1T
1 2 0
=K K-T -1 : symmetric matrix
=h hBT1 2 0=h h h hB BT T1 1 2 2 2 equations and linear in B.
![Page 34: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/34.jpg)
Linear Equations
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
[ ]=b B B B B B BT
11 12 22 13 23 33
=h h vB bi j ijT T
where
= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT
1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3
=h h h hB BT T1 1 2 2 =h hBT1 2 0
Rewrite equation by vectorizing B.
=
B
B B B
B B B
B B B
11 12 13
21 22 23
31 32 33
Linear equation in B
−
v v= 0
vb
T T11 22
T12v b =T
12 0=h hBT1 2 0
=h h h hB BT T1 1 2 2 v = vb bT T
11 22: 2 equations per view
![Page 35: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/35.jpg)
Linear Equations
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
[ ]=b B B B B B BT
11 12 22 13 23 33
=h h vB bi j ijT T
where
= + + + v ij i j i j i j i j i j i j i j i j i jh h h h h h h h h h h h h h h h h hT
1 1 1 2 2 1 2 2 3 1 1 3 3 2 2 3 3 3
=h h h hB BT T1 1 2 2 =h hBT1 2 0
Rewrite equation by vectorizing B.
=
B
B B B
B B B
B B B
11 12 13
21 22 23
31 32 33
Linear equation in B
v b =T12 0=h hBT1 2 0
=h h h hB BT T1 1 2 2 v = vb bT T
11 22 −
−
v v
v= 0
v v
b
v
n n
n
1 T 1 T11 221 T
12
T T11 22
T12
Minimum configuration: • 3 images for general K. • 2images when w=0.
![Page 36: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/36.jpg)
Solving for Intrinsic Parameters
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
= − + − − =
− − +
K K
y x y
x x y x y
y x y y
x y x y y x y y
y x y y x y y y x y y
x y x y y x y y
p s - p fsf f f f f
B B Bs p s - p f ps s
B B Bf f f f f f f f
B B Bp s - p f s p s - p f p p s - p f p
f f f f f f f f
2 2 2
11 12 132-T -1
21 22 232 2 2 2 2 2 2
31 32 332 2
2 2 2 2 2 2 2
1
( )1
( ) ( )+1
Recall
/
/
= − −
= − + −
=
y
y
x
p B B B B B B B
c B B p B B B B B
f c B
212 13 11 23 11 22 12
233 13 12 13 11 23 11
11
( ) ( )
[ ( )] /
/
/
/ /
= −
= −
= −
y
x y
x y y x
f cB B B B
s B f f c
p sp f B f c
211 11 22 12
212
213
( )
Given B, solve for intrinsic parameters:
![Page 37: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/37.jpg)
Solving for Extrinsic Parameters
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
−
−
=′
=′
r
r
K h
K h
z
z
11
12
1
2
1
1
−′ = K hz 11
where
[ ] [ ]′ r r rh h = th Kz 13 22 31
−
=
=
×
′
r r
K
r
t hz
3 1 2
13
1
![Page 38: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/38.jpg)
Summary of Linear Parameter Estimation
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
u
v
X Y( , ,0)
• Reduce dimensionality of a 3D point to 2D assuming the point lies on a plane.
• Estimate a homography between 3D plane and the 2D projected image.
• Constrain the estimated homography using orthogonality of rotation matrix
• Substitute where B is a symmetric matrix.
• Formulate a system of linear equations in B.
• Compute the intrinsic and extrinsic parameters.
r r r rT T1 1 2 2= =h h hK K K K hT T
1 1 2-T -1 -T -1
2
=K Kh h-T -1T1 2 0r rT1 2 = 0
B = K K-T -1
−
v v= 0
vb
T T11 22
T12
![Page 39: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/39.jpg)
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
x ij
x̂ K R t Xij i i j( , , , )
![Page 40: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/40.jpg)
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
, ,ˆ
= =
−∑∑K R t
x x K R t XI J
ij ij i i ji j
2
1 1
minimize ( , , , )reprojection
measurement
x ij
x̂ K R t Xij i i j( , , , )
![Page 41: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/41.jpg)
Lens Distortion
![Page 42: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/42.jpg)
[ ] =
x XK R tL
1 1
Camera body configuration (extrinsic parameter)
Spatial relationship between sensor and pinhole (internal parameter)
Lens configuration (internal parameter)
![Page 43: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/43.jpg)
Lens Distortion Radial Distortion
![Page 44: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/44.jpg)
Lens Distortion Radial Distortion
Straight lines are not straight in the image due to lens distortion.
![Page 45: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/45.jpg)
Lens Distortion Radial Distortion
Lens distortion correction
![Page 46: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/46.jpg)
Radial Lens Distortion
=x xL rimg ideal( )
Image undistortion
r
Radial lens distortion model assumes that the distortion is a function of distance between image point to the principal point.
x imgx ideal
: a point measured in the image (distorted)
: a point that follows perspective projection (undistorted) ,x yp p= xr ideal
![Page 47: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/47.jpg)
Radial Lens Distortion
rx img
x ideal
= xr ideal
where = + + +L r
2 41 2( ) 1 k r k r
Polynomial fitting of distortion
=x xL rimg ideal( )
![Page 48: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/48.jpg)
Radial Lens Distortion
= + +
= + +x
y
u u u - p r r
v v v - p r
k
r
k
k k
2 4img ideal ideal
2 4img idea
1 2
1l id l 2ea
( )[ ]
( )[ ]
where and are measurements and and are the projected point.
u img
− = −
x x
y y
u uku - p r u - p rv vkv - p r v - p r
2 4img ideal1ideal ideal
2 4img ideal2ideal ideal
( ) ( )
( ) ( )
Mapping between an ideal point and image point (2nd order):
v img u ideal v ideal
r x imgx ideal
Linear equations in k1 and k2:
: unknowns
![Page 49: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/49.jpg)
Radial Lens Distortion
= + +
= + +x
y
u u u - p r r
v v v - p r
k
r
k
k k
2 4img ideal ideal
2 4img idea
1 2
1l id l 2ea
( )[ ]
( )[ ]
where and are measurements and and are the projected point.
u img
− − = − −
x x
y y
m mm mx m x m
m mm my m y m
u uu - p r u - p r
v vv - p r v - p rk
ku uu - p r u - p r
v vv - p r v - p r
1 11 2 1 4img idealideal 1 ideal 1
1 11 2 1 4img idealideal 1 ideal 1
1
22 4img idealideal ideal
2 4img ideideal ideal
( ) ( )
( ) ( )
( ) ( )
( ) ( )
al
Mapping between an ideal point and image point (2nd order):
v img u ideal v ideal
r x imgx ideal
: unknowns
Linear equations in k1 and k2:
for m points
![Page 50: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/50.jpg)
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
, ,ˆ
= =
−∑∑K R t
x x K R t XI J
ij ij i i ji j
2
1 1
minimize ( , , , )reprojection
measurement
x ij
x̂ K R t Xij i i j( , , , )
![Page 51: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/51.jpg)
Geometric Error Minimization
Z. Zhang, “A Flexible New Technique for Camera Calibration”, PAMI 2000.
=x x
y y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
11 12 13 1
21 22 23 2
31 32 33 31 11
x XR tKu
v
x ij
x̂ K R t Xij i i j( , , , )
, ,= =
−∑∑K R t,
x K R t Xm n
ij i i jk ki j
L k k1 2
2
1 2,
1 1
minimize ( , , , , , )reprojection
measurement
![Page 52: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/52.jpg)
Geometric Error Minimization
![Page 53: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/53.jpg)
![Page 54: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/54.jpg)
![Page 55: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/55.jpg)
![Page 56: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/56.jpg)
Demo with MATLAB
http://www.vision.caltech.edu/bouguetj/calib_doc/
![Page 57: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/57.jpg)
• Download the MATLAB calibration toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc/download/index.html
• Print out a calibration pattern: http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/pattern.pdf
• Take images from multiple angles and save them in the calibration toolbox folder. • Type calib and load images.
• Type pattern dimensions and annotate corners.
• Calibrate the camera.
![Page 58: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/58.jpg)
Lens Large Distortion
![Page 59: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/59.jpg)
![Page 60: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/60.jpg)
Fisheye Lens Distortion
![Page 61: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/61.jpg)
Fisheye Lens Distortion
![Page 62: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/62.jpg)
Fisheye Lens Model Logarithmic Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
=x xL rimg ideal ideal( )
=r c krimg ideallog(1+ )
rideal x img
x ideal
rimg
rL r
rimg
ideal
( ) =
Fisheye have a high resolution at the fovea, and non linearly decreasing resolution towards the periphery, i.e.,
∆rideal
∆rimg
= =
∆ ∆>
∆ ∆r r r r
r rr r
ideal 1 ideal 2
ideal ideal
img img
>r r1 2for
=x xc kr
rideal
img idealideal
log(1+ )
where
![Page 63: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/63.jpg)
Fisheye Lens Model Field of View (FOV) Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
∝ =r r kimg ideal( ) θ θ
=
=
_
_
r kr kw /
r
r w
img
img max
img
img max
2
tan
tan( / 2)
θ
θ
z
C
X
x ideal x img
ridealrimg
,x yp p
θ=
wmax2θ
_rimg max
=
=
wr r
wr w
rw
img ideal
imgideal
1arctan(2 tan )
2tan( )
2tan( / 2)
rideal x img
x ideal
rimgSpatial resolution is proportional to angular resolution, i.e.,
=x xL rimg ideal ideal( )r
L rrimg
ideal
( ) =
=_
kwrimg max 2
Spatial resolution:
where w is angular field of view.
![Page 64: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/64.jpg)
Fisheye Lens Model Field of View (FOV) Distortion
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
z
C
X
x ideal x img
ridealrimg
,x yp p
θ=
wmax2θ
_rimg max
=x xw
rwrimg ideal ideal
ideal
1arctan(2 tan )
2
rideal x img
x ideal
rimg=x xL rimg ideal ideal( )
rL r
rimg
ideal
( ) =
where w is angular field of view.
![Page 65: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/65.jpg)
Distortion Correction Procedure
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
1. Detect edges from an image.
![Page 66: Camera Calibrationcis580/Spring2016/... · Intrinsic Bias • A scale factor that converts physical focal length to pixel unit, i.e., f (mm) f x (pixel). • Position of image center](https://reader034.vdocuments.us/reader034/viewer/2022050120/5f50a8c0ae53c503ad6df603/html5/thumbnails/66.jpg)
Distortion Correction Procedure
F. Devernay and O. Faugeras, “Straight Lines Have to be Straight”, Machine Vision and Applications, 2001.
1. Detect edges from an image.
2. Link edges that are likely to belong to the same edge.
3. Select distortion model and initialize the distortion parameters with a reasonable value.
4. Fit lines with the linked edges with least squares.
5. Optimize the distortion parameters to fit the edges to the ideal lines.
Detected edge segment
Ideal edge