Camera Projection Matrix
Slides by HyunSoo Park
x XR 3 3 tK
=
x x
x y
Xu f s p r r r t
Yz v f p r r r t
Zr r r t
img 11 12 13 1
img 21 22 23 2
31 32 33 31 11
http://www.joshuanava.biz/perspective/in-other-words-the-observer-simply-points-in-the-same-direction-as-the-lines-in-order-
to-find-their-vanishing-point.html
First person coordinate system World coordinate system
Camera Model
Ground plane
Camera Model (1st Person Perspective)Camera
Ground plane
Camera Model (1st Person Perspective)Camera
1 1
x
y
u f p X
v f p Y
Z
Recall camera projection matrix:
+
Camera intrinsic parameter
: metric space to pixel space
Ground plane
Camera Model (1st Person Perspective)Camera
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
+
Camera intrinsic parameter
: metric space to pixel space
Ground plane
Camera Model (1st Person Perspective)Camera
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
3D world (metric)2D image (pix)
Ground plane
Camera Model (1st Person Perspective)Camera
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
3D world (metric)2D image (pix)
Origin at camera
-1
1
K XC
u X
v Y
Z
CX
Ground plane
Camera Model (multiple 1st Person Perspective)Camera
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
3D world (metric)2D image (pix)
Origin at camera
1
1 1
-1
1 1 C
11
K X
u X
v Y
Z
Where is origin?1C
X
2CX
2
2 2
-1
2 2 C
21
K X
u X
v Y
Z
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
3D world (metric)2D image (pix)
X
Ground plane
Camera
X3D world
Origin at world coordinate
Camera Model (3rd Person Perspective)
1 1
x
y
u f p X
v f p Y
Z
K
Recall camera projection matrix:
3D world (metric)2D image (pix)
X
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation matrix)
X
1C
WR
2C
WR
1
x1 x2 x3
C y1 y2 y3
z1 z2 z3
=
X X
r r r
r r r
r r r
Coordinate transformation from world to camera:
X
3D world Camera
1CX
Coordinate Transform (Rotation)
2D coordinate transform:
World
x
y
= ( , )X x y
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
?
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
2 2cos sindet = cos sin 1
sin cos
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
rX
rX
: x axis of camera seen from the worldrX
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
rX
rX
: x axis of camera seen from the worldrX
: y axis of camera seen from the worldYr
Yr
rY
Coordinate Transform (Rotation)
= ( , )X x y
2D coordinate transform:
World
Camera
C C C= ( , )X x y
C
C
cos sin=
sin cos
x
y
x
y
: x axis of world seen from the camera1r
: y axis of world seen from the camera2r
r1
r1
2r 2
r
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
= ( , , )X x y z
World
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
?
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
?
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
Degree of freedom?
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
Degree of freedom?
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
rX
Yr
Zr
rX
Yr
Z X Y= r r r
rX
Yr
Zr
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
rX
Yr
Zr
rX
Yr
Z X Y= r r r
rX
: camera x axis seen
from the world coord.Zr
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
Yr
Zr
rX
Yr
Z X Y= r r r
: camera y axis seen
from the world coord.
rX
Yr
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
1r
2r
3 1 2= r r r
1r : world x axis seen
from the camera
coord.
r1 2
r3r
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RC
WSO(3)
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C C
W W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
1r
2r
3 1 2= r r r
1r : world x axis seen
from the camera
coord.
r1 2
r3r
2r : world y axis seen
from the camera
coord.
3r
x1 x2 x3
y1 y2 y3
z1 z2
C
z3
C
C1 1
1
x
y
x
y
u f p
v f p
f p r r r
f p r r r
r
X
Y
Zr r
X
Y
Z
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Pure Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
1r : world x axis seen
from the camera
coord.
r1 2
r3r
2r : world y axis seen
from the camera
coord.
3r
K
Camera projection of world point:
x1 x2 x3
y1 y2 y3
z1 z2
C
z3
C
C1 1
1
x
y
x
y
u f p
v f p
f p r r r
f p r r r
r
X
Y
Zr r
X
Y
Z
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Pure Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
1r : world x axis seen
from the camera
coord.
r1 2
r3r
2r : world y axis seen
from the camera
coord.
3r
K
Camera projection of world point:
K
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Euclidean Transform)
X
Coordinate transformation from world to camera:
x1 x2 x3
C
y1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
C C
W,R t
World
= ( , , )X x y z
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Euclidean Transform)
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C C
W y1 y2 y3 y
z1 z2 z3 z
C= =
1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C
= ( , , )X x y z
C C
W,R t
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Euclidean Transform)
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C C
W y1 y2 y3 y
z1 z2 z3 z
C= =
1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C
= ( , , )X x y z
C C
W,R t
Ct
CtWhere is translation from world to camera
seen from camera.
Ground plane
Camera
3D world
Origin at world coordinate
Geometric Interpretation
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C C
W y1 y2 y3 y
z1 z2 z3 z
C= =
1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C
= ( , , )X x y z
C C
W,R t
Ct
Ctwhere is translation from world to camera
seen from camera.
Rotate and then, translate.
Ground plane
Camera
3D world
Origin at world coordinate
Geometric Interpretation
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C C
W y1 y2 y3 y
z1 z2 z3 z
C= =
1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C
= ( , , )X x y z
C C
W,R t
Ctwhere is translation from world to camera
seen from camera.
Rotate and then, translate.
x1 x2 x3 x
C
W y1 y2 y3 y
z1 z2 z3 z
C
1
= = 11
1
XXR CX
r r r -C
r r r -C
r r r -C
cf) Translate and then, rotate.
where is translation from world to camera
seen from world.C
C
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection Matrix
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C C
W y1 y2 y3 y
z1 z2 z3 z
C= =
1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C
= ( , , )X x y z
C C
W,R t
C
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z
C
C
C1
1
1
1
x
y
x
y
u f p
v f p
f p r r r t
f p r r r t
r r r t
X
Y
Z
X
Y
Z
K
Camera projection of world point:
K C
WR
Ct
Image Projection: Sanity Check
r
= 0
0
C
-r 0 -1 0
= 0 0 -1
1 0 0
R
Image Projection
r
r
= 0
0
C
-r 0 -1 0
= 0 0 -1
1 0 0
R
Image Projection
r
r
= 0
0
C
-r 0 -1 0
= 0 0 -1
1 0 0
R
cos
= sin
0
C
r
r
-sin cos 0
= 0 0 -1
-cos -sin 0
R
Image Projection K = [200 0 100;0 200 100;0 0 1];
radius = 5;
theta = 0:0.02:2*pi;
for i = 1 : length(theta)camera_offset = [radius*cos(theta(i)); radius*sin(theta(i)); 0];camera_center = camera_offset + center_of_mass';
rz = camera_center-center_of_mass';rz = rz / norm(rz);ry = [0 0 1]';rx = Vec2Skew(ry)*rz; % cross productR = [rx'; ry'; rz'];C = camera_center;P = K * R * [ eye(3) -C];
proj = [];for j = 1 : size(sqaure_point,1)
u = P * [sqaure_point(j,:)';1];proj(j,:) = u'/u(3);
end end
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
11 12 13 14
21 22 23 24
31 32 33 341
1
Xu p p p p
Yv p p p p
Zp p p p
What does each number mean?
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
What is point at infinity in world x direction?
X
0
0
1
X
11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
What is point at infinity in world x direction?
X
0
0
1
X
This point is at infinite but finite in image.
Xu 11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
Point at infinity
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
What is point at infinity in world x direction?
X
0
0
1
X
This point is at infinite but finite in image.
Xu 11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
X
0
0
1
X
Xu
11 14 11
31 34 31
21 24 21
31 34 31
+lim
+
+lim
+
X
X
p X p pu
p X p p
p X p pv
p X p p
11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
X
0
0
1
X
Xu
11 14 11
31 34 31
21 24 21
31 34 31
+lim
+
+lim
+
X
X
p X p pu
p X p p
p X p pv
p X p p
11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
11
X 21
311
u
u p
v p
p
11
X 21
31
u
p
p
p
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
Xu
12 14 12
32 34 32
22 24 22
32 34 32
+lim
+
+lim
+
X
X
p Y p pu
p Y p p
p Y p pv
p Y p p
11 12 13 14
21 22 23 24
31 32 33 34
0
01
1
u p p p p
v p p p p
p p p p
12
Y 22
321
u
u p
v p
p
12
Y 22
32
u
p
p
p
Yu
Ground plane
Camera
3D world
Geometric Interpretation
C C
W,R t
x1 x2 x3 x
y1 y2 y3 y
z1 z2 z3 z 11 1
x
y
u f p r r r t
v f p r r r t
X
Y
Zr r r t
Camera projection of world point:
K C
WR
Ct1
=
XP
13 14 13
33 34 33
23 24 23
33 34 33
+lim
+
+lim
+
X
X
p Z p pu
p Z p p
p Z p pv
p Z p p
11 12 13 14
21 22 23 24
31 32 33 34
0
0
11
u p p p p
v p p p p
p p p p
13
Z 23
331
u
u p
v p
p
13
Z 23
33
u
p
p
p
Celestial Navigation
Far far away: point at infinity
Earth My location
Practiceimg
m
ccd
3840= = 0.00153 1224pix
0.0048
Wf f
W
img
x
3840= = 1920pix
2 2
Wp
img
y
2160= = 1080pix
2 2
Hp
0.0070
0.7520
-0.2738
C
-0.8496 0.0498 0.5731
-0.3216 -0.8203 -0.4067
0.4180 -0.5299 0.6835
R
Practiceimg
m
ccd
3840= = 0.00153 1224pix
0.0048
Wf f
W
img
x
3840= = 1920pix
2 2
Wp
img
y
2160= = 1080pix
2 2
Hp
0.0070
0.7520
-0.2738
C
-0.8496 0.0498 0.5731
-0.3216 -0.8203 -0.4067
0.4180 -0.5299 0.6835
R
3
-0.2374 -0.9565 2.0138 1.2723
0.0578 -1.5763 0.2404 1.2508
0.0004 -0.0005 0.0007 0.0006
P= KR I C
img
m
ccd
3840= = 0.00153 1224pix
0.0048
Wf f
W
img
x
3840= = 1920pix
2 2
Wp
img
y
2160= = 1080pix
2 2
Hp
0.0070
0.7520
-0.2738
C
-0.8496 0.0498 0.5731
-0.3216 -0.8203 -0.4067
0.4180 -0.5299 0.6835
R
3
-0.2374 -0.9565 2.0138 1.2723
0.0578 -1.5763 0.2404 1.2508
0.0004 -0.0005 0.0007 0.0006
P= KR I C
R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];
C = [0.00700.7520-0.2738];
P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)
[-0.2374 0.0578]/ 0.0004
=
[-593.5 144.5]
Practiceimg
m
ccd
3840= = 0.00153 1224pix
0.0048
Wf f
W
img
x
3840= = 1920pix
2 2
Wp
img
y
2160= = 1080pix
2 2
Hp
0.0070
0.7520
-0.2738
C
-0.8496 0.0498 0.5731
-0.3216 -0.8203 -0.4067
0.4180 -0.5299 0.6835
R
3
-0.2374 -0.9565 2.0138 1.2723
0.0578 -1.5763 0.2404 1.2508
0.0004 -0.0005 0.0007 0.0006
P= KR I C
R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];
C = [0.00700.7520-0.2738];
P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)
[-0.9565 -1.5763] / -0.0005
=
[1,913 3,152]
img
m
ccd
3840= = 0.00153 1224pix
0.0048
Wf f
W
img
x
3840= = 1920pix
2 2
Wp
img
y
2160= = 1080pix
2 2
Hp
0.0070
0.7520
-0.2738
C
-0.8496 0.0498 0.5731
-0.3216 -0.8203 -0.4067
0.4180 -0.5299 0.6835
R
3
-0.2374 -0.9565 2.0138 1.2723
0.0578 -1.5763 0.2404 1.2508
0.0004 -0.0005 0.0007 0.0006
P= KR I C
R = [-0.8496 0.0498 0.5731-0.3216 -0.8203 -0.40670.4180 -0.5299 0.6835];
C = [0.00700.7520-0.2738];
P = K * R * [eye(3) -C]u_x = P(1:2,1)/P(3,1)u_y = P(1:2,2)/P(3,2)u_z = P(1:2,3)/P(3,3)
[2.0138 0.2404]/0.0007
=
[2877 343]
Z direction in the world coordinate system
Z direction in the world coordinate system
uv
Z direction in the world coordinate system
z point at infinity
uv
: z vanishing pointvz
z =T
0 0 1 0
Calibration via Vanishing Points
v1 v3
v2
v1
v3
K v1 3
K v1 1
Strong perspectivenessWeak perspectiveness
Orthographic CameraAffine camera:
x1 x2 x1 x2
A y1 y2 y1 y2
0 0 0 0
0 0 0 0
1 0 0 0 1 0 0 0 1
P
x x
y y
f p r r f / d p r r
f p r r d f / d p r r
d
1f
Orthographic camera:
X Y0 p p
Orthographic CameraAffine camera:
x1 x2 x1 x2
A y1 y2 y1 y2
0 0 0 0
0 0 0 0
1 0 0 0 1 0 0 0 1
P
x x
y y
f p r r f / d p r r
f p r r d f / d p r r
d
1f
Orthographic camera:
X Y0 p p
x1 x2 x3
O y1 y2 y3
0
0
0 0 0 1
P
r r r
r r r
Lens Radial Distortion