Lin ZHANG, SSE, 2016
Lecture 7Measurement Using a Single Camera
Lin ZHANG, PhDSchool of Software Engineering
Tongji UniversityFall 2016
Lin ZHANG, SSE, 2016
If I have an image containing a coin, can you tell me the diameter of that coin?
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Vector representation
{ , , }a xi y j zk x y z
Length (or norm) of a vector
2 2 2a x y z
Normalized vector (unit vector)
{ , , }a x y za a a a
We say if and only if,a 0 0, 0, 0x y z
Vector operations
Lin ZHANG, SSE, 2016
if 1 1 1( , , ),a x y z
2 2 2( , , ),b x y z
1 2 1 2 1 2( , , ),a b x x y y z z
then
1 2 1 2 1 2cosa b a b x x y y z z
Dot product (inner product)
Laws of dot product:
, ( )= +a b b a a b c a b a c
Theorem
0a b a b (why?)
a
b
Vector operations
Lin ZHANG, SSE, 2016
Cross product
1 1 1 1 1 1
1 1 1
2 2 2 2 2 2
2 2 2
i j ky z z x x y
a b x y z i j ky z z x x y
x y z
Vector operations
Lin ZHANG, SSE, 2016
c a b
Cross product
is also a vector, whose direction is determined by the right‐hand law and
a
bc
,c a c b
sinc a b
represents the oriented area of the parallelogram taking and as two sidesc
a b(easy to prove)
1 2 2 1r r r r
(why?)
Vector operations
Lin ZHANG, SSE, 2016
they are not equal to zero at the same time, and
Cross product
Theorem|| 0a b a b (why?)
Theorem
|| , ,a b
a b 0
1 2 3 1 2 1 3( )r r r r r r r
Property
(easy to understand)
Vector operations
Lin ZHANG, SSE, 2016
Mixed product (scalar triple product or box product)
Geometric Interpretation: it is the (signed) volume of the parallelepiped defined by the three vectors given
1 2 3
1 2 3
1 2 3
( , , ) ( )a a ab b bc c c
a b c a b c
Vector operations
Lin ZHANG, SSE, 2016
Mixed product (scalar triple product or box product)
1 2 3
1 2 3
1 2 3
( , , ) ( )a a ab b bc c c
a b c a b c
( ) cos
sin cos
a b c a b c
a b c
a
cBase h
Vector operations
Lin ZHANG, SSE, 2016
Mixed product (scalar triple product or box product)
1 2 3
1 2 3
1 2 3
( , , ) ( )a a ab b bc c c
a b c a b c
Property:
( , , ) ( , , ) ( , , ) a b c b c a c a b( , , ) ( , , ) ( , , ) a b c b a c a c b
Vector operations
why?
Lin ZHANG, SSE, 2016
Mixed product (scalar triple product or box product)Theorem
, ,a b c are coplanar ( , , ) 0 a b c
, ,a b c are coplanar , , ,v they are not equal to zero at the same time, and v a b c 0
Vector operations
why?
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
For a normal point on a plane , ,Tx y 0Its homogenous coordinate is where k can be anynon‐zero real number
, ,1 ,Tk x y
For a homogenous coordinate
Homogenous coordinate for a point is not only one
' ' ', ,T
x y z
we usually rewrite it as ' ' ' '/ , / ,1T
x z y z
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
Converting from homogenous coordinate to inhomogeneous coordinate
''
''
''
'
xxzyy
zz
For a normal point on a plane 0Its homogenous coordinate is where k can be anynon‐zero real number
, ,1 ,Tk x y , ,Tx y
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
Geometric interpretation
1e
2e
o
1o
1
00 0( , )M x y
3e
1e
2e
In plane , in the 2D frame01 1 2( : , )o e e 0 0: ( , )M x y
Coordinate of any point on line OMin the frame is the homogeneous coordinate of M
1 2 3( : , , )o e e e
These points can be represented as
0 0, ,1 Tk x y
, one point
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
Geometric interpretation
1e
2e
o
1o
1
00 0( , )M x y
3e
1e
2e
How about a line passing through O and parallel to ?0
In plane , in the 2D frame01 1 2( : , )o e e 0 0: ( , )M x y
Coordinate of any point on line OMin the frame is the homogeneous coordinate of M
, one point
These points can be represented as
0 0, ,1 Tk x y
1 2 3( : , , )o e e e
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
Geometric interpretation
1e
2e
o
1o
1
03e
1e
2e
How about a line passing through O and parallel to ?0
Consider a line passing through Oand M(x0, y0, 0)T
0 0( , )M x y We define: it meets at an infinity point, and also the homogeneous coordinate of such a point can be represented as points on OM
0
So, infinity point has the form (kx0, ky0, 0)T
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?Normal case:
line k (x0, y0, 1)
a normal point (x0, y0) on the plane 0
The homogeneous coordinate of this normal point is k(x0, y0 ,1)
abnormal case:
line (kx0, ky0, 0)
Define: it meets at an infinity point
0
The homogeneous coordinate of this infinity point is k(x0, y0 ,0)
make an analogy
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• What is homogeneous coordinate?
Geometric interpretation
1e
2e
o
1o
1
03e
1e
2e
How about a line passing through O and parallel to ?0
0 0( , )M x y
One infinity point determines an orientation
We define: all infinity points on comprise an infinity line
0
In fact, plane meets at the infinity line
01 2oe e
Homogeneous equation of the infinity line is 0x + 0y + 1z = 0
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry
Properties of a projective plane• Two points determine a line; two lines determine a point (the
second claim is not correct in the normal Euclidean plane)• Two parallel lines intersect at an infinity point; that means one
infinity point corresponds to a specific orientation• Two parallel planes intersect at the infinity line
0 + infinity line = Projective plane
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry
JingHu High‐speed railway: rails will “meet” at the vanishing point
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
On a projective plane, please determine the line passing two points
1e
2e
o1
0M
',o ox x determine two linesxx’ actually is the intersection between oxx’ and 0
'1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x
x'x
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
1e
2e
o1
0x
'xM Thus, locates on xx’( , , )M x y z oM resides on the plane oxx’ ', ,o oM ox x are coplanar
1 1 1
2 2 2
0x y zx y zx y z
On a projective plane, please determine the line passing two points '
1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
1 1 1
2 2 2
0x y zx y zx y z
1 1 1 1 1 1
2 2 2 2 2 2
0y z z x x y
x y zy z z x x y
1 1 1 1 1 1
2 2 2 2 2 2
, ,T
y z z x x yy z z x x y
Homogeneous coordinate of the lineHomogeneous coordinate of the infinity line is (0,0,1)T
On a projective plane, please determine the line passing two points '
1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
1 1 1
2 2 2
0x y zx y zx y z
1 1 1 1 1 1
2 2 2 2 2 2
0y z z x x y
x y zy z z x x y
TheoremOn the projective plane, the line passing two points is',x x
' l x x
1 1 1 1 1 1
2 2 2 2 2 2
, ,T
y z z x x yy z z x x y
On a projective plane, please determine the line passing two points '
1 1 1 2 2 2( , , ) , ( , , )T Tx y z x y z x x
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
A point is on the line0 0 0( , , )Tx y zx
0T x l
( , , )Ta b cl
0 x l(It is )
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinateTheorem: On the projective plane, the intersection of two lines is the point ' x l l',l lProof: Two lines 1 1 1 0,a x b y c z 2 2 2 0a x b y c z
'1 1 1 2 2 2( , , ) , ( , , )T Ta b c a b c l l
Inhomogeneous form ,x yX Yz z
1 1 1
2 2 2
00
a X b Y ca X b Y c
1 1 1 1
2 2 2 2
1 1 1 1
2 2 2 2
,
c b a cc b a c
X Ya b a ba b a b
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
1 1 1 1
2 2 2 2
1 1 1 1
2 2 2 2
, ,1
c b a cc b a c
ka b a ba b a b
x
Homogenous form of the cross point is
1 1 1 1 1 1
2 2 2 2 2 2
, ,c b a c a bc b a c a b
x
let 1 1
2 2
a bk
a b
1 1 1 1 1 1
2 2 2 2 2 2
, ,b c c a a bb c c a a b
x ' x l l
Theorem: On the projective plane, the intersection of two lines is the point ' x l l',l l
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• Lines in the homogeneous coordinate
Example: find the cross point of the lines 1, 1x y
o x
y
1x
1y 1 2 3
1 2 3
1 0 ( 1) 00 1 ( 1) 0x x xx x x
Homogeneous coordinates of the two lines are (1,0, 1) , (0,1, 1)T T Cross point is
(1,0, 1) (0,1, 1) 1,1,1T T
Homogeneous form
Theorem: On the projective plane, the intersection of two lines is the point ' x l l',l l
1 2
3 3
,x xx yx x
Lin ZHANG, SSE, 2016
(1,0, 1) (1,0, 2) 1 0 1 0,1,01 0 2
T T
i j k
Foundations of Projective Geometry• Lines in the homogeneous coordinate
Example: find the cross point of the lines 1, 2x x
o x
y
1x
Homogeneous coordinates of the two lines are (1,0, 1) , (1,0, 2)T T Cross point is
2x 1 2 3
1 2 3
1 0 ( 1) 01 0 ( 2) 0x x xx x x
Homogeneous form
Theorem: On the projective plane, the intersection of two lines is the point ' x l l',l l
1 2
3 3
,x xx yx x
Lin ZHANG, SSE, 2016
Foundations of Projective Geometry• DualityIn projective geometry, lines and points can swap their positions
0T x lIf x is a variable, it represents the points lying on the line l;If l is a variable, it represents the lines passing a fixed point x
How to interpret?
The line passing two points is',x x ' l x xThe cross point of two lines is',l l ' x l l
Duality Principle: To any theorem of projective geometry, there corresponds a dual theorem, which may be derived by interchanging the roles of points and lines in the original theorem
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Matrix differentiation
• Function is a vector and the variable is a scalar
1 2( ) ( ), ( ),..., ( ) Tnf t f t f t f t
Definition
1 2 ( )( ) ( ), ,...,T
ndf tdf t df tdfdt dt dt dt
Lin ZHANG, SSE, 2016
• Function is a matrix and the variable is a scalar11 12 1
21 22 2
1 2
( ) ( ),..., ( )( ) ( ),..., ( )
( ) ( )
( ) ( ),..., ( )
m
mij n m
n n nm
f t f t f tf t f t f t
f t f t
f t f t f t
Definition111 12
221 22
1 2
( )( ) ( ) ,...,
( )( ) ( ) ( ),...,
( ) ( ) ( ),...,
m
mij
n m
n n nm
df tdf t df tdt dt dt
df tdf t df t df tdfdt dt dt
dt dt
df t df t df tdt dt dt
Matrix differentiation
Lin ZHANG, SSE, 2016
• Function is a scalar and the variable is a vector
1 2( ), ( , ,..., )Tnf x x xx x
Definition
1 2
, ,...,T
n
df f f fd x x x
x
In a similar way,
1 2( ), ( , ,..., )nf x x xx x
1 2
, ,...,n
df f f fd x x x
x
Matrix differentiation
Lin ZHANG, SSE, 2016
• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T
n mx x x y y y x y x x xDefinition
1 1 1
1 2
2 2 2
1 2
1 2
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
n
nT
m m m
n m n
y y yx x x
y y yd x x xd
y y yx x x
x x x
x x xyx
x x x
Matrix differentiation
Lin ZHANG, SSE, 2016
• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T
n mx x x y y y x y x x xIn a similar way,
1 2
1 1 1
1 2
2 2 2
1 2
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
m
mT
m
n n n n m
y y yx x x
y y yd x x xd
y y yx x x
x x x
x x xyx
x x x
Matrix differentiation
Lin ZHANG, SSE, 2016
• Function is a vector and the variable is a vectorExample:
11 2 2
2 1 1 2 2 3 22
3
( ), , ( ) , ( ) 3
( )
xy
x y x x y x xy
x
xy x x x
x
1 2
1 11
1 2
2 23
1 2
3 3
( ) ( )
2 0( ) ( ) 1 3
0 2( ) ( )
T
y yx x x
d y yd x x
xy y
x x
x x
y x xx
x x
Matrix differentiation
Lin ZHANG, SSE, 2016
• Function is a scalar and the variable is a matrix
11 12 1
1 2
( ) n
m m mn
f f fx x x
dfd
f f fx x x
XX
( ), m nf X X
Definition
Matrix differentiation
Lin ZHANG, SSE, 2016
• Useful results1, nx a
,T Td d
d d
a x x aa ax x
Then,
How to prove?
(1)
Matrix differentiation
Lin ZHANG, SSE, 2016
• Useful results1,m n nA x (2) Then, T
dA Ad
xx
1,m n nA x (3) Then,T T
Td A Ad
xx
1,n n nA x (4) Then, ( )T
Td A A Ad
x x xx
1 1, ,m n m n X a b (5) Then,T
Tdd
a Xb abX
1 1, ,n m m n X a b (6) Then,T T
Tdd
a X b baX
1nx (7) Then, 2Td
d
x x xx
Matrix differentiation
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• What is Camera Calibration• Single Camera Calibration• Bird‐view Generation
Lin ZHANG, SSE, 2016
Lagrange multiplier
• Single‐variable function
( )f x is differentiable in (a, b). At , f(x) achieves an extremum
0 ( , )x a b
0| 0x
dfdx
• Two‐variables function
( , )f x y is differentiable in its domain. At , f(x, y) achieves an extremum
0 0( , )x y
0 0 0 0( , ) ( , )| 0, | 0x y x yf fx y
Lin ZHANG, SSE, 2016
Lagrange multiplier
• In general case1( ), nf x x If is a stationary point of 0x
0 0 01 2
| 0, | 0,..., | 0n
f f fx x x
x x x
Lin ZHANG, SSE, 2016
Lagrange multiplier
• Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints
Problem: find stationary points for 1( ),x x ny f under m constraints ( ) 0, 1, 2,...,kg k m x
is a stationary point of with constraints
Solution:
11
( ; ,..., ) ( ) ( )m
m k kk
F f g
x x xIf is a stationary point of F, then,
0 10 20 0( , , ..., )m x
0x ( )f xJoseph‐Louis LagrangeJan. 25, 1736~Apr.10, 1813
Lin ZHANG, SSE, 2016
Lagrange multiplier
• Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints
Solution:1
1( ; ,..., ) ( ) ( )
m
m k kk
F f g
x x x
is a stationary point of F0 10 0( , ,..., )m x
1 2 1 2
0, 0,..., 0, 0, 0,..., 0n m
F F F F F Fx x x
n + m equations!at that point
Problem: find stationary points for 1( ),x x ny f under m constraints ( ) 0, 1, 2,...,kg k m x
Lin ZHANG, SSE, 2016
Lagrange multiplier
• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.
y=xp0
?
The distance is 2 2( , ) ( 1) ( 0)f x y x y
Now we want to find the stationary point of f(x, y) under the constraint
( , ) 0g x y y x According to Lagrange multiplier method, construct another function
2 2( , , ) ( ) ( ) ( 1) ( )F x y f x g x x y y x Find the stationary point for ( , , )F x y
Lin ZHANG, SSE, 2016
Lagrange multiplier
• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.
y=xp0
?
0
0
0
FxFyF
2( 1) 02 0
0
xy
x y
0.50.51
xy
(0.5,0.5,1) is a stationary point of ( , , )F x y (0.5,0.5) is a stationary point of f(x,y) under constraints
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird‐view Generation
Lin ZHANG, SSE, 2016
LS for Inhomogeneous Linear SystemConsider the following linear equations system
1 2 1
1 2 2
3 1 1 32 4 2 1 4x x x
x x x
Matrix form: A x bA x b
It can be easily solved 1
2
12
xx
Lin ZHANG, SSE, 2016
LS for Inhomogeneous Linear SystemHow about the following one?
1 21
1 22
1 2
3 1 1 32 4 2 1 4
1 2 62 6
x xx
x xx
x x
It does not have a solution!
What is the condition for a linear equation system can be solved?
A x b
Can we solve it in an approximate way?A: we can use least squares technique!
Carl Friedrich Gauss
Lin ZHANG, SSE, 2016
LS for Inhomogeneous Linear SystemLet’s consider a system of p linear equations with q unknowns
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
...
...
......
q q
q q
p p pq q p
a x a x a x
a x a x a xA
a x a x a x
b
bx b
b
We consider the case: p>q, and rank(A)=qIn general case, there is no solution!
Instead, we want to find a vector x that minimizes the error:22
1 11
( ) ( ... )p
i iq q ii
E a x a x A
x b x b
unknowns
Lin ZHANG, SSE, 2016
LS for Inhomogeneous Linear System2*2
arg min ( ) arg minE A x x
x x x b
1* T TA A A
x bPseudoinverse of A
How about the pseudoinverse of A when A is square and non-singular?
Lin ZHANG, SSE, 2016
LS for Homogeneous Linear SystemLet’s consider a system of p linear equations with q unknowns
11 1 12 2 1
21 1 22 2 2
1 1 2 2
... 0
... 0
...... 0
q q
q q
p p pq q
a x a x a x
a x a x a xA
a x a x a x
x 0
We consider the case: p>q, and rank(A)=q
unknowns
Theoretically, there is only a trivial solution: x = 0
So, we add a constraint to avoid the trivial solution 2
1x
Lin ZHANG, SSE, 2016
LS for Homogeneous Linear SystemWe want to minimize , subject to 2
1x 2
2( )x xE A
*2
arg min ( ), . ., 1x
x x xE s t
Use the Lagrange multiplier to solve it,
2 2*2 2
arg min 1x
x x xA
(1)
Solving the stationary point of the Lagrange function,
2 2
2 2
2 2
2 2
1
10
x x0
x
x x
A
A
(3)
(2)
Lin ZHANG, SSE, 2016
LS for Homogeneous Linear System
Then, we have
2 2
2 21x x
0x
A
(3)
x xTA A
x is the eigen‐vector of ATA associated with the eigenvalue
2
2x x x x x xT T TE A A A
The unit vector x is the eigenvector associated with the minimum eigenvalue of TA A
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Homography Estimation
Problem definition: Given a set of points and a corresponding set of points in a projective plane, compute the projective transformation that takes to
ix 'ix
ix'ix
' , 1, 2,...,i iH i n x xWe know there existing an H satisfying Coordinates of and are known, ix 'ix we need to find Hwhere H is a homography matrix
11 12 13
21 22 23
31 32 33
a a aH a a a
a a a
It has 8 degrees of freedom
Lin ZHANG, SSE, 2016
Homography Estimation
11 12 13
21 22 23
31 32 33 1
a a acu xcv a a a yc a a a
11 12 13
21 22 23
31 32 33
a x a y a cua x a y a cva x a y a c
11 12 13
31 32 33
21 22 23
31 32 33
a x a y a ua x a y aa x a y a va x a y a
4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom
Lin ZHANG, SSE, 2016
Homography Estimation4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom 11
12
13
21
22
23
31
32
33
1 0 0 00
0 0 0 1
aaaa
x y ux uy ua
x y vx vy vaaaa
Thus, four correspondence pairs generate 8 equations
Lin ZHANG, SSE, 2016
Homography Estimation4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom
0 (1)A x
8 9 9 1
Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space
( ) 8Rank A
Lin ZHANG, SSE, 2016
Homography Estimation• How about the case when there are more than 4 correspondence pairs?• Use the LS method (for homogeneous case) to solve the model
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Camera calibration is a necessary step in 3D computer vision in order to extract metric information from 2D images
• It estimates the parameters of a lens and image sensor of the camera; you can use these parameters to correct for lens distortion, measure the size of an object in world units, or determine the location of the camera in the scene
• These tasks are used in applications such as machine vision to detect and measure objects. They are also used in robotics, for navigation systems, and 3‐D scene reconstruction
What is camera calibration?
Lin ZHANG, SSE, 2016
What is camera calibration?
Lin ZHANG, SSE, 2016
• Camera parameters include• Intrinsics• Extrinsics• Distortion coefficients
What is camera calibration?
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• For simplicity, usually we use a pinhole camera model
Single Camera Calibration
Lin ZHANG, SSE, 2016
• To model the image formation process, 4 coordinate systems are required• World coordinate system (3D space)• Camera coordinate system (3D space)• Retinal coordinate system (2D space)• Pixel coordinate system (2D space)
Single Camera Calibration
Lin ZHANG, SSE, 2016
• To model the image formation process, 4 coordinate systems are required
Single Camera Calibration
Lin ZHANG, SSE, 2016
• From the world CS to the camera CS
Single Camera Calibration
, , Tw w wX Y Z is a 3D point represented in the WCS
In the camera CS, it is represented as,
R tc w
c w
c w
X XY YZ Z
a rotation matrix (orthogonal)
3 3
a translation vector3 1
Lin ZHANG, SSE, 2016
• From the world CS to the camera CS
Single Camera Calibration
, , Tw w wX Y Z is a 3D point represented in the WCS
In the camera CS, it is represented as,
R tc w
c w
c w
X XY YZ Z
1
1 1
R t0
c w
c wT
c w
X XY YZ Z
Homogeneous form
(1)
Lin ZHANG, SSE, 2016
• From the camera CS to the retinal CS
Single Camera Calibration
We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS
Lin ZHANG, SSE, 2016
• From the camera CS to the retinal CS
Single Camera Calibration
We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS
x
y
z
f
O is the optical center, f is the focal length, P = [Xc, Yc, Zc]T is a scene point while P’=(x, y) is its image on the retinal plane
Lin ZHANG, SSE, 2016
• From the camera CS to the retinal CS
Single Camera Calibration
We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS
cc
cc
cc
c
XfX
ZxY
y YfZ
Z
0 0 00 0 0
1 0 0 1 01
c
cc
c
Xx f
YZ y f
Z
Homogeneous form
(2)
Lin ZHANG, SSE, 2016
• From the retinal CS to the pixel CS
Single Camera Calibration
The unit for retinal CS (x-y) is physical unit (e.g., mm, cm) while the unit for pixel CS (u-v) is pixel
One pixel represents dx physical units along the x‐axis and represents dy physical units along the y‐axis; the image of the optical center is (u0, v0)
0
0
1 0
101 1
0 0 1
udxu x
v v ydy
Lin ZHANG, SSE, 2016
• From the retinal CS to the pixel CS
Single Camera Calibration
If the two axis of the pixel plane are not perpendicular to each other, another parameter s is introduced to represent the skewness of the two axis
0
0
1 0
101 1
0 0 1
udxu x
v v ydy
0
0
1
101 1
0 0 1
s udxu x
v v ydy
(3)
Lin ZHANG, SSE, 2016
Single Camera CalibrationFrom Eqs.1~3, we can have
0 0
0 0
0 0
0
1 00 0 0
10 . 0 0 0 0 0 .1 0 0 1 0
0 0 1 0 0 1 01 1
0 00 0 .0 0 1 0
1
c c
c cc
c c
c
c
c
fs u sf uX Xdx dxu fY YfZ v v f vZ Zdy dy
Xu u
Yv
Z
0
0 00 0 . 0 .1
0 0 1 0 0 0 11 1
R tR t
0
w w
w wT
w w
X Xu
Y Yv v
Z Z
Intrinsic Extrinsic
Lin ZHANG, SSE, 2016
Single Camera Calibration
0
0[ ] 0 [ ]1 0 0 1
1 1
K R t R t
w w
w wc
w w
X Xuu
Y YZ v v
Z Z
Intrinsic Extrinsic
, the coordinates of the principal point in the image plane 0 0,u v, the scale factors in image u and v axes and
, describing the skewness of the two image axesR and t determines the rigid transformation from the world coordinate system to the camera coordinate system Altogether, there are 11 parameters to be determined
Lin ZHANG, SSE, 2016
Single Camera Calibration
• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion• Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion
Lin ZHANG, SSE, 2016
Single Camera Calibration
• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion• Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion
2 4 61 2 3
2 4 61 2 3
1
1
distorted
distorted
u u k r k r k r
v v k r k r k r
where 2 2 2r u v are the radial distortion coefficients of the lens 1 2 3, ,k k k
Lin ZHANG, SSE, 2016
Single Camera Calibration
• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion
• Tangential distortion occurs when the lens and the image plane are not parallel
Lin ZHANG, SSE, 2016
Single Camera Calibration
• To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion
• Tangential distortion occurs when the lens and the image plane are not parallel
2 21 2
2 22 1
+ 2 2
2 2
distorted
distorted
u u uv r u
v v uv r v
are the tangential distortion coefficients of the lens 1 2,
Lin ZHANG, SSE, 2016
Single Camera Calibration
• The purpose of the camera calibration is to determine the values for the extrinsics, intrinsics, and distortion coefficients
• How to do?• Zhengyou Zhang’s method[1] is a commonly used modern approach
• A calibration board is needed; several images of the board need to be captured; based on the correspondence pairs (pixel coordinate and world coordinate of a feature point), equation systems can be obtained; by solving the equation systems, parameters can be determined
[1] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000
Lin ZHANG, SSE, 2016
Single Camera Calibration
Calibration board
Lin ZHANG, SSE, 2016
Single Camera Calibration
A set of Calibration board images (50~60)
Lin ZHANG, SSE, 2016
Single Camera Calibration
• Matlab provides a “Camera Calibrator”• Straightforward to use• However, based on my experience, it is not as accurate as the routine provided in openCV3.0, especially for large FOV cameras (such as fisheye camera); thus, for some accuracy critical applications, I recommend to use the openCV function, though a little more complicated
• It exports “cameraParams” as the calibration result
Lin ZHANG, SSE, 2016
Single Camera Calibration
Lin ZHANG, SSE, 2016
Single Camera Calibration
• For our purpose (measuring geometric metrics of a planar object), we use the camera parameters to undistort the image
• The essence of this step is to make sure the transformation from a physical plane to the image plane can be represented by a linear projective matrix; or in other words, a straight line should be mapped to a straight line
Lin ZHANG, SSE, 2016
Single Camera CalibrationOriginal image
Undistorted image
Lin ZHANG, SSE, 2016
Contents
• Foundations of Projective Geometry• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• Homograpy Estimation• What is Camera Calibration• Single Camera Calibration• Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Our task is to measure the geometric properties of objects on a plane (e.g., conveyor belt)
• Such a problem can be solved if we have its bird‐view image; bird’s‐eye‐view is easy for object detection and measurement
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Three coordinate systems are required• Bird’s‐eye‐view image coordinate system• World coordinate system• Undistorted image coordinate system
X
Y
X
Y
X
Y
Bird’s‐eye‐view image WCS Undistorted image
Similarity Projective
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is
B WP
W IP
Then, given a position on bird’s‐eye‐view, we can get its corresponding position in the undistorted image as
, ,1 TB Bx y
1x
B
I W I B W B
xP P y
• Basic idea for bird’s‐eye‐view generation
Then, the intensity of the pixel can be determined using some interpolation technique based on the neighborhood around on the undistorted image
, ,1 TB Bx y
xI
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is
B WP
W IP
• Basic idea for bird’s‐eye‐view generation
The key problem is how to obtain and ?B WP W IP
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Determine B WP
X
Y
X
Y
H(mm)
M (pixels)
N (pixels)
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Determine B WP
02
02
1 11 0 0 1
W B B
W B B W B
H HNM Mx x x
H Hy y P yM
For a point on bird’s‐eye‐view, the corresponding point on the world coordinate system is,
, ,1 TB Bx y
Please verify!!
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Determine W IP
The physical plane (in WCS) and the undistorted image plane can be linked via a homography matrix W IP
x xI W I WP
If we know a set of correspondence pairs , 1,x x N
Ii Wi i
W IP can be estimated using the least‐square method
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
• Determine W IP
A set of point correspondence pairs; for each pair, we know its coordinate on the undistorted image plane and its coordinate in the WCS
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
When and are known, the bird’s‐eye‐view can be generated via,
W IP B WP
1 1x
B B
I W I B W B B I B
x xP P y P y
Bird’s‐eye‐view Generation
Lin ZHANG, SSE, 2016
Bird’s‐eye‐view Generation
Original image Bird’s‐eye‐view
Lin ZHANG, SSE, 2016
Bird‐view GenerationAnother example
Original fish‐eye image Undistorted image
Lin ZHANG, SSE, 2016
Bird‐view GenerationAnother example
Bird’s‐eye‐viewOriginal fish‐eye image
Lin ZHANG, SSE, 2016
Thanks for your attention