today - cse.sc.educse.sc.edu/~tongy/csce867/lect/lect6.pdf · estimate projection matrix by svd the...
TRANSCRIPT
Today
Camera calibration
Conventional Camera Calibration
Input: pairs of 3D object points and 2D image points
General strategy:
• Take pictures from calibration patterns with known coordinates of 3D features (e.g., points, lines, or curves)
• Identify the corresponding image features
• obtain the projection matrix by minimizing projection error
• obtain intrinsic parameters from the projection matrix
11,vu
22 ,vu
33 ,vu
NN vu ,
111 ,, ZYX
222 ,, ZYX
333 ,, ZYX
NNN ZYX ,,
Camera Calibration from Projection Matrix P:
A Linear Method
0'' )(
34
)()(
314
)(
1 IIWtWt upuMpM pp
0'' )(
34
)()(
324
)(
2 IIWtWt vpvMpM pp
For every pair of 3D-2D points, we have a pair of linear equations:
11,vu
22 ,vu
33 ,vu
NN vu ,
111 ,, ZYX
222 ,, ZYX
333 ,, ZYX
NNN ZYX ,,
N pairs We can setup a linear equation systems for N
corresponding pairs with 2N equations:
• Known: 2D and 3D coordinates
• Unknown: 4x3 parameters of P
Av = 0
A is a 2𝑁 × 12 matrix containing the known
v is a 12 × 1 vector containing the unknown tttt ppp 343242141 pppv
Camera Calibration from Projection Matrix P:
A Linear Method
)()()()(
31
)()()(
31
)(
)(
1
)(
1
)(
1
)(
131
)(
1
)(
1
)(
131
)(
1
'1'00
'001'
'1'00
'001'
I
N
I
N
tW
N
tW
N
I
N
I
N
tW
N
tW
N
IItWtW
IItWtW
vvMM
uuMM
vvMM
uuMM
A
Require at least 6 pairs of non-coplanar points
Estimate Projection Matrix by SVD
The projection matrix can be estimated as a solution to
The solution lies in the null space of A – the set of all vectors x
that satisfies Ax =0.
𝑟𝑎𝑛𝑘 𝐴 + 𝑛𝑢𝑙𝑙𝑖𝑡𝑦 𝐴 = min(𝑟𝑜𝑤 𝐴 , 𝑐𝑜𝑙 𝐴 )
• Case 1: If rank(A) = 11, the solution to v is unique up
to a scaling factor
• Case 2: if rank(A) < 11, the solution to v is a linear
combination of all null vectors of A
Av = 0
Estimate v by performing SVD on A
What about rank(A)=12?
Brief Review on SVD
Singular Value Decomposition:
• Any mxn matrix can be written as the product of three matrices
T
nnnmmmnm VDUA
Singular values 𝜎𝑖 are fully determined by A
D: 𝑑𝑖𝑗 = 0 𝑖𝑓 𝑖𝑗; 𝑑𝑖𝑖 = 𝜎𝑖 (𝑖 = 1,2, … , 𝑛)
s1 s2 … sN 0; 𝑁 = 𝑟𝑎𝑛𝑘(𝐴)
Both U and V are not unique
Columns of each are mutual orthogonal vectors
nnnn
n
n
n
mmmm
m
m
mnmm
n
n
vvv
vvv
vvv
uuu
uuu
uuu
aaa
aaa
aaa
21
22212
12111
2
1
21
22221
11211
21
22221
11211
00
0
0
00
s
s
s
𝑽𝟏𝑻
U1
Brief Review on SVD
U contains the eigenvector of 𝐴𝐴𝑇
Columns of U called the left-singular vectors
V contains the eigenvector of 𝐴𝑇𝐴 Columns of V called the right-singular vectors
Both U and V are unitary matrices:
𝑈𝑈𝑇 = 𝐼𝑚×𝑚 and 𝑉𝑉𝑇 = 𝐼𝑛×𝑛
Estimate Projection Matrix by SVD
When rank(A) = 11, v can be estimated by performing SVD on A
v is the last column of the V matrix with an unknown scalar
How to solve the scalar 𝛼?tt
33 rp and tt
33 rp 13 r
2
11
2
10
2
9 '''
1
vvv
'vv The null vector of A
Projection matrix P is solved!
The sign of 𝛼 sign(𝛼) is
unknown!
tttt ppp 343242141 pppv
Discussion on the Rank of A
• Rank(A)=11 with a proper configuration of the 3D points
• In practice, rank(A)=12 (full rank) may happen due to the
image noise and error in location estimation
• Rank(A)<11
• Rank(A)=8 if the 3D points are coplanar
• Results in infinite solutions
Issues in the Linear Method
• Numerical issue
• Outliers in observations
Numerical Stability: Data Normalization
Motivation: the entries in A have large difference in unit and scale
)()()()(
31
)()()(
31
)(
)(
1
)(
1
)(
1
)(
131
)(
1
)(
1
)(
131
)(
1
'1'00
'001'
'1'00
'001'
I
N
I
N
tW
N
tW
N
I
N
I
N
tW
N
tW
N
IItWtW
IItWtW
vvMM
uuMM
rrMM
uuMM
A
Unit: mm
scale: vary
Unit: no
Scale: 0 or 1
Unit: mm*pixel
scale: vary
Unit: pixel
scale: the image size
Numerical instable: resulting problem when data
contains significant noise
Solution: data normalization
Numerical Stability: Data Normalization
• Replace A by A’ by centering and normalizing each point,
respectively.
• Centering: translate 2D/3D points wrt their centroid
• Normalization: scaling the coordinates by a scale factor
• E.g., the scale factor can be computed as the inverse of
average distance to the centroid
m′ = H𝟐𝑫 m
M′ = H𝟑𝑫 M
H𝟐𝑫 =
𝟏
𝒔𝟐𝑫𝟎 −
𝒎𝒙
𝒔𝟐𝑫
𝟎𝟏
𝒔𝟐𝑫−𝒎𝒚
𝒔𝟐𝑫𝟎 𝟎 𝟏
H𝟑𝑫 =
𝟏
𝒔𝟑𝑫𝟎 𝟎 −
𝑴𝑿
𝒔𝟑𝑫
𝟎𝟏
𝒔𝟑𝑫𝟎 −
𝑴𝒀
𝒔𝟑𝑫
𝟎 𝟎𝟏
𝒔𝟑𝑫−𝑴𝒁
𝒔𝟑𝑫𝟎 𝟎 𝟎 𝟏
Numerical Stability: Data Normalization
• Replace A by A’ by centering and normalizing each point,
respectively.
• Solve the problem of and recover P′ from v′
• Recover P=H𝟐𝑫
−𝟏P′H𝟑𝑫
A′v′ = 0
Note: this preprocessing step is recommended for all
methods involving image data
m′ = H𝟐𝑫 m
M′ = H𝟑𝑫 M
H𝟐𝑫 =
𝟏
𝒔𝟐𝑫𝟎 −
𝒎𝒙
𝒔𝟐𝑫
𝟎𝟏
𝒔𝟐𝑫−𝒎𝒚
𝒔𝟐𝑫𝟎 𝟎 𝟏
H𝟑𝑫 =
𝟏
𝒔𝟑𝑫𝟎 𝟎 −
𝑴𝑿
𝒔𝟑𝑫
𝟎𝟏
𝒔𝟑𝑫𝟎 −
𝑴𝒀
𝒔𝟑𝑫
𝟎 𝟎𝟏
𝒔𝟑𝑫−𝑴𝒁
𝒔𝟑𝑫𝟎 𝟎 𝟎 𝟏
Robust Linear Method with RANSAC
The linear method is sensitive to image noise and localization errors.
outliers
Estimated line
Ideal line
Robust Linear Method with RANSAC
A better solution can be obtained by using a robust method such as
the RANSAC (Random Sample Consensus) method.
Assumptions:
• The model can be estimated from K data items
• There are N (N>K) data items in total
• The whole data set contains a fraction 𝛼 of outliers
• A guess about the probability that the training set does not
contain any outliers
Robust Linear Method with RANSAC
• Use green points to estimate
the parameters
• Use red points to compute
errors
Robust Linear Method with RANSAC
Repeat: Step 1: randomly partition all N points into a training set (with K>6 points)
and a testing set (with N-K points).
Step 2: compute the projection matrix P using the training set
Step 3: for each point in the testing set, compute its projection error
if , increment a counter for “inliers”
Until run a sufficient number of times
Step 4: use the training set resulting the highest number of inliers plus all
the inlier points to recompute P
vAi
Threshold
How to determine?
Robust Linear Method with RANSAC
The probability that all K data in a subset are good is
The probability that all 𝑠 different subsets/iterations will contain
at least one or more ourliers is
The probability that at least one random subset has no outliers is
given by
The number of iterations/subsets needed is computed as
sKP ))1(1(1
K)1(
sK ))1(1(
))1(1ln(
)1ln(K
Ps
343
242
141
3
0302
0301
p
p
p
t
tvtfvf
tutfuf
Pt
t
t
z
t
zYv
tt
v
zXu
tt
u
p
p
p
r
rr
rr
Estimate Camera Parameters from P
310 pptu 320 pp
tv Why?
34ptz 33 pr
sign(𝛼)
is determined from
is negative if the object is before the camera Zt
Zt
How to determine
343
242
141
3
0302
0301
p
p
p
t
tvtfvf
tutfuf
Pt
t
t
z
t
zYv
tt
v
zXu
tt
u
p
p
p
r
rr
rr
Estimate Camera Parameters from P
2
011 uf t
u pp2
022 vf t
v pp
uX fpupt /34014 vY fpvpt /34024
ufu /3011 ppr vfv /3022 ppr