geometry of multiple views - ele.puc-rio.brvisao/cap 4 geometry of multiple views rio2.pdf ·...
TRANSCRIPT
10/05/2017 Geometry of Multiple Views 1
Geometry of Multiple Views
Raul Queiroz Feitosa
10/05/2017 Geometry of Multiple Views 2
Objetive
This chapter elucidates the geometric and
algebraic constraints that hold among two views
of the same scene.
10/05/2017 Geometry of Multiple Views 3
Geometry of Multiple Views
Contents: Epipolar Geometry
The Calibrated Case
The Uncalibrated Case
Computing E and F
The 8-point Algorithm
The Luong linear method
The Luong non linear method
The Hartley method
10/05/2017 Geometry of Multiple Views 4
Epipolar Geometry
define the epipolar plane
epipolar lines
baseline
virtual image
plane
virtual image
plane
epipoles
^
^
^
^
10/05/2017 Geometry of Multiple Views 5
Epipolar Geometry
The Epipolar Constraint
The point corresponding to p on the right camera
lies necessarily over the epipolar line l´ of p.
^
^
^
^
^
^
^
^
10/05/2017 Geometry of Multiple Views 6
The Calibrated Case
The epipolar constraint implies that
where: is the homogeneous coordinate vector on the left normalized image
is the homogeneous coordinate vector on the right normalized image
t is the coordinate vector of the translation from O to O’
R is the rotation matrix relating both camera frames
[t] is the skew symmetric matrix such that [t] a = t a
0ˆˆˆˆ0ˆˆ0ˆˆ
'''pp pt ppt p ERR T pOOOpO
p̂
pˆ
^ ^
^
^
^
10/05/2017 Geometry of Multiple Views 7
The Calibrated Case
The epipolar constraint implies that
recall that
0ˆˆˆˆ0ˆˆ0ˆˆ
'''pp pt ppt p ERR T pOOOpO
^ ^
if t =[ tx ty tz]T then [t] =
0 -tz ty
tz 0 -tx
-ty tx 0
10/05/2017 Geometry of Multiple Views 8
The Essential Matrix
Definition:
E = [t] R
Properties:
E ′ (ET ) is the vector that represents the epipolar line
associated to the point ′ ( ) on the left (right) image.
E is singular with two equal nonzero singular values.
the eigenvector of E (E T ) associated to eigenvalue equal to zero
is the coordinate vector of the epipole e ′ (e) of the left (right)
camera.
The Calibrated Case
0ˆˆ 'pp ET
p̂p̂
p̂p̂
^ ^
10/05/2017 Geometry of Multiple Views 9
The Uncalibrated Case
From the previous chapter
This implies that
where F = K-T E K′-1 is the fundamental matrix.
p p p p ˆ and ˆ KK
0'pp FT
10/05/2017 Geometry of Multiple Views 10
The Uncalibrated Case
The Fundamental Matrix
Definition:
F = K-T E K′-1
Properties:
F p ′ (FT p ) is the vector that represents the epipolar line
associated to the point p′ (p) on the left (right) image.
F has rank equal to 2.
the eigenvector of F (F T ) associated to eigenvalue equal to zero
is the coordinate vector of the epipole e ′ (e) of the left (right)
camera (why?).
0'pp FT
10/05/2017 Geometry of Multiple Views 11
The Uncalibrated Case
F is defined up to a scale factor ( )
F has rank 2 → det (F )=0
Thus F admits seven independent parameters.
F can be parameterized this way:
where e=(α,β) and e´=(α´,β´) are the epipoles.
0'pp FT
badcacbd
dccd
baab
F
10/05/2017 Geometry of Multiple Views 12
Computing E and F The Eight-point Algorithm
From the previous slide
since this equation is homogeneous in the coefficients of F we can set F33=1 (scale ambiguity eliminated), and use
eight point correspondences
0
1
10
333231
232221
131211
v
u
v,u,T
FFF
FFF
FFF
F 'pp
10/05/2017 Geometry of Multiple Views 13
For exactly 8 pairs of corresponding points:
Note that the rank 2 property of F is ignored!
Computing E and F The Eight-point Algorithm
1
1
1
32
12
11
888888888888
222222222222
111111111111
F
F
F
vuvvvuvuvuuu
vuvvvuvuvuuu
vuvvvuvuvuuu
10/05/2017 Geometry of Multiple Views 14
For more than 8 pairs of corresponding points, apply least
square to minimize
which leads to the overconstrained homogeneous system
Note that the rank 2 property of F is ignored!
Computing E and F The Eight-point Algorithm
0
11
1
1
12
11
222222222222
111111111111
F
F
pppppppppppp vuvvvuvuvuuu
vuvvvuvuvuuu
vuvvvuvuvuuu
p
1
i
T
i
i
pp 2
F
10/05/2017 Geometry of Multiple Views 15
Step 1:
Starting with F produced by the eight-point algorithm,
use least square to find the epipoles e=(α,β) and e´=(α´,β´) that minimize |F T e|2 and |F e´|2 .
Recall that e (e´)|2 is the eigenvector of F(F T ) corresponding to the zero
eingenvalue.
Computing E and F The Luong linear method
10/05/2017 Geometry of Multiple Views 16
Step 2:
Substitute α, β, α´ and β´ in equation 10.6 below
This leads to a linear parameterization of the matrix F in
a, b, c and d.
Computing E and F The Luong linear method
badcacbd
dccd
baab
F
10/05/2017 Geometry of Multiple Views 17
Step 3 :
Estimate a, b, c and d using least square by minimizing
Recall that the parameterization of eq. 10.6 guarantees
the rank-2 property!
Computing E and F The Luong linear method
p
1
i
T
i
i
pp 2
F
10/05/2017 Geometry of Multiple Views 18
Minimize
where
denotes the (signed geometric distance between the
point p and the line l, and
Fp and FTp´ are the epipolar lines associated to p and p´ .
The result of the linear algorithm is a good starting
solution.
Computing E and F
The Luong non linear method
p
1i
i
T
ii pppp FF ,d,d 22
lp ,d
10/05/2017 Geometry of Multiple Views 19
The geometric distance in this case will be given by:
where the subscript indicates the component of a vector.
Computing E and F
The Luong non linear method
2
2
2
1
2
2 ,dii
iiii
pp
pppp
FF
FF
22
2
1
2
2
2
2
1
2
2 ,d
i
T
i
T
ii
i
T
i
T
i
T
ii
T
i
pp
pp
pp
pppp
FF
F
FF
FF
10/05/2017 Geometry of Multiple Views 20
Thus the function to be minimized is
Computing E and F
The Luong non linear method
p
i
ii
i
T
i
Tii
2
2
2
2
1
2
2
2
1
11pp
ppppF
FFFF
10/05/2017 Geometry of Multiple Views 21
Step 1:
Transform the image coordinates using appropriate
translation and scaling operator:
so that they are centered at the origin and have average
distance to the origin equal to .
What are these transforms?
Computing E and F
The Hartley method
ii pp ~ii pp ~
2
10/05/2017 Geometry of Multiple Views 22
Step 2:
Use linear least square to compute minimizing
Step 3:
Construct the singular value decomposition of ; assuming that S=diag (r,s,t) , where t ≤ s ≤ r, compute
Step 4:
Compute the final estimate
Computing E and F
The Hartley method
F~
p
1
i
T
i
i
pp 2~~~ F
T SF ~
Tr,s,0 F diag
FF T
10/05/2017 Geometry of Multiple Views 23
Computing E and F
10/05/2017 Geometry of Multiple Views 24
Computing E and F
Using RANSAC to deal with outliers: Determine:
n=9 – the smallest number of points required
k – the number of iterations required
t – the Sampson distance threshold (pixels) to identify a point that fits well
d – the number of nearby points required
Until k iterations have occurred
Draw a sample of n points from the data uniformly and at random
Apply some algorithm to compute the Fundamental Matrix that best fits that set of n points
For each data point outside the sample Compute the geometric distance (slide 19) to the epipolar lines and test it against t
end
If there are d or more points close to the function, then there is a good fit
Refit the geometric Matrix using these points
end
Use the best fit from this collection, using the fitting error as criterion.
Exercise 1
Download the package of MATLAB functions that
implement approaches for fundamental matrix
estimation. Test_RANSAC_Feature_Matching is the key
function. Read its help and follow the recommendations
below: In the first run collect a set of at least 8 true correspondences and a
couple of false correspondences;
Test different approaches for fundamental matrix estimation;
Disable the RANSAC by setting the input parameter Dmax to a very
large number;
Save the figures produced at the end of each run in a separate .fig file
and compare the results.
10/05/2017 Geometry of Multiple Views 25
Exercise 2
Compare the F matrix estimation obtained in
previous exercise with the ones obtained with
methods implemented in MATLAB
(estimateFundamentalMatrix).
10/05/2017 Geometry of Multiple Views 26
10/05/2017 Geometry of Multiple Views 27
Next Topic
Stereopsis