Lecture 4.3 Estimating homographies from feature correspondences Thomas Opsahl
Homographies induced by central projection
2
π» π»
β’ Homography π»ποΏ½ = ποΏ½β²
π» =β1 β2 β3β4 β5 β6β7 β8 β9
β’ Point-correspondences can be determined automatically β’ Erroneous correspondences are common β’ Robust estimation is required to find π»
1 1 2 2
2 3 1
π πβ²
Homographies induced by central projection
3
π»
1 2 3
π»1β3 π»1β2
π» 1 2 1 2
β’ Homography π»ποΏ½ = ποΏ½β²
π» =β1 β2 β3β4 β5 β6β7 β8 β9
β’ Point-correspondences can be determined automatically β’ Erroneous correspondences are common β’ Robust estimation is required to find π»
π πβ²
Estimating the homography between overlapping images
β’ Establish point correspondences ππ β πβ²π β Find key points ππ β Img1 and πβ²π β Img2 β Represent key points by suitable descriptors β Determine correspondences ππ β πβ²π by matching descriptors β Some wrong correspondences are to be expected
β’ Estimate the homography π» such that πβ²π = π»ππ βπ
β Robust estimation with RANSAC β Improved estimation based on RANSAC inliers
β’ This homography enables us to compose the images into a
larger image β Image mosaicing β Panorama
4
Adaptive RANSAC
Objective To robustly fit a model π = π π;πΆ to a data set π containing outliers Algorithm 1. Let π = β, ππΌπΌ = β and #ππππππππππ = 0
2. while π > #ππππππππππ repeat 3-5
3. Estimate parameters πΆπ‘π‘π‘ from a random π-tuple from π
4. Determine inlier set ππ‘π‘π‘, i.e. data points within a distance π of the model π = π π;πΆπ‘π‘π‘
5. If ππ‘π‘π‘ > ππΌπΌ , set ππΌπΌ = ππ‘π‘π‘, πΆ = πΆπ‘π‘π‘, π = ππΌπΌ
π and π = πππ 1βπ
πππ 1βππ with π = 0.99 Increase #ππππππππππ by 1
5
Estimating the homography
β’ Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for π point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
6
Estimating the homography
β’ Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for π point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
β’ The homography has 8 degrees of freedom, but it is custom to treat all 9 entries of the matrix
as unknowns instead of setting one of the entries to 1 which excludes all potential solutions where this entry is 0
β’ Let us solve the equation π»ποΏ½ = ποΏ½β² for the entries of the homography matrix π»ποΏ½ = ποΏ½β² β1 β2 β3β4 β5 β6β7 β8 β9
π’π£1
=π’β²π£β²1
7
Basic homography estimation
8
1
2
3
1 2 3 1 2 3 4
4 5 6 4 5 6 5
7 8 9 7 8 9 6
7
8
9
0 0 0 11 0 0 0
1 1 1 0 0 0
hhh
h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h
hhh
β² β² β² β² β²+ + = β β β β² β² β² β² β²= β + + = β β β β β² β² β² β² β² β² + + = β β β
000
A
= β =
h 0
Basic homography estimation
Observe that the third row in π΄ is a linear combination of the first and second row πππ3 = βπ’β² β πππ1 β π£β² β πππ2
Hence every correspondence ππ β πβ²π contribute with 2 equations in the 9 unknown entries
9
1
2
3
1 2 3 1 2 3 4
4 5 6 4 5 6 5
7 8 9 7 8 9 6
7
8
9
0 0 0 11 0 0 0
1 1 1 0 0 0
hhh
h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h
hhh
β² β² β² β² β²+ + = β β β β² β² β² β² β²= β + + = β β β β β² β² β² β² β² β² + + = β β β
000
A
= β =
h 0
Basic homography estimation
β’ Since π» (and thus π) is homogeneous, we only need the matrix π΄ to have rank 8 in order to determine π up to scale
β’ It is sufficient with 4 point correspondences where no 3 points are collinear
β’ We can calculate the non-trivial solution to the equation π΄π = π by SVD
svd π΄ = ππππ
β’ The solution is given by the right singular vector without a singular value which is the last column of π, i.e. π = π9
10
1
2
31 1 1 1 1 1 1
41 1 1 1 1 1 1
52 2 2 2 2 2 2
62 2 2 2 2 2 2
7
8
9
0 0 0 1 01 0 0 0 0
0 0 0 1 01 0 0 0 0
hhhu v v u v v vhu v u u u v uhu v v u v v vhu v u u u v uhhhA
β² β² β² β β β β² β² β²β β β β² β² β² =β β β β² β² β²β β β
=
h 0
Basic homography estimation
β’ Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for π point-correspondences 2. A way to determine which of the point correspondences that are inliers for a given homography
Direct Linear Transform
1. Build the matrix π΄ from at least 4 point-correspondences π’π ,π£π β π’β²π ,π£β²π 2. Obtain the SVD of π΄: π΄ = ππππ 3. If π is diagonal with positive values in descending order along the main diagonal,
then π equals the last column of π 4. Reconstruct π» from π
1 1 1 1 1 1 1
1 1 1 1 1 1 1
0 0 0 11 0 0 0
u v v u v v vA u v u u u v u
β² β² β²β β β β² β² β²= β β β
11
Basic homography estimation
β’ The basic DLT algorithm is never used with more than 4 point-correspondences β’ This is because the algorithm performs better when all the terms of π΄ has a similar scale
β Note that some of the terms will always be of scale 1 β’ To achieve this, it is common to extend the algorithm with a normalization and a
denormalization step
Normalized Direct Linear Transform 1. Normalize the set of points ππ = π’π , π£π π by computing a similarity transform π that
translates the centroid to the origin and scales such that the average distance from the origin is 2
2. In the same way normalize the set of points πβ²π = π’β²π ,π£β²π π by computing a similarity transform πβ²
3. Apply the basic DLT algorithm on the normalized points to obtain a homography π»οΏ½ 4. Denormalize to get the homography: π» = πβ²β1π»οΏ½π
12
Basic homography estimation
β’ Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for π point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
β’ For a point correspondence π’π , π£π β π’β²π , π£β²π and homography π», we can choose from
several errors β Algebraic error: ππ = π΄ππ where
β Geometric errors: 1. ππ = π π»ππ ,πβ²π + π ππ ,π»β1πβ²π (Reprojection error) 2. ππ = π ππ ,π»β1πβ²π 3. ππ = π π»ππ ,πβ²π
0 0 0 11 0 0 0
i i i i i i ii
i i i i i i i
u v v u v v vA
u v u u u v uβ² β² β²β β β
= β² β² β²β β β
13
Notation
Euclidean distance π β,β
Inhomogenous π»ποΏ½π π»ππ
Inhomogeneous π»β1ποΏ½β²π π»β1πβ²π
Robust homography estimation
β’ Finally we would typically re-estimate π» from all correspondences in ππΌπΌ β Normalized DLT β Minimize Ο΅ = βππ in an iterative optimization method like Levenberg Marquardt
RANSAC estimation of homography For a set of point-correspondences S = ππ β¦ πβ²π , perform π iterations, where π is determined adaptively
1. Estimate π»π‘π‘π‘ from 4 random correspondences ππ β¦ πβ²π using the basic DLT algorithm 2. Determine the set of inlier-correspondences ππ‘π‘π‘ = ππ β¦ πβ²π ππ’π β πβππ ππ < π
Here one can choose ππ = π π»ππ ,πβ²π + π ππ ,π»β1πβ²π and π = 5.99π where π is the expected uncertainty in key-point positions
3. If ππ‘π‘π‘ > ππΌπΌ update π, homography and inlier set: π» = π»π‘π‘π‘, ππΌπΌ = ππ‘π‘π‘
14
Image mosaicing
15
β’ Let us compose these two images into a larger image
Image mosaicing
β’ Find key points and represent by descriptors
16
Image mosaicing
β’ Establish point-correspondences by matching descriptors β’ Several wrong correspondences
17
Image mosaicing
β’ Establish point-correspondences by matching descriptors β’ Several wrong correspondences
18
Image mosaicing
β’ Estimate homography π»ποΏ½ = ποΏ½β² β OpenCV
#include "opencv2/calib3d.hpp" cv::findHomography(srcPoints, dstPoints, CV_RANSAC);
β Matlab tform = estimateGeometricTransform(srcPoints,dstPoints,βprojectiveβ);
19
π»
Image mosaicing
β’ Represent the images in common coordinates (Note the additional translation!) β OpenCV
#include "opencv2/calib3d.hpp" cv::warpPerspective(img1, img2, H, output_size);
β Matlab img2 = imwarp(img1,tform);
20
Image mosaicing
β’ Now we can compose the images
21
22
Overwriting
23
Blending with a ramp
24
Blending with a ramp + histogram equalization
SVD How to use β’ Matlab
[U,S,V] = svd(A); Right singular vectors are columns in V
β’ OpenCV cv::SVD::compute(A, S, U, Vtranspose, cv::SVD::FULL_UV); Right singular vectors are rows in Vtranspose
β’ Eigen Eigen::JacobiSVD<Eigen::MatrixXd> svd(A, Eigen::ComputeFullU | Eigen::ComputeFullV); Right singular vectors are columns in svd.matrixV()
25
Singular Value Decomposition The singular value decomposition of a real π Γ π matrix π΄ is a factorization π΄ = πSππ Here π is a orthogonal π Γ π matrix, π is a orthogonal π Γ π matrix and π is a real positive diagonal π Γ π matrix The diagonal entries of π = diag π1, β¦ , πmin π,π are known as the singular values of π΄ and the columns of π = π1, β¦ ,ππ and π = π1, β¦ ,ππ are known as the left and right singular vectors of π΄ respectively The nullspace of A is the span of the right singular vectors ππ that corresponds to a zero singular value ππ (or does not have a corresponding singular value)
SVD Singular Value Decomposition The singular value decomposition of a real π Γ π matrix π΄ is a factorization π΄ = πSππ Here π is a orthogonal π Γ π matrix, π is a orthogonal π Γ π matrix and π is a real positive diagonal π Γ π matrix The diagonal entries of π = diag π1, β¦ , πmin π,π are known as the singular values of π΄ and the columns of π = π1, β¦ ,ππ and π = π1, β¦ ,ππ are known as the left and right singular vectors of π΄ respectively The nullspace of A is the span of the right singular vectors ππ that corresponds to a zero singular value ππ (or does not have a corresponding singular value)
Applications of SVD Solving homogeneous linear equations like
π΄π = π Method For theoretical problems, π‘ β null π΄ so π is a linear combination of the right singular vectors ππ that correspond to a zero singular value ππ
π = οΏ½ππππ ; ππ β β, ππ = 0 (or missing)
For practical problems, the presence of noise force us to expand the solution by including those right singular vectors that correspond to small singular values ππ β 0
π = οΏ½ππππ ; ππ β β, ππ β 0 (or missing)
26
SVD Example
From [U,S,V] = svd(A); we get
From this we see that A has: β’ 2 left singular vectors
β’ 2 nonzero singular values β’ 3 right singular vectors Since π3 does not have a corresponding singular value, π = π3 is a non-trivial solution to Aπ = 0 and π = π β π3; π β β\ 0 is the family of all non-trivial solutions
27
1 2 3 04 5 6 0
Axyz
=
=
x 0
0.3863 0.9224 9.5080 0 00.9224 0.3863 0 0.7729 0
0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082
U S
V
β β = = β
β = β β β β
1 2 3
0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082
β = β = = β β β
v v v
1 29.5080 0.7729s s= =
1 2
0.3863 0.92240.9224 0.3863β β
= = β u u
SVD Example
This time singular value decomposition give us the following singular values and right singular vectors:
This time all right singular vectors correspond to a non-zero singular value, so the equation does not have any non-trivial solutions!
If this equation came from a practical problem, instead of looking for solutions to π΄π = 0, we might be looking for the π that minimize π΄π Since π1 β 0, π2 β 0, π3 β 0, we would conclude that π = π3 solves the equation in a least-squares sense Check:
28
1.0792 2.0656 3.08494.0959 5.0036 6.0934 01.0679 2.0743 3.0655 04.0758 5.0392 6.0171
A
xyz
=
=
x 0
1 2 3
1 2 3
13.6295 1.0849 0.05060.4336 0.8103 0.39420.5635 0.0975 0.82030.7032 0.5778 0.4143
s s s= = =
β β = β = β = β β
v v v 3
0.00910.42880.01050.0341
A
= β β
v
Summary
β’ Homography π»ποΏ½ = ποΏ½β²
π» =β1 β2 β3β4 β5 β6β7 β8 β9
β’ Automatic point-correspondences
β’ Wrong correspondences are common
β’ RANSAC estimation
β Basic DLT (Direct Linear Transform) on 4 random correspondences
β Inliers determined from the reprojection error ππ = π π»ππ ,πβ²π + π ππ ,π»β1πβ²π
β’ Improve estimate by normalized DLT on inliers or iterative methods for an even better estimate
β’ Additional reading β Szeliski: 6.1.1 β 6.1.3
29