Structure from motion
• Multi-view geometry• Affine structure from motion• Projective structure from motion
Planches :– http://www.di.ens.fr/~ponce/geomvis/lect4.ppt – http://www.di.ens.fr/~ponce/geomvis/lect4.pdf
Epipolar Constraint
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
Epipolar Constraint: Calibrated Case
Essential Matrix(Longuet-Higgins, 1981)
Properties of the Essential Matrix
• E p’ is the epipolar line associated with p’.
• E p is the epipolar line associated with p.
• E e’=0 and E e=0.
• E is singular.
• E has two equal non-zero singular values (Huang and Faugeras, 1989).
T
T
Epipolar Constraint: Small MotionsTo First-Order:
Pure translation:Focus of Expansion
Epipolar Constraint: Uncalibrated Case
Fundamental Matrix(Faugeras and Luong, 1992)
Properties of the Fundamental Matrix
• F p’ is the epipolar line associated with p’.
• F p is the epipolar line associated with p.
• F e’=0 and F e=0.
• F is singular.
T
T
The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F | =1.
Minimize:
under the constraint2
Non-Linear Least-Squares Approach (Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an appropriate rank-2 parameterization.
The Normalized Eight-Point Algorithm (Hartley, 1995)
• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’.
• Use the eight-point algorithm to compute F from thepoints q and q’ .
• Enforce the rank-2 constraint.
• Output T F T’.T
i i i i
i i
Data courtesy of R. Mohr and B. Boufama.
With
out n
orm
aliz
atio
nW
ith n
orm
aliz
atio
nMean errors:10.0pixel9.1pixel
Mean errors:1.0pixel0.9pixel
Trinocular Epipolar Constraints
These constraints are not independent!
Trinocular Epipolar Constraints: Transfer
Given p and p , p can be computed
as the solution of linear equations.
1 2 3
Trifocal Constraints
Trifocal Constraints
All 3x3 minorsmust be zero!
Calibrated Case
Trifocal Tensor
Trifocal ConstraintsUncalibrated Case
Trifocal Tensor
Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p .Do it again.
2 3 2 3T( p , p , p )=01 2 3
Properties of the Trifocal Tensor
Estimating the Trifocal Tensor
• Ignore the non-linear constraints and use linear least-squaresa posteriori.
• Impose the constraints a posteriori.
• For any matching epipolar lines, l G l = 0.
• The matrices G are singular.
• They satisfy 8 independent constraints in theuncalibrated case (Faugeras and Mourrain, 1995).
2 1 3T i
1i
For any matching epipolar lines, l G l = 0. 2 1 3T i
The backprojections of the two lines do not define a line!
Multiple Views (Faugeras and Mourrain, 1995)
Two Views
Epipolar Constraint
Three Views
Trifocal Constraint
Four Views
Quadrifocal Constraint(Triggs, 1995)
Geometrically, the four rays must intersect in P..
Quadrifocal Tensorand Lines
Scale-Restraint Condition from Photogrammetry
The Euclidean (perspective) Structure-from-Motion Problem
Given m calibrated perspective images of n fixed points Pj we can write
Problem: estimate the m 3x4 matrices Mi = [Ri ti] and
the n positions Pj from the mn correspondences pij .
2mn equations in 11m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
The Euclidean Ambiguity of Euclidean SFM
If Ri, ti, and Pj are solutions,
So are Ri’, ti’, and Pj’, where
In fact, the absolute scale cannot be recovered since:
When the intrinsic and extrinsic parameters are known
Euclidean ambiguity up to a similarity transformation.
The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write
Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .
ij ij
2mn equations in 8m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
j
The Affine Ambiguity of Affine SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and
Q is an affinetransformation.
When the intrinsic and extrinsic parameters are unknown
The Affine Epipolar Constraint
Note: the epipolar lines are parallel.
Affine Epipolar Geometry
The Affine Fundamental Matrix
where
With
out n
orm
aliz
atio
nW
ith n
orm
aliz
atio
nMean errors:10.0pixel9.1pixel
Mean errors:1.0pixel0.9pixel
Perspective case..
Mean errors: 3.24 and 3.15pixel (without normalization160.92 and 158.54pixel).
Affine case..
An Affine Trick..
The Affine Epipolar Constraint
Note: the epipolar lines are parallel.
An Affine Trick.. Algebraic Scene Reconstruction Method
Affine reconstruction. Mean relative error: 3.2%
The Affine Structure of Affine Images
Suppose we observe a static scene with m fixed cameras..
The set m-tuples of allimage points in a sceneis a 3D affine space!
When do m+1 points define a p-dimensional subspace Y of ann-dimensional affine space X equipped with some coordinateframe basis?
Writing that all minors of size (p+2)x(p+2) of D are equal tozero gives the equations of Y.
Rank ( D ) = p+1, where
has rank 4!
From Affine to Vectorial Structure
Idea: pick one of the points (or their center of mass)as the origin.
Singular Value Decomposition
Singular Value Decomposition square roots of
Singular Value Decomposition
Singular Value Decomposition
What if we could factorize D? (Tomasi and Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
Affine reconstruction. Mean relative error: 2.8%
Back to perspective:Euclidean motion from E (Longuet-Higgins, 1981)
• Given F computed from n > 7 point correspondences, and its SVD F= UWVT, compute E=U diag(1,1,0) VT.
• There are two solutions t’ = u3 and t’’ = -t’ to ETt=0.
• Define R’ = UWVT and R” = UWTVT where
(It is easy to check R’ and R” are rotations.)
• Then [tx’]R’ = -E and [tx’]R” = E. Similar reasoning for t”.
• Four solutions. Only two of them place the reconstructedpoints in front of the cameras.
100001010
W
Euclidean reconstruction. Mean relative error: 3.1%
A different view of the fundamental matrix
• Projective ambiguity ! M’Q=[Id 0] MQ=[A b].
• Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1)T.
• This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b.
• Or: z (b x p) = z’ (b x Ap’).
• Finally: pTFp’ = 0 with F = [bx] A.
Projective motion from the fundamental matrix
• Given F computed from n > 7 point correspondences, compute b as the solution of FTb=0 with |b|2=1.
• Note that: [ax]2 = aaT - |a|2Id for any a.
• Thus, if A0 = - [bx] F,
[bx] A0 = - [bx]2 F = - bbTF + |b|2 F = F.
• The general solution is M = [A b] with
A = A0 + ( b | b | b).
Two-view projective reconstruction. Mean relative error: 3.0%
Bundle adjustment
Use nonlinear least-squares to minimize:
Bundle adjustment. Mean relative error: 0.2%
Projective SFM from multiple images
z11p11 … z1np1n
… … …zm1pm1 … zmnpmn
M1
…Mm
P_1 … P_n= , D = MP
• If the zij’s are known, can be done via SVD. In principlethe zij’s can be found pairwise from F (Triggs 96).
• Alternative, eliminate zij from the minimization of E=|D-MP|2
• This reduces the problem to the minimization ofE = ij |pij x MiPj|2
under the constraints |Mi|2=|Pj|2=1 with |pij|2=1.
• Bilinear problem.
Bilinear projective reconstruction. Mean relative error: 0.2%
From uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera:
Problem: what is Q ?
Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.
Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.
What is some parameters are known?
Weak-perspective camera:
Zero skew:
Problem: what is Q ?
0
Self calibration!
П1
Chasles’ absolute conic: x12+x2
2+x32 = 0, x4 = 0.
Kruppa (1913); Maybank & Faugeras (1992)
Triggs (1997);Pollefeys et al. (1998,2002)
, u0, v0
The absolute quadric u0 = v0 = 0The absolute quadratic complex 2 = 2, = 0
u0
v0
kl
f
x’ ≈ P ( H H-1 ) xH = [ X y ]
Relation between K, , and *