the trifocal tensor multiple view geometry. scene planes and homographies plane induces homography...

The Trifocal Tensor

Multiple View Geometry

Scene planes and homographies

plane induces homography between two views

0HFFH TT

H'eF

ve'Fe'H

Hee'

6-point algorithm

6655 Hx'xHx'xe'

x1,x2,x3,x4 in plane, x5,x6 out of plane

Compute H from x1,x2,x3,x4

He'F

Three-view geometry

The trifocal tensor

Three back-projected lines have to meet in a single line

Incidence relation provides constraint on lines

Let us derive the corresponding algebraic constraint…

Notations

iii lll

0]|[IP ]a|[AP' 4 ]b|[BP" 4

0

llPπ T

l'al'Al'P'π' T

4

TT

l"bl"Bl"P"π" T

4

TT

Incidence

iii lll

0

llPπ T

l'al'Al'P'π' T

4

TT

l"bl"Bl"P"π" T

4

TT

2rank is ]π"π'[πM ,,

e.g. is part of bundle formed by ’ and ”

l"bl'a0l"Bl'Al]m,m,m[M T

4T

4

TT

321

321 βmαmm "lβb'lαa0 T4

T4

'la T4k

"lb T

4k "lB'la'lA"lbl TT4

TT4

Incidence relation

"lBa'l'lAb"l"lB'la'lA"lbl T4

TT4

TTT4

TT4

"lba'l'lab"l T4

TT4

Tiiil "lba'l"lba'l T

4TT

4T

ii

T4

T4 babaT iii

"lT'l Tiil

The Trifocal Tensor

"lT,T,T'll 321TT

Trifocal Tensor = {T1,T2,T3}

"lT'l,"lT'l,"lT'l 3T

2T

1T

Only depends on image coordinates and is thus independent of 3D projective basis

"lT'll' TTi 'lT"ll" TT

iAlso and but no simple relation

General expression not as simple asT

4T

4 babaT iii

DOF T: 3x3x3=27 elements, 26 up to scale 3-view relations: 11x3-15=18 dof

8(=26-18) independent algebraic constraints on T(compare to 1 for F, i.e. rank-2)

Homographies induced by a plane

Line-line-line relation

Eliminate scale factor:

(up to scale)

Point-line-line relation

Point-line-point relation

note: valid for any line through x”, e.g. l”=[x”]xx”arbitrary

Point-point-point relation

note: valid for any line through x’, e.g. l’=[x’]xx’arbitrary

Overview incidence relations

Non-incident configurationincidence in image does not guarantee incidence in space

Epipolar lines

if l’ is epipolar line, then satisfied for arbitrary l”

inversely,

epipolar lines are right and left null-space of

Epipoles

With points

becomes respectively

Epipoles are intersection of right resp. left null-space of

(e=P’C and e”=P”C)

Extracting F

21Fgood choice for l” is e” (V3

Te”=0)

Computing P,P‘,P“

?

ok, but not

specifically, (no derivation)

matrix notation is impractical

Use tensor notation instead

0" jkikj

i Tllx

jkiT

Definition affine tensor

• Collection of numbers, related to coordinate choice, indexed by one or more indices

• Valency = (n+m)• Indices can be any value between 1

and the dimension of space (d(n+m)

coefficients)

Conventions

0iijbA

Einstein’s summation:(once above, once below)

ii

iji

ij bAbA

Index rule: jbA iij ,0

Contravariant indices

Covariant indices

More on tensors

• Transformations

iji

j xAx

ijji llA

(covariant)

(contravariant)

Some special tensors

• Kronecker delta

• Levi-Cevita epsilon

(valency 2 tensor)

(valency 3 tensor)

Trilinearities

Compute F and P from T

matrix notation is impractical

Use tensor notation instead

0 jkikj

i Tllx

jkiT

Definition affine tensor

• Collection of numbers, related to coordinate choice, indexed by one or more indices

• Valency = (n+m)• Indices can be any value between 1

and the dimension of space (d(n+m)

coefficients)

Conventions

0iijbA

Contraction:(once above, once below)

ii

iji

ij bAbA

Index rule: jbA iij ,0

More on tensors

• Transformations

iji

j xAx

ijji llA

(covariant)

(contravariant)

Some special tensors

• Kronecker delta

• Levi-Cevita epsilon

(valency 2 tensor)

(valency 3 tensor)

Trilinearities

Transfer: epipolar transfer

Transfer: trifocal transfer

Avoid l’=epipolar line

Transfer: trifocal transferpoint transfer

line transfer

degenerate when known lines are corresponding epipolar lines

Computation of Trifocal Tensor

• Linear method (7-point)

• Minimal method (6-point)

• Geometric error minimization method

• RANSAC method

Basic equations

Three points

Correspondence Relation #lin. indep.Eq.

4

Two points, one line

One points, two line

2

1

2Three lines

At=0 (26 equations)

(more equations)min||At|| with ||t||=1

Normalized linear algorithm

At=0

ijlmimk

ljjmk

liilk

mjjlk

mik TxxTxxTxxTxxx 0

03333 ilk

lk

iik

lk

lik TTxTxTxxx 2,1, li

Points

Lines

or

Normalization: normalize image coordinates to ~1

Normalized linear algorithm

ObjectiveGiven n7 image point correspondences across 3 images,or a least 13 lines, or a mixture of point and line corresp., compute the trifocal tensor.Algorithm(i) Find transformation matrices H,H’,H” to normalize 3 images(ii) Transform points with H and lines with H-1

(iii) Compute trifocal tensor T from At=0 (using SVD)(iv) Denormalize trifocal tensor st

r

k

t

j

sri

jki TT ˆ"H'HH 11

Internal constraints

27 coefficients 1 free scale 18 parameters 8 internal consistency constraints

(not every 3x3x3 tensor is a valid trifocal tensor!)

(constraints not easily expressed explicitly)

Trifocal Tensor satisfies all intrinsic constraintsif it corresponds to three cameras {P,P’,P”}

Maximum Likelihood Estimation

i

iiiiii ddd 222 "x̂,x"'x̂,x'x̂,x

iii x"x'x

iiiiii X"P"x̂,XP''x̂,X]0|I[x̂

data

cost function

parameterization

(24 parameters+3N)

also possibility to use Sampson error (24 parameters)

ObjectiveCompute the trifocal tensor between two images

Algorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute interest correspondences (and F) between 1&2 and 2&3

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select at random 6 correspondences and compute T

(b) Calculate the distance d for each putative match

(c) Compute the number of inliers consistent with T (d<t)

Choose T with most inliers

(iv) Optimal estimation: re-estimate T from all inliers by minimizing ML cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by computed T

Optionally iterate last two steps until convergence

Automatic computation of T

108 putative matches 18 outliers

88 inliers 95 final inliers

(26 samples)

(0.43)(0.23)

(0.19)

additional line matches

Matrix formulation for m-View

Consider one object point X and its m images: ixi=PiXi, i=1, …. ,m:

i.e. rank(M) < m+4 .

Laplace expansions

• The rank condition on M implies that all (m+4)*(m+4) minors of M are equal to 0.

• These can be written as sums of products of camera matrix parameters and image coordinates.

Matrix formulation

for non-trivially zero minors, one row has to be taken from each image (m).

4 additional rows left to choose

lk

jihgfedcba

000000000000000

ihgfedcba

jkl

only interesting if 2 or 3 rows from view

The three different types

1. Take the 2 remaining rows from one image block and the other two from another image block, gives the 2-view constraints.

2. Take the 2 remaining rows from one image block 1 from another and 1 from a third, gives the 3-view constraints.

3. Take 1 row from each of four different image blocks, gives the 4-view constraints.