The Trifocal Tensor

Multiple View Geometry

Scene planes and homographies

plane induces homography between two views

0HFFH TT

H'eF

ve'Fe'H

Hee'

6-point algorithm

6655 Hx'xHx'xe'

x1,x2,x3,x4 in plane, x5,x6 out of plane

Compute H from x1,x2,x3,x4

He'F

Three-view geometry

The trifocal tensor

Three back-projected lines have to meet in a single line

Incidence relation provides constraint on lines

Let us derive the corresponding algebraic constraint…

Notations

iii lll

0]|[IP ]a|[AP' 4 ]b|[BP" 4

0

llPπ T

l'al'Al'P'π' T

4

TT

l"bl"Bl"P"π" T

4

TT

Incidence

iii lll

0

llPπ T

l'al'Al'P'π' T

4

TT

l"bl"Bl"P"π" T

4

TT

2rank is ]π"π'[πM ,,

e.g. is part of bundle formed by ’ and ”

l"bl'a0l"Bl'Al]m,m,m[M T

4T

4

TT

321

321 βmαmm "lβb'lαa0 T4

T4

'la T4k

"lb T

4k "lB'la'lA"lbl TT4

TT4

Incidence relation

"lBa'l'lAb"l"lB'la'lA"lbl T4

TT4

TTT4

TT4

"lba'l'lab"l T4

TT4

Tiiil "lba'l"lba'l T

4TT

4T

ii

T4

T4 babaT iii

"lT'l Tiil

The Trifocal Tensor

"lT,T,T'll 321TT

Trifocal Tensor = {T1,T2,T3}

"lT'l,"lT'l,"lT'l 3T

2T

1T

Only depends on image coordinates and is thus independent of 3D projective basis

"lT'll' TTi 'lT"ll" TT

iAlso and but no simple relation

General expression not as simple asT

4T

4 babaT iii

DOF T: 3x3x3=27 elements, 26 up to scale 3-view relations: 11x3-15=18 dof

8(=26-18) independent algebraic constraints on T(compare to 1 for F, i.e. rank-2)

Homographies induced by a plane

Line-line-line relation

Eliminate scale factor:

(up to scale)

Point-line-line relation

Point-line-point relation

note: valid for any line through x”, e.g. l”=[x”]xx”arbitrary

Point-point-point relation

note: valid for any line through x’, e.g. l’=[x’]xx’arbitrary

Overview incidence relations

Non-incident configurationincidence in image does not guarantee incidence in space

Epipolar lines

if l’ is epipolar line, then satisfied for arbitrary l”

inversely,

epipolar lines are right and left null-space of

Epipoles

With points

becomes respectively

Epipoles are intersection of right resp. left null-space of

(e=P’C and e”=P”C)

Extracting F

21Fgood choice for l” is e” (V3

Te”=0)

Computing P,P‘,P“

?

ok, but not

specifically, (no derivation)

matrix notation is impractical

Use tensor notation instead

0" jkikj

i Tllx

jkiT

Definition affine tensor

• Collection of numbers, related to coordinate choice, indexed by one or more indices

• Valency = (n+m)• Indices can be any value between 1

and the dimension of space (d(n+m)

coefficients)

Conventions

0iijbA

Einstein’s summation:(once above, once below)

ii

iji

ij bAbA

Index rule: jbA iij ,0

Contravariant indices

Covariant indices

More on tensors

• Transformations

iji

j xAx

ijji llA

(covariant)

(contravariant)

Some special tensors

• Kronecker delta

• Levi-Cevita epsilon

(valency 2 tensor)

(valency 3 tensor)

Trilinearities

Compute F and P from T

matrix notation is impractical

Use tensor notation instead

0 jkikj

i Tllx

jkiT

Definition affine tensor

• Collection of numbers, related to coordinate choice, indexed by one or more indices

• Valency = (n+m)• Indices can be any value between 1

and the dimension of space (d(n+m)

coefficients)

Conventions

0iijbA

Contraction:(once above, once below)

ii

iji

ij bAbA

Index rule: jbA iij ,0

More on tensors

• Transformations

iji

j xAx

ijji llA

(covariant)

(contravariant)

Some special tensors

• Kronecker delta

• Levi-Cevita epsilon

(valency 2 tensor)

(valency 3 tensor)

Trilinearities

Transfer: epipolar transfer

Transfer: trifocal transfer

Avoid l’=epipolar line

Transfer: trifocal transferpoint transfer

line transfer

degenerate when known lines are corresponding epipolar lines

Computation of Trifocal Tensor

• Linear method (7-point)

• Minimal method (6-point)

• Geometric error minimization method

• RANSAC method

Basic equations

Three points

Correspondence Relation #lin. indep.Eq.

4

Two points, one line

One points, two line

2

1

2Three lines

At=0 (26 equations)

(more equations)min||At|| with ||t||=1

Normalized linear algorithm

At=0

ijlmimk

ljjmk

liilk

mjjlk

mik TxxTxxTxxTxxx 0

03333 ilk

lk

iik

lk

lik TTxTxTxxx 2,1, li

Points

Lines

or

Normalization: normalize image coordinates to ~1

Normalized linear algorithm

ObjectiveGiven n7 image point correspondences across 3 images,or a least 13 lines, or a mixture of point and line corresp., compute the trifocal tensor.Algorithm(i) Find transformation matrices H,H’,H” to normalize 3 images(ii) Transform points with H and lines with H-1

(iii) Compute trifocal tensor T from At=0 (using SVD)(iv) Denormalize trifocal tensor st

r

k

t

j

sri

jki TT ˆ"H'HH 11

Internal constraints

27 coefficients 1 free scale 18 parameters 8 internal consistency constraints

(not every 3x3x3 tensor is a valid trifocal tensor!)

(constraints not easily expressed explicitly)

Trifocal Tensor satisfies all intrinsic constraintsif it corresponds to three cameras {P,P’,P”}

Maximum Likelihood Estimation

i

iiiiii ddd 222 "x̂,x"'x̂,x'x̂,x

iii x"x'x

iiiiii X"P"x̂,XP''x̂,X]0|I[x̂

data

cost function

parameterization

(24 parameters+3N)

also possibility to use Sampson error (24 parameters)

ObjectiveCompute the trifocal tensor between two images

Algorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute interest correspondences (and F) between 1&2 and 2&3

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select at random 6 correspondences and compute T

(b) Calculate the distance d for each putative match

(c) Compute the number of inliers consistent with T (d<t)

Choose T with most inliers

(iv) Optimal estimation: re-estimate T from all inliers by minimizing ML cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by computed T

Optionally iterate last two steps until convergence

Automatic computation of T

108 putative matches 18 outliers

88 inliers 95 final inliers

(26 samples)

(0.43)(0.23)

(0.19)

additional line matches

Matrix formulation for m-View

Consider one object point X and its m images: ixi=PiXi, i=1, …. ,m:

i.e. rank(M) < m+4 .

Laplace expansions

• The rank condition on M implies that all (m+4)*(m+4) minors of M are equal to 0.

• These can be written as sums of products of camera matrix parameters and image coordinates.

Matrix formulation

for non-trivially zero minors, one row has to be taken from each image (m).

4 additional rows left to choose

lk

jihgfedcba

000000000000000

ihgfedcba

jkl

only interesting if 2 or 3 rows from view

The three different types

1. Take the 2 remaining rows from one image block and the other two from another image block, gives the 2-view constraints.

2. Take the 2 remaining rows from one image block 1 from another and 1 from a third, gives the 3-view constraints.

3. Take 1 row from each of four different image blocks, gives the 4-view constraints.