Lecture 3

Displacement Models (contd)

Affine Mosaic

Projective Mosaic

Instantaneous Velocity Model

3-D Rigid Motion

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡+⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

′′′

Z

Y

X

TTT

ZYX

ZYX

11

1

γβγαβα

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡+⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−′−′−′

Z

Y

X

TTT

ZYX

ZZYYXX

00

0

γβγαβα

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡+

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡+

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

′′′

Z

Y

X

TTT

ZYX

ZYX

100010001

00

0

γβγαβα

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡+

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

ΩΩ−Ω−ΩΩΩ−

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

3

2

1

12

13

23

00

0

VVV

ZYX

ZYX

&

&

&

321

213

132

VXYZ

VZXY

VYZX

+Ω−Ω=

+Ω−Ω=

+Ω−Ω=

&

&

&

3-D Rigid Motion

321

213

132

VXYZ

VZXY

VYZX

+Ω−Ω=

+Ω−Ω=

+Ω−Ω=

&

&

&

VXX +×Ω=&

Cross Product

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

ΩΩΩ

=Ω

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

3

2

1

,ZYX

X

Orthographic Projection

(u,v) is optical flow213

132

321

213

132

VZxyvVyZxu

VXYZ

VZXY

VYZX

+Ω−Ω==+Ω−Ω==

+Ω−Ω=

+Ω−Ω=

+Ω−Ω=

&

&

&

&

&

XxYy

==

Plane+orthographic(Affine)

cYbXaZ ++=

14

133

122

322

21

211

Ω−=Ω−Ω=Ω−=

Ω−Ω=Ω=

Ω+=

caba

aVb

caba

aVb

bAxu +=

ZxVvyZVu

132

321

Ω−Ω+=Ω−Ω+=

yaxabvyaxabu

432

211

++=++=(1) Show this

Perspective Projection (arbitrary flow)

ZfYy

ZfXx

=

=

ZZy

ZYf

ZZfYYfZyv

ZZx

ZXf

ZZfXXfZ

xu

&&&&&

&&&&&

−=−

==

−=−

==

2

2

212331

2

2213

32

1

)(

)(

yf

xyf

yZVx

ZVfv

xf

xyf

yxZV

ZVfu

Ω−

Ω+−Ω+Ω−=

Ω+

Ω−Ω−−Ω+=

321

213

132

VXYZ

VZXY

VYZX

+Ω−Ω=

+Ω−Ω=

+Ω−Ω=

&

&

&

Plane+Perspective (pseudo perspective)

yacx

ab

aZ

cYbXaZ

−−=

++=11

u fVZ

VZ

x yf

xyf

x

v fVZ

xVZ

yf

xyf

y

= + − − − +

= − + − + −

( )

( )

12

33

1 2 2

21 3

3 2 1 2

Ω ΩΩ Ω

Ω ΩΩ Ω

254876

52

4321

yaxyayaxaav

xyaxayaxaau

++++=

++++=(2) Show this

3D Scenes

When the scene is neither planar not distant, the relation between two consecutive frames is described by the fundamental matrix.

What the heck is epipolar geometry?

Epipolar Geometry

Epipolar GeometryCoplanar Condition

(B – A) . A x B = 0

M'= R (M – T)

(M – T) T . T x M = 0R-1 = RT

(RT M')T .T x M = 0A x B = S.B therefore M' T R S M = 0

M'T E M = 0 where E =RS is the essential matrixa1

Slide 14

a1 The Essential Matrix is the mapping between points and epipolar linessubail, 10/9/2001

Vector Cross Product to Matrix-vector multiplication

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−=

=×

00

0

.

xy

xz

yz

BBBB

BBS

SABA

Camera Model Revisited: Perspective

ZfYy

ZfXx

ZYX

ff

Ch

=

=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1000000

Origin at the lensImage plane in front of the lens

MmPPMm=

=−1

Epipolar GeometryThe Fundamental Matrix is the Essential Matrix described inpixel coordinates.

m and m' are the points in pixel coordinates corresponding to m and m'

M = P-1 m and M'= P'-1 m'

m = f/Z . M and m'= f'/Z’ M'

M'T E M = 0 m'T F m = 0

(P'-1m')T E (P-1 m) = m'T P' -T E P-1 m

Epipolar Geometry

LeftLeftcameracamera

RightRightcameracamera

Left cameraLeft cameraimage planeimage plane

Right cameraRight cameraimage planeimage plane

3D World

Defined for two static camerasDefined for two static cameras

P

Cl Cr

T

xl xr

eler

Epipolar Geometry

EpipolarEpipolar line:line: set of world set of world points that project to same points that project to same point in left image, when point in left image, when projected to right image projected to right image forms a line.forms a line.

EpipolarEpipolar plane:plane: plane plane defined by the camera defined by the camera centers and world point.centers and world point.

EpipoleEpipole:: intersection of image intersection of image plane with line connecting plane with line connecting camera centerscamera centers

Essential MatrixP

Cl Cr

T

xl xr

eler

( )TPRP −= lr

( ) 0=×− Τll PTTP

CoplanarityCoplanarity constraint between vectors (constraint between vectors (PPll--TT), ), TT, , PPll..

0=Τlr ΕPP

essential matrixessential matrix

Pl Pr

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−

−

00

0

xy

xz

yz

TTTT

TT

0=Τlr PSRP

Fundamental Matrix

lll TR += PP

rrr TR += PP

( )TPRP −= lr Τ= lrRRR

rl TT Τ−= RT(A)(A)

(B)(B)

lll M Px =

rrr M Px =

0=Τlr ΕPP

( ) 01 =−Τ−Τllrr EMM xx

0=Τlr Fxx

fundamental matrixfundamental matrix

P

Cl Cr

T

xl xr

eler

PlPr

relative rotation

relative translation

Fundamental Matrix

Rank 2 matrix (due to S)Given a point in left camera x, epipolar line in right camera is: ur=Fx

0'' =⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛= ΤΤ xxxx

mhgfedcba

F

Fundamental Matrix

3x3 matrix with 9 componentsRank 2 matrix (due to S)7 degrees of freedomGiven a point in left camera x, epipolar line in right camera is: ur=Fx

0=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛= ΤΤ

rlrl

mhgfedcba

F xxxx

Fundamental Matrix

Longuet Higgins (1981)Hartley (1992)Faugeras (1992)Zhang (1995)

Fundamental matrix captures the relationship between the corresponding points in two views.

Fundamental Matrix

,011 333231

232221

131211

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡′′

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

i

iT

i

i

yx

fffffffff

yx

0)()()(

,01

333231232221131211

333231

232221

131211

=+′+′++′+′++′+′

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

+′+′+′+′+′+′

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

fyfxffyfxfyfyfxfx

fyfxffyfxffyfxf

yx

iiiii

i

i

iT

i

i

To solve the equation, the rank(M) must be 8.

0)()()( 333231232221131211 =+′+′++′+′++′+′ fyfxffyfxfyfyfxfx iiiii

0333231232221131211 =+′+′+′+′′+′++′+′ ffyfxfyfyxfxyfxfyxfxx iiiiiiii

0

1

11

1

222222222222

111111111111

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

′′′′′′

′′′′′′′′′′′′

= f

yxyyyxyxyxxx

yxyyyxyxyxxxyxyyyxyxyxxx

Mf

nnnnnnnnnnn

MMMMMMMMM

Fundamental Matrix

Computation of Fundamental Matrix

Normalized 8-point algorithm

Objective:Compute fundamental matrix F such that

AlgorithmNormalize the image by and

Find centroid of points in each image, determine the range, and normalize all points between 0 and 1

Linear solution determining the eigen vector corresponding to the smallest eigen value of A,

0=′ ii Fxx

ii Txx =ˆ ii T xx ′′=′ˆ

01ˆˆˆˆˆˆˆˆˆˆˆˆ...........................1ˆˆˆˆˆˆˆˆˆˆˆˆ

888888888888

111111111111

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

′′′′′′′

′′′′′′

= fyxyyyxyxyxxx

yxyyyxyxyxxxAf

Normalized 8-point algorithm Construct

Normalize

Constraint enforcement SVD decomposition

Rank enforcement

De-normalization:(3) Show this

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

987

654

321

ˆ

fffffffff

F

||ˆ||/ˆˆ FFF =

VUF ′⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

3

2

1

000000

ˆ

σσ

σ

)( 321 σσσ ≥≥

VUF ′⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

0000000

~2

1

σσ )0( 3 =σ

TFTF T ~′=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1000

0

yy

xx

dada

T

Robust Fundamental Matrix Estimation (by Zhang)

Uniformly divide the image into 8×8 grid.Randomly select 8 grid cells and pick one pair of corresponding points from each grid cell, then use Hartley’s 8-point algorithm to compute Fundamental Matrix Fi. For each Fi, compute the median of the squared residuals Ri.

Ri = mediank[d(p1k,Fip2k) + d(p2k,F’ip1k)]Select the best Fi according to Ri.Determine outliers if Rk>Th.Using the remaining points compute the fundamental Matrix F by weighted least square method.

Results

Results

Homework Due 1/23/2008

Show (1), (2) and (3).

Experiment with five different opencvroutines.