Relations between image coordinates

Given coordinates in one image, and the tranformationBetween cameras, T = [R t], what are the image coordinatesIn the other camera’s image.

Definitions

Essential Matrix: Relating between image coordinates

camera coordinate systems, related by a rotation R and a translation T:

€

O x→

O' x→

' OO'→

Are Coplanar, so:

€

x'=R t

000 1

⎡

⎣ ⎢

⎤

⎦ ⎥x€

O' x→

'• O x→

× OO'→ ⎛

⎝ ⎜

⎞ ⎠ ⎟= 0

€

x'=R t

000 1

⎡

⎣ ⎢

⎤

⎦ ⎥x

€

x'•r t × Rx( ) = 0

€

O' x→

'• O x→

× OO'→ ⎛

⎝ ⎜

⎞ ⎠ ⎟= 0

€

x'• E x( ) = 0

x'E x = 0

E =

0 −tz ty

tz 0 −tx

−ty tx 0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥R

What does the Essential matrix do?

It represents the normal to the epipolar line in the other image

€

n = E x

xon epipolar line' ⇒ n ⋅ x ' = 0

n1x1 + n2x2 + n31 = 0

y = mx + b( )⇒ b = −n3, m = −n1

n2

The normal defines a line in image 2:

What if cameras are uncalibrated? Fundamental Matrix

Choose world coordinates as Camera 1. Then the extrinsic parameters for camera 2 are just R and tHowever, intrinsic parameters for both cameras are unknown.Let C1 and C2 denote the matrices of intrinsic parameters. Then the pixel coordinates measured are not appropriate for the Essential matrix. Correcting for this distortion creates a new matrix: the Fundamental Matrix.

C=

€

x'measured = C2x ' xmeasured = C1x

x '( )tE x = 0⇒ C2

−1x 'measured( )tE C1

−1xmeasured( ) = 0

x 'measured( )tF xmeasured = 0

F = C2−tEC1

−1

Computing the fundamental Matrix

Computing : I Number of Correspondences Given perfect image points (no noise) in general position. Each point correspondence generates one constraint on the fundamental matrix

Constraint for one point

Each constraint can be rewritten as a dot product.Stacking several of these results in:

Stereo Reconstruction

> 8 Point matches

C

Reconstruction Steps

Determining Extrinsic Camera Parameters

€

M1 = C1 Ir 0 [ ]

M2 = C2 Rr t [ ]

Why can we just use this as the external parameters for camera M1? Because we are only interested in the relative position of the two cameras.

• First we undo the Intrinsic camera distortions by defining newnormalized cameras

€

M1

norm = C1−1M1 and M2

norm = C2−1M2

Determining Extrinsic Camera Parameters• The normalized cameras contain unknown parameters

€

M1

norm = C1−1M1 ⇒ M

1

norm = Ir 0 [ ]

M2norm = C2

−1M2 ⇒ M2norm = R

r t [ ]

• However, those parameters can be extracted from the Fundamental matrix

€

F =C2−tEC1

−1

E = C2t FC1

€

E =0 −tz ty

tz 0 −tx

−ty tx 0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥R =

r t ×R

Extract t and R from the Essential Matrix

How do we recover t and R? Answer: SVD of E

€

E =USV t

S diagonal

U,V orthogonal and det() =1 (rotation)

€

R = UWV t or R = UW tV t

W =

0 −1 0

1 0 0

0 0 1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

rt = u3 or

r t = −u3

Reconstruction AmbiguitySo we have 4 possible combinations of translations and rotations giving 4 possibilities for M2

norm = [R | t]

1. M2norm = [UWtVt | t]

2. M2norm = [UWVt | t]

3. M2norm = [UWtVt | -t]

4. M2norm = [UWVt | -t]

Which one is right?

Both Cameras must be facing the same direction

Which one is right?

How do we backproject?

€

x'measured = C2x ' xmeasured = C1x

Knowing Ci allows us to determine the undistorted image points :x'= C2

−1x'measured x = C1−1xmeasured

Recalling the projection equations allows to relate the world point and the image points.x'= C2

−1x'measured x = C1−1xmeasured

z' x '= C2−1C2 M2

norm X zx = C1−1C1 I | 0[ ]X

z' x '= M2norm X zx = I | 0[ ]X

Backprojection to 3D

We now know x, x’, R, and tNeed X

Solving…

€

zx i = M norm X i

x i

y i

1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥=

m1t

m2t

m3t

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥X i ⇒

z ui = m1t ⋅X i

zv i = m2t ⋅X i

z = m3t ⋅X i

⎧

⎨ ⎪

⎩ ⎪

⇒(m3

t ⋅X i) ui = m1t ⋅X i

(m3t ⋅X i)v i = m2

t ⋅X i

⎧ ⎨ ⎩

⇒(m3

t ⋅X i) ui − m1t ⋅X i = 0

(m3t ⋅X i)v i − m2

t ⋅X i = 0

⎧ ⎨ ⎩

⇒ui(m3

t )− m1t

v i(m3t )− m2

t

⎡

⎣ ⎢

⎤

⎦ ⎥X i = 0

Solving…

€

u'i(2m3

t )−2m1t

v 'i(2m3

t )−2m2t

⎡

⎣ ⎢

⎤

⎦ ⎥X i = 0

Combining 1 & 2 :ui(m3

t )− m1t

v i(m3t )− m2

t

u'i(2m3

t )−2m1t

v 'i(2m3

t )−2m2t

⎡

⎣

⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥

X i = 0

AX i = 0It has a solvable form! Solve using minimum eigenvalue-Eigenvector approach (e.g. Xi = Null(A))

Where 2mti denotes the ith

row of the second camera’s normalized projection matrix.

Finishing up

What else can you do with these methods? Synthesize new views

60 deg!Image 1 Image 2

Avidan & Shashua, 1997

Faugeras and Robert, 1996

Undo Perspective Distortion (for a plane)Original images (left and right)

Transfer and superimposed images

The ``transfer'' image is the left image projectively warped so that points on the plane containing the Chinese text are mapped to their position in the right image.

•The ``superimpose'' image is a superposition of the transfer and right image. The planes exactly coincide. However, points off the plane (such as the mug) do not coincide.

*This is an example of planar projectively induced parallax. Lines joining corresponding points off the plane in the ``superimposed'' image intersect at the epipole.

Its all about point matches

Point match ambiguity in human perception

Traditional Solutions

• Try out lots of possible point matches.

• Apply constraints to weed out the bad ones.

Find matches and apply epipolar uniqueness constraint

Compute lots of possible matches

1) Compute “match strength”2) Find matches with highest strength

Optimization problem with many possible solutions

Example