december 12 th, 2001c. geyer/k. daniilidis grasp laboratory slide 1 structure and motion from...

December 12th, 2001 C. Geyer/K. DaniilidisGRASP Laboratory

Slide 1

Structure and Motion from Uncalibrated Catadioptric Views

Structure and Motion from Uncalibrated Catadioptric Views

Christopher Geyerand

Kostas Daniilidis

Christopher Geyerand

Kostas Daniilidis

of the

GRASP Laboratory,University of Pennsylvania

of the

GRASP Laboratory,University of Pennsylvania


Slide 2

The big picture:The big picture:

Two views obtained from a parabolic catadioptric camera1 are sufficient to perform a Euclidean reconstruction from point correspondences

NO PROJECTIVE AMBIGUITY!NO PROJECTIVE AMBIGUITY!Fine print: 1. with non-varying intrinsics; if intrinsics vary, three frames are necessary; it is assumed that aspect ratio is 1 and there is no skew.


Slide 3

AssumptionsAssumptions

• Parabolic mirror• Aspect ratio known• Skew known• Camera and mirror aligned• Orthographic projection


Slide 4

OutlineOutline

• Review– Related work– What’s a catadioptric sensor?– The projection model

• Theory– Define representation of image features: circle space– Define the catadioptric fundamental matrix

• Algorithm– Computation of the fundamental matrix– Reconstruction

• Experiment• Conclusion & future work


Slide 5

Related workRelated work

• Nayar, Baker– theory of catadioptric image formation [ICCV

`98]

• Geyer, Daniilidis– theory of projective geometry induced in

catadioptric images[see December `01 IJCV, ECCV `00]

• Svoboda, Pajdla– description of epipolar geometry

[ECCV `98]

• Kang– non-linear self-calibration [CVPR `00]

generalcata-

dioptric

2,3 n-view


Slide 6

What’s a catadioptric camera?What’s a catadioptric camera?

Catadioptric camera combines a mirror and lens. We investigate cameras with a parabolic mirror and orthographically projecting lens

Mirror

CCD

Lens

sample image


Slide 8

Projection modelProjection model (Review)

F

P

s

R

Q

• The image of P is the orthographic projection of the intersection of the line through F and P with the parabola.


Slide 9

JuvN=ikjjjjjjjjcx + 2 fx

- z+"

x2+y2+z2

cy+ 2 fy

- z+"

x2+y2+z2

y{zzzzzzzz

Projection modelProjection model

• The formula for this projection is:

(where (x,y,z) is the space point and (u,v) is the image point)

Uh-oh! It’s not linear! How do we apply SFM algorithms?

(Review)


Slide 10


• But wait! Neither is the perspective projection formula:JuvN=ikjjjjjcx + fx

z

cy+ fyz

y{zzzzzThat’s why we use homo-geneous coordinates invented by Möbius and Feuerbach in 1827.

FeuerbachFeuerbachMöbiusMöbius

(Review)


Slide 11


• Using homogeneous coordinates we can write:

Embedding image points in a higher dimensional space linearizes the projection equation

ikjjjjjuv1y{zzzzz=l P

ikjjjjjjjjxyz1

y{zzzzzzzz

U=l PX

(Review)


Slide 12


Can we perform a similar trick for Can we perform a similar trick for the parabolic projection?the parabolic projection?

Yes, represent points in a higher Yes, represent points in a higher dimensional space, lying on a dimensional space, lying on a paraboloid.paraboloid.

But first we review properties of But first we review properties of line images.line images.

(Review)


Slide 14

A circle! But what kind?A circle! But what kind?Must have same dof as line imagesMust have same dof as line images


Slide 15

One thatintersects..One thatintersects..


Slide 16

reduces 3 dof to 2 dofreduces 3 dof to 2 dof

One that the calibratingintersects.. conic antipodallyOne that the calibratingintersects.. conic antipodally

Calibrating conic

we call ω΄

Calibrating conic

we call ω΄


Slide 17

• Equation of calibrating conic:

• Equation of image of absolute conic

• Same radii; ratio of radii = i; both encode the intrinsics


Hcx - uL2+Hcy - vL2 =H2 fL2Hcx - uL2+Hcy - vL2 =Hi 2 fL2


Slide 18

Circle SpaceCircle Space

Is there a cleverrepresentationof the space ofcircles?

Yes! And it usesthe paraboloid!

See Dan Pedoe’s Geometry

(Theory)

http://www.amazon.com/exec/obidos/tg/stores/detail/-/books/0486658120/reader/2/107-1080608-1246135


Slide 19

Choose a point PChoose a point P

NOT NECESSARILY THE SAME PARABOLOID


Slide 20

Find the cone with vertex Ptangent to the paraboloid

Find the cone with vertex Ptangent to the paraboloid


Slide 21

Find the plane through the inters-ection of the cone and the paraboloid

Find the plane through the inters-ection of the cone and the paraboloid


Slide 22

Orthographically project the intersection:a circle centered at the shadow of P

Orthographically project the intersection:a circle centered at the shadow of P


Slide 23

What is the result of varying P?

What is the result of varying P?


Slide 24

SummarySummary


Slide 25

Lifting of image pointsLifting of image points

p

p (lifting)~


Slide 26

In other words…In other words…

• We add a fourth coordinate:

p=ikjjjjjuv1y{zzzzzpŽ=

ikjjjjjjjjj

uv

u2+v2+c1

y{zzzzzzzzz


Slide 27

SummarySummary

• In this “circle space” we can represent line images and image points

• How do we represent circles intersecting the calibrating conic antipodally?

(Theory)


Slide 28

Collinear point representationscorrespond to coaxal circles

Collinear point representationscorrespond to coaxal circles


Slide 29

Coplanar point representations map to circles intersecting a circle antipodallyCoplanar point representations map to circles intersecting a circle antipodally


Slide 30

How does the plane changewhen the image plane translates?

How does the plane changewhen the image plane translates?


Slide 31

Circle spaceCircle space

• What is the significance of this plane?

• As it turns out it is the polar plane of the point representation

(Theory)


Slide 32

Absolute conicAbsolute conic (Theory)

Image of the absolute conic

Calibratingconic

Plane of antipodallyintersecting circles


Slide 33

Projections of space pointsProjections of space points

spacepoint

it’simage


Slide 34

Projections of space pointsProjections of space points

image of the absolute conic

spacepoint

it’simage


Slide 35

But uncalibratedBut uncalibrated


Slide 36

Transformations of Circle SpaceTransformations of Circle Space

Lineartransfor-mation


Slide 37

Step to linearizationStep to linearization (Theory)


Slide 38

LinearizationLinearization

• There is a 3x4 K such that

(the perspective projection)

in particular

KpŽ=

ikjjjjjjjj

xzyz1

y{zzzzzzzz

K =

ikjjjjjjjjj

1 0 0 - cx0 1 0 - cy

- cx2f

-cy2f

14f

1- 16f2+4cx2+4cy2

16f

y{zzzzzzzzz

(Theory)


Slide 39

LinearizationLinearization

The image of the absoluteconic is in the kernel of K

KwŽ=0

(Theory)


Slide 40

Two viewsTwo views

P

p

q


Slide 41

On to the epipolar constraintOn to the epipolar constraint

• If p and q are projections of the same point in two catadioptric images associated with K1 and K2 then

and

are the perspective projections.

q¢=K2 qŽ p¢=K1 pŽ


Slide 42

The epipolar constraint The epipolar constraint

The perspective projections satisfy

for some essential matrix E. Therefore

this is the epipolar constraint for parabolic catadioptric cameras.

p¢T Eq¢=0

pŽT K1T EK2 qŽ=0


Slide 43

Fundamental matrixFundamental matrix

• The 4x4 matrix has rank 2 and since

it satisfies

F=K1T EK2

FwŽ2=0 and FT wŽ1 =0

K2wŽ2=0 and K1wŽ1 =0

(Theory)


Slide 45

Reconstruction algorithmReconstruction algorithm

Non-varying intrinsics:1. Compute F from >= 16 point matches


Slide 46


Non-varying intrinsics:1. Compute F from > 15 point matches2. Ensure F is rank 2: project to rank 2 manifold (using SVD)


Slide 47


Non-varying intrinsics:1. Compute F from > 15 point matches2. Ensure F is rank 2: project to rank 2 manifold (using SVD)3. Find ω by intersecting left and right nullspaces of F

~


Slide 48


Non-varying intrinsics:1. Compute F from > 15 point matches2. Ensure F is rank 2: project to rank 2 manifold (using SVD)3. Find ω by intersecting left and right nullspaces of F4. Compute E = K-TFK-1 (K found from ω)

~

~


Slide 49


Non-varying intrinsics:1. Compute F from > 15 point matches2. Ensure F is rank 2: project to rank 2 manifold (using SVD)3. Find ω by intersecting left and right nullspaces of F4. Compute E = K-TFK-1 (K found from ω)5. Project E to manifold of essential matrices (average its singular values)

~

~


Slide 50


Non-varying intrinsics:1. Compute F from > 15 point matches2. Ensure F is rank 2: project to rank 2 manifold (using SVD)3. Find ω by intersecting left and right nullspaces of F4. Compute E = K-TFK-1 (K found from ω)5. Project E to manifold of essential matrices (average its singular values)6. Do Euclidean reconstruction with E

~

~


Slide 51

Varying intrinsicsVarying intrinsics

Three views w/absolute conics ω1, ω2, ω3 compute fundamental matrices between pairs of views: F12, F23, F31

Intersect nullspaces of F12T and F31

to find ω1

Similarly for ω2 and ω3

~ ~ ~

~ ~

~


Slide 52

ExperimentExperiment

First viewFirst view


Slide 53


Second view same intrinsicsSecond view same intrinsics


Slide 54


Generate point correspondencesmanually or automatically

Generate point correspondencesmanually or automatically


Slide 56

catadioptric fundamental matrix

ConclusionConclusion

• We have defined a represent-ation of image features in a parabolic catadioptric image

• We have shown that the catadioptric fundamental matrixcatadioptric fundamental matrix yields a bilinear constraint on lifted image points

• Its left and right nullspaces encode the image of the absolute conic.


Slide 57

Conclusion (cont’d)Conclusion (cont’d)

• Non-varying intrinsics: in two views create Euclidean reconstruction

• Varying intrinsics: three views are sufficient to compute Euclidean structure

• Though aspect ratio 1 and skew 0 is assumed, this still beats the perspective case with same assumption where there is a projective ambiguity


Slide 58

Future WorkFuture Work

• Necessary and sufficient conditions on matrix F

• Critical motions• Ambiguous surfaces• Relaxing aspect ratio and skew

assumption• Hyperbolic mirrors• Thanks!

december 12 th, 2001c. geyer/k. daniilidis grasp laboratory slide 1 structure and motion from...

Documents

catadioptric projection

catadioptric images

image plane

parabolic projection

catadioptric sensor

equation of image

image pointuh

space point