projective structure from motion

56
Computer Vision Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

Upload: randall-rosario

Post on 03-Jan-2016

30 views

Category:

Documents


1 download

DESCRIPTION

Projective structure from motion. Marc Pollefeys COMP 256. Some slides and illustrations from J. Ponce, …. Last class: Affine camera. The affine projection equations are. how to find the origin? or for that matter a 3D reference point?. affine projection preserves center of gravity. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Projective structure from motion

ComputerVision

Projective structure from motion

Marc PollefeysCOMP 256

Some slides and illustrations from J. Ponce, …

Page 2: Projective structure from motion

ComputerVision Last class: Affine camera

• The affine projection equations are

1

~

~

j

j

j

yi

xi

ij

ij

ZYX

P

Py

x

how to find the origin? or for that matter a 3D reference point?

affine projection preserves center of gravity

i

ijijij xxx~ i

ijijij yyy~

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

Page 3: Projective structure from motion

ComputerVision Orthographic factorization

• The orthographic projection equations are

• where

njmijiij ,...,1,,...,1,Xx P

• All equations can be collected for all i and j

• where

n

mmnmm

n

n

X,...,X,X , ,

xxx

xxxxxx

212

1

21

22221

11211

X

P

P

P

Px

XPx

X ~

~x

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and X are resp. 2mx3 and 3xn matrices and

therefore the rank of x is at most 3

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

Page 4: Projective structure from motion

ComputerVision Orthographic factorization

• Factorize m through singular value decomposition

• An affine reconstruction is obtained as follows

TVUx

T~,

~VXUP

n

mmnmm

n

n

X,...,X,X

xxx

xxxxxx

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

Page 5: Projective structure from motion

ComputerVision

Aug 26/28 - Introduction

Sep 2/4 Cameras Radiometry

Sep 9/11 Sources & Shadows Color

Sep 16/18 Linear filters & edges

(hurricane Isabel)

Sep 23/25 Pyramids & Texture Multi-View Geometry

Sep30/Oct2 Stereo Project proposals

Oct 7/9 Tracking (Welch) Optical flow

Oct 14/16 - -

Oct 21/23 Silhouettes/carving (Fall break)

Oct 28/30 - Structure from motion

Nov 4/6 Project update Proj. SfM

Nov 11/13 Camera calibration Segmentation

Nov 18/20 Fitting Prob. segm.&fit.

Nov 25/27 Matching templates (Thanksgiving)

Dec 2/4 Matching relations Range data

Dec ? Final project

Tentative class schedule

Page 6: Projective structure from motion

ComputerVision Further Factorization work

Factorization with uncertainty

Factorization for indep. moving objects

Factorization for dynamic objects

Perspective factorization

Factorization with outliers and missing pts.

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

(Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))

(Sturm & Triggs 1996, …)

Page 7: Projective structure from motion

ComputerVision

Structure from motion of multiple moving objects

one object:

multiple objects:

Page 8: Projective structure from motion

ComputerVision SfM of multiple moving objects

Page 9: Projective structure from motion

ComputerVision

Structure from motion of deforming objects

Extend factorization approaches to deal with dynamic shapes

(Bregler et al ’00; Brand ‘01)

Page 10: Projective structure from motion

ComputerVision Representing dynamic shapes

represent dynamic shape as varying linear combination of basis shapes

k

kk (t)ScS(t)

(fig. M.Brand)

Page 11: Projective structure from motion

ComputerVision Projecting dynamic shapes

PtScR k

kk

(figs. M.Brand)Rewrite:

Page 12: Projective structure from motion

ComputerVision Dynamic image sequences

One image:

Multiple images(figs. M.Brand)

Page 13: Projective structure from motion

ComputerVision Dynamic SfM factorization?

Problem: find J so that M has proper structure

Page 14: Projective structure from motion

ComputerVision Dynamic SfM factorization

(Bregler et al ’00)

Assumption: SVD preserves order and orientation of basis shape components

Page 15: Projective structure from motion

ComputerVision Results

(Bregler et al ’00)

Page 16: Projective structure from motion

ComputerVision Dynamic SfM factorization

(Brand ’01)

constraints to be satisfied for M

constraints to be satisfied for M, use to compute J

hard!

(different methods are possible, not so simple and also not optimal)

Page 17: Projective structure from motion

ComputerVision Non-rigid 3D subspace flow

• Same is also possible using optical flow in stead of features, also takes uncertainty into account

(Brand ’01)

Page 18: Projective structure from motion

ComputerVision Results

(Brand ’01)

Page 19: Projective structure from motion

ComputerVision

(Brand ’01)

Results

Page 20: Projective structure from motion

ComputerVision Results

(Bregler et al ’01)

Page 21: Projective structure from motion

ComputerVision

PROJECTIVE STRUCTURE FROM MOTION

Reading: Chapter 13.

• The Projective Structure from Motion Problem• Elements of Projective Geometry• Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices• Projective Structure and Motion from Multiple Images

Page 22: Projective structure from motion

ComputerVision The Projective Structure-from-Motion Problem

Given m perspective images of n fixed points P we can write

Problem: estimate the m 3x4 matrices M andthe n positions P from the mn correspondences p .

i

j ij

2mn equations in 11m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

Page 23: Projective structure from motion

ComputerVision

The Projective Ambiguity of Projective SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and Q is an arbitrary non-singular 4x4 matrix.

When the intrinsic and extrinsic parameters are unknown

Q is a projective transformation.

Page 24: Projective structure from motion

ComputerVision Projective Spaces: (Semi-Formal) Definition

Page 25: Projective structure from motion

ComputerVision

A Model of P( R )3

Page 26: Projective structure from motion

ComputerVision Projective Subspaces and Projective Coordinates

Page 27: Projective structure from motion

ComputerVision

Projective Subspaces and Projective Coordinates

Projectivecoordinates

P

Page 28: Projective structure from motion

ComputerVision

Projective Subspaces

Given a choice of coordinate frame

Line: Plane:

Page 29: Projective structure from motion

ComputerVision

Affine and Projective Spaces

Page 30: Projective structure from motion

ComputerVision

Affine and Projective Coordinates

Page 31: Projective structure from motion

ComputerVision

Cross-Ratios

Collinear points

Pencil of coplanar lines Pencil of planes

{A,B;C,D}=sin(+)sin(+)

sin(++)sin

Page 32: Projective structure from motion

ComputerVision

Cross-Ratios and Projective Coordinates

Along a line equipped with the basis

In a plane equipped with the basis

In 3-space equipped with the basis

*

*

Page 33: Projective structure from motion

ComputerVision

Projective Transformations

Bijective linear map:

Projective transformation:( = homography )

Projective transformations map projective subspaces ontoprojective subspaces and preserve projective coordinates.

Projective transformations map lines onto lines andpreserve cross-ratios.

Page 34: Projective structure from motion

ComputerVision

Perspective Projections induce projective transformationsbetween planes.

Page 35: Projective structure from motion

ComputerVision

Geometric SceneReconstruction

Idea: use (A,O”,O’,B,C)as a projective basis.

Page 36: Projective structure from motion

ComputerVision

Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. ReportUIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

Page 37: Projective structure from motion

ComputerVision

Motion estimation from fundamental matrices

Q

Facts:

b’ can be found using LLS.

Once M and M’ are known,P can be computed with LLS.

Page 38: Projective structure from motion

ComputerVision

Projective Structure from Motion and Factorization

Factorization??

Algorithm (Sturm and Triggs, 1996) • Guess the depths;• Factorize D;• Iterate.

Does it converge? (Mahamud and Hebert, 2000)

Page 39: Projective structure from motion

ComputerVision

Bundle adjustment - refining structure and motion

• Minimize reprojection error

– Maximum Likelyhood Estimation (if error zero-mean Gaussian

noise)– Huge problem but can be solved

efficiently (exploit sparseness)

m

i

n

jjiijD

ji 1 1

2

X̂,P̂X̂P̂,xmin

Page 40: Projective structure from motion

ComputerVision

• Developed in photogrammetry in 50´s

Bundle adjustment

Page 41: Projective structure from motion

ComputerVision Non-linear least squares

• Linear approximation of residual• allows quadratic approximation of sum- of-squares

J0e

0T1-T

0TT

JJJ

0J2J2J

e

e

JJ 00 ee T

J)λdiag(JJJN' TT

N

(extra term = descent term)

Minimization corresponds to finding zeros of derivative

Levenberg-Marquardt: extra term to deal with singular N(decrease/increase if success/failure to descent)

Page 42: Projective structure from motion

ComputerVision Bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

• Jacobian of has sparse block structure– cameras independent of other cameras,– points independent of other points

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

i

n

jjiijD

1 1

2M̂P̂,m

Needed for non-linear minimization

Page 43: Projective structure from motion

ComputerVision Bundle adjustment

Eliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

Page 44: Projective structure from motion

ComputerVision Sequential SfM

• Initialize motion from two images• Initialize structure• For each additional view

– Determine pose of camera– Refine and extend structure

• Refine structure and motion

Page 45: Projective structure from motion

ComputerVision

Initial projective camera motion

• Choose P and P´compatible with F

Reconstruction up to projective ambiguity

(reference plane;arbitrary)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Same for more views?

different projective basis

Page 46: Projective structure from motion

ComputerVision Initializing projective structure

• Reconstruct matches in projective frame by minimizing the reprojection error

Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Page 47: Projective structure from motion

ComputerVision Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches

• Compute pose from (RANSAC,6pts)

F

X

x

Inliers: inx,X x X DD iii P

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Page 48: Projective structure from motion

ComputerVision

• Refining structure

• Extending structure2-view triangulation

X~

P

1

3

(Iterative linear)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Refining and extending structure

Page 49: Projective structure from motion

ComputerVision Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!

Page 50: Projective structure from motion

ComputerVision

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

Page 51: Projective structure from motion

ComputerVision Metric structure and motion

Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)

(solution possible based on model selection)

use self-calibration (see next class)

Page 52: Projective structure from motion

ComputerVision Dealing with dominant planes

Page 53: Projective structure from motion

ComputerVision

PPPgric

HHgric

Page 54: Projective structure from motion

ComputerVision Farmhouse 3D models

(note: reconstruction much larger than camera field-of-view)

Page 55: Projective structure from motion

ComputerVision Application: video augmentation

Page 56: Projective structure from motion

ComputerVision

Next class: Camera calibration (and self-calibration)

Reading: Chapter 2 and 3