multiview 3d video / outline advanced topics multimedia ...vca.ele.tue.nl/sites/default/files/5lsh0...

1

1

PdW-SZ-EB / 2017

Fac. EE SPS-VCA

Adv. Topics MMedia Video / 5LSH0 /

Mod 02 3D intro & multiview video

Advanced Topics Multimedia Video

(5LSH0), Module 02

3D Geometry, Camera Model & Projection Matrix

Egor Bondarev, Peter H.N. de With ( [email protected] )

Credit for some slides to James Hays, Derek Hoiem,

Alexei Efros, Steve Seitz, and David Forsyth

2

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Multiview 3D video / Outline

∗ Camera geometry

– Pinhole camera model

– Projective geometry

– Projection matrix,

– Intrinsic/extrinsic camera parameters

∗ Camera calibration

3

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



3D world projected on 2D photo views 4

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



3D world projected on 2D photo views

2

5

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



3D world projected on 2D photo views 6

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



But from another view…

7

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Mapping between image and world coordinates



• Vanishing points and lines

– Projection matrix

So, this hour we learn

8

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



How can we create something that

captures the scenery?

3

9

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Designing camera

– Idea 1: put a photon’s sensor in front of an object

– Do we get a good image?

Slide source: Seitz

10

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Designing camera - Pinhole camera

– Idea 2: add a barrier with opening to block off most

of the rays

– This reduces blurring

– The opening known as the apertureSlide source: Seitz

11

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Pinhole camera Model

Figure from Forsyth

f

f = focal length

c = center of the camera

c

12

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera obscura: the pre-camera

∗ Known from ancient

periods in China and

Greece (e.g. Mo-Ti,

China, 470BC to 390BC)

Camera obscura at Lisbon Castle

4

13

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera Obscura used for Tracing

Lens Based Camera Obscura, 1568

14

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



First Photograph

Oldest surviving photograph

Joseph Niepce, 1826

Photograph of the first photograph

Stored at UT Austin

15

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Pinhole vs Lens Camera

Figure from Forsyth

16

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




∗ Camera geometry






5

17

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry

18

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Point of observation

Figures © Stephen E. Palmer

Projection:

Dimensionality Reduction Machine (3D to 2D)

3D world 2D image

19

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projection can be tricky…

Slide source: Seitz20

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projection can be tricky…

Slide source: Seitz

6

21

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry

What is lost?

∗ Length

Which is closer?

Who is taller?

22

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry

What is lost?

∗ Length

23

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Length is not preserved

Figure by David Forsyth

B’

C’

A’

24

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective GeometryWhat is lost?

∗ Length

∗ Angles

Perpendicular?

Parallel?

7

25

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective GeometryWhat is preserved?

∗ Straight lines are still straight

26

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Vanishing points and lines

Two parallel rails intersect in the image plane at the vanishing point

27

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




Vanishingpoint

Vanishingline

Vanishingpoint

Vertical vanishingpoint

(at infinity)

Slide from Efros, Photo from Criminisi

28

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




oVanishing Point

oVanishing Point

Vanishing Line

8

29

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry:

world coordinates � image coordinates

Camera

Center

(tx, ty, tz)

=

Z

Y

X

P.

.

. f Z Y

=

v

up

.

Optical

Center

(u0, v0)

v

u

30

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry Intro• Imagine a projector that is projecting a 2D image onto a screen

• It's easy to identify the X and Y dimensions of the projected image

31

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry Intro• You can see the W dimension too

• The W dimension is the distance from the projector to the screen.

32

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective Geometry Intro• what does W do?

• imagine what happens with X and Y, if you decrease W

• the whole 2D image becomes smaller – X and Y decreases

• So, W affects the size (a.k.a. scale) of the image.

9

33

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Homogeneous coordinates

Converting to homogeneous coordinates

homogeneous image coordinates

• In projections on our image sensor, we do not know distances to objects

• So, we need to write scale-invariant coordinates (independent on distance)

• Use homogeneous coordinates

• By adding one more parameter

34

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




Converting to homogeneous coordinates

homogeneous image coordinates

homogeneous 3D scene

coordinates

Converting from homogeneous coordinates

35

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




Invariant to scaling

Point in Cartesian is ray in Homogeneous

=

⇒

=

w

y

wx

kw

ky

kwkx

kw

ky

kx

w

y

x

k

Homogeneous

Coordinates

Cartesian

Coordinates

36

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




∗ Camera geometry






10

37

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera Projection Matrix

38

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Basic Perspective Projection

∗ If you know P(xv,yv,zv) and d, what is P(xs,ys)?

– Where does a point in view space end up on the screen?

xv

yv

-zvd

P(xv,yv,zv)P(xs,ys)

39

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Basic Perspective Projection

∗ Similar triangles give:

v

vs

z

x

d

x=

v

vs

z

y

d

y=

yv

-zv

P(xv,yv,zv)P(xs,ys)

View Plane

d

40

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Simple Perspective Transformation

∗ Using homogeneous coordinates we can write:

– Our big advantage of homogeneous coordinates:

• Scale (distance) invariant

≡

dz

z

y

x

d

y

x

v

v

v

v

s

s

vs

d

PP

=

0100

0100

0010

0001

11

41

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Moving to Camera Matrix

∗ That was a simple geometric case

∗ With actual camera, we have a set of parameters,

which introduce function F

Fvs

d

PP

=

0100

0100

0010

0001

42

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Moving to Camera Matrix∗ Camera (imager sensor and lenses) may change representation of actual scene

– Varying focal length

43

PdW-SZ-EB / 2017

Fac. EE SPS-VCA





– Skew (non-rectangular pixels)

– Radial distortions

44

PdW-SZ-EB / 2017

Fac. EE SPS-VCA







– Principal Point offset

12

45

PdW-SZ-EB / 2017

Fac. EE SPS-VCA







– Principal Point offset

– Pixel size (aspect ratio)

46

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera geometry / Concept – (1)

∗ To understand the 3D structure of

objects/scene, a relation between

point coordinates in the 3D world

to pixels position is required.

∗ We have 2 coordinates systems:

image and world.

∗ Goal: map points in 3D space (the

world coordinate system) to the

image plane (the image

coordinates system).

47

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera geometry / Concept – (2)

∗ Link the world and image coordinates by a set of

parameters known as intrinsic and extrinsic parameters.

∗ Intrinsic parameters:

• focal length,

• width and height of the pixel on the sensor,

• position of the principal point (origin of the image coordinates

system).

∗ Extrinsic parameters:

• camera position,

• camera orientation.

48

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Pinhole Camera Projection Matrix

[ ] XtRKx =

x: Image Coordinates: (u,v,1), also Ps

K: Intrinsic Matrix (3x3)

R: Rotation (3x3)

t: Translation (3x1)

X: World Coordinates: (X,Y,Z,1), also Pv

∗ To take the above camera nuances in account:

vs

d

PP

=

0100

0100

0010

0001

13

49

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



[ ] X0IKx =

=

10100

000

000

1z

y

x

f

f

v

u

w

K

Slide Credit: Saverese

Intrinsic Assumptions

• Square pixels

• Optical center at (0,0)

• No skew

Projection Matrix with Assumptions

Known focal length f

50

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Remove assumption: known optical center

[ ] X0IKx =

=

10100

00

00

1

0

0

z

y

x

vf

uf

v

u

w


• Square pixels

• No skew

Known optical center u, v

51

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Remove assumption: square pixels

[ ] X0IKx =

=

10100

00

00

1

0

0

z

y

x

v

u

v

u

w β

α

Intrinsic Assumptions• No skew

Pixel aspect ratio is alfa / betta

52

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Remove assumption: non-skewed pixels

[ ] X0IKx =

=

10100

00

0

1

0

0

z

y

x

v

us

v

u

w β

α


• No

Note: different books use different notation for parameters

Skew is not 0, but s

14

53

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Oriented and Translated Camera

∗ Extrinsic parameters define orientation and location of

the camera in the world coordinate system

54

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Oriented and Translated Camera

[ ] XtIKx =

Extrinsic Assumptions• No rotation, camera axes aligned with World Coordinate System (WCS)

• Principal point is located in the center of WCS

55

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Allow camera translation

[ ] XtIKx =

=

1100

010

001

100

0

0

1

0

0

z

y

x

t

t

t

v

u

v

u

w

z

y

x

β

α

Extrinsic Assumptions• No rotation

Known camera offset from

origin of WCS – tx, ty, tz

56

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



3D Rotation of Points

Rotation around the coordinate axes, counter-clockwise:

−

=

−

=

−=

100

0cossin

0sincos

)(

cos0sin

010

sin0cos

)(

cossin0

sincos0

001

)(

γγ

γγ

γ

ββ

ββ

β

αα

ααα

z

y

x

R

R

R

p

p’

γ

y

z


15

57

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




Rotation around the coordinate axes


58

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




Rotation around the coordinate axes


59

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Allow also camera rotation

[ ] XtRKx =

=

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

v

us

v

u

w

z

y

x

β

α

60

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Degrees of freedom in Camera Matrix

[ ] XtRKx =

=

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

v

us

v

u

w

z

y

x

β

α

5 6

16

61

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera Matrix

=

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

v

us

v

u

w

z

y

x

β

α

We can finally map a 3D point to the image!

62

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



=

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

v

us

v

u

w

z

y

x

β

α

Unfortunately, not yet.

We need to find these parameters for each camera, first

63

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




∗ Camera geometry






64

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Projective camera / Summary – (2)

∗ Combining the camera matrix (intrinsic parameters) and

the rotation/translation matrix (extrinsic parameters) we

obtain the camera calibration matrix .

∗ Projection matrix 3x4 has 11 degrees of freedom (scaling

invariance).

intrinsic

parameters

extrinsic

parameterscamera calibration

matrix

world

coordinatespixel

coordinates

17

65

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Camera calibration

∗ Goal: estimating coefficients of the

camera calibration matrix.

– Once the camera calibration matrix

parameters are known, the camera is

“calibrated”.

∗ Simple calibration algorithm

– It is assumed that the world coordinates of

3D points are known with their

corresponding 2D pixel coordinates.

– Points are usually arranged in a special

pattern for easy calibration.

66

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Linear method for estimating matrix C – (1)

∗ World-point coordinates and image-pixel positions are

linked with the camera calibration matrix

∗ is the 3x4 projection matrix, can be written as

∗ Algorithm consists of two steps:

– 1. Compute matrix C with a set of known 3D positions and their

respective positions in the image.

– 2. Extrinsic and intrinsic parameters are estimated from C.

67

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




∗ Use world point coordinates and their corresponding

pixel coordinates in the image to determine .

– Each correspondence generates two equations

which can be written as

68

PdW-SZ-EB / 2017

Fac. EE SPS-VCA




∗ Stack equations into one equation system:

∗ The equation has 12 unknown parameters: at least 6

correspondence points are required.

∗ Typically, more points are used.

– Equation system gets over-constrained.

– Equation is then solved using a least squares minimization.

18

69

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Looking Back

=

1100

0

1 333231

232221

131211

0

0

z

y

x

trrr

trrr

trrr

v

us

v

u

w

z

y

x

β

α

homogeneous 3D scene coordinatesPinhole camera model

Camera projection matrix

70

PdW-SZ-EB / 2017

Fac. EE SPS-VCA



Questions

multiview 3d video / outline advanced topics multimedia ...vca.ele.tue.nl/sites/default/files/5lsh0...

Documents