computer vision on rolling shutter cameras part ii ... · computer vision on rolling shutter...

35
Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical Engineering Linköping University Per-Erik Forssén , Erik Ringaby, Johan Hedborg

Upload: trinhdang

Post on 15-Jul-2018

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Computer Vision on Rolling Shutter Cameras

PART II: Rolling Shutter Geometry

Computer Vision LaboratoryDept. of Electrical Engineering

Linköping University

Per-Erik Forssén, Erik Ringaby, Johan Hedborg

Page 2: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Tutorial overview

2

1:30-2:00pm Introduction Per-Erik

2:00-2:15pm Rolling Shutter Geometry Per-Erik

2:15-3:00pm Rectification and Stabilisation Erik

3:00-3:30pm Break

3:30-3:45pm Rolling Shutter and the Kinect Erik

3:45-4:30pm Structure from Motion Johan

Page 3: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Projective GeometryTextbook material

3

Hartley & ZissermanMultiple View Geometry

2nd ed 2004

FaugerasThree-Dimensional Computer Vision

1993

Page 4: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraPin-hole camera model

4

Page 5: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraPin-hole camera model

A brightly illuminated scene will be projected onto a wall opposite of the pin-hole.

The image is rotated 180°.

5

Page 6: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole camera

From similar triangles we get:

6

Page 7: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole camera

More generally we write:

f-focal length, s-skew, a-aspect ratio,c=[cx cy]-projection of optical centre

7

Page 8: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole camera

Motivation:

f-focal length, s-skew, a-aspect ratio,c=[cx cy]-projection of optical centre

8

Page 9: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraProjection of 3D points Xc in the camera coordinate system:

In order to relate several camera poses, we need to use a common world coordinate system (WCS) :

d is a translation of the origin, and R is a rotation

9

Page 10: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraWe can simplify this to a single projection operation on the 3D points Xw:

where P is a 3x4 matrix, and

10

Page 11: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraWe can simplify this to a single projection operation on the 3D points Xw:

where P is a 3x4 matrix, and

This matrix P has the explicit form:

11

Page 12: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

The pin-hole cameraWe can simplify this to a single projection operation on the 3D points Xw:

where P is a 3x4 matrix, and

This matrix P has the explicit form:

12

Page 13: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Rolling shutter modelNow recall the rolling shutter readout:

13

Page 14: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Rolling shutter modelFor a moving camera, projection in frame k becomes:

14

Page 15: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Rolling shutter modelFor a moving camera, projection in frame k becomes:

In the global shutter case, we have one pose (Rk,dk) per frame

In the rolling shutter case, we instead get one pose (R(x2),d(x2)) per image row, x2

15

Page 16: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Time coordinateWhen interpolating the camera pose based on the image row, x2 it is convenient to express time in number of rows, instead of seconds.

Recall that the frame period T, is divided into readout time tr and inter-frame delay td.

tr corresponds to number of image rows Nr, and td corresponds to number of blank-rows Nb.

16

1/f = T = tr + td

Nb = Nrtd/tr = Nr(tr/f − 1)

Page 17: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

TriangulationTriangulation is the process of estimating a 3D point X from two projections x1 and x2.

17

Page 18: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

TriangulationFor the two points, we have:

18

x2 ∼ P2Xx1 ∼ P1X

Page 19: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

TriangulationFor the two points, we have:

Triangulation is typically solved by so calledoptimal triangulation [Hartley&Zisserman’04]

The point X is sought, for which the squared re-projection error in both images is minimized.

There exists a closed form solution, that is found by solving a 3rd degree polynomial.

19

x2 ∼ P2Xx1 ∼ P1X

X∗ = arg minX

�d2(x1,P1X) + d2(x2,P2X)

Page 20: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Rolling shutter triangulationIf we generalize triangulation to a (moving) rolling shutter rig, we get:

This has an unique solution, if, and only if t1=t2, which happens if the point is projected in both images at the same time.

That is, when the two points have the same y-coordinate. (Very rare!)

20

X∗ = arg minX

�d2(x1,P1X(t1)) + d2(x2,P2X(t2))

Page 21: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Rolling shutter SfMSuggestion from [Ait-Aider & Berry, ICCV’09]: Solve for triangulation of all points, and the object motion at the same time (structure-from-motion SfM).

The projection constraints for a correspondence now assumes the form:

The set of all such triangulation constraints can uniquely define the solution, if we assume a parametric form for R(t) and d(t). The most simple one is a linear motion(6dof).

21

x1 ∼ K [R(t1)|d(t1)]Xx2 ∼ KR2 [R(t2)|d2 + d(t2)]X

Page 22: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

DegeneracyThe full optimization problem now looks like this:

where

This problem has a unique solution when the motion is different from a pure translation along the x-axis.

That is, when for most points.(Otherwise we never observe point motion.)

22

P1 = K [R(t1)|d(t1)]P2 = KR2 [R(t2)|d2 + d(t2)]

t1 �= t2

{X∗k}

K1 ,R,d = arg min

{Xk}K1 ,R,d

�K�

k=1

d2(x1,k,P1Xk) + d2(x2,k,P2Xk)

Page 23: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

DegeneracyFor a translation parallel to the line between the optical centra, there is an equivalent stationary structure

23

Illustration by Ait-Aider and Berry

Page 24: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

DegeneracyFor a translation parallel to the line between the optical centra, there is an equivalent stationary structure

24

Illustration by Ait-Aider and Berry

⟵Translating structure

Page 25: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

DegeneracyFor a translation parallel to the line between the optical centra, there is an equivalent stationary structure

25

Illustration by Ait-Aider and Berry

⟵Translating structure

⟵Equivalent stationarystructure

Page 26: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

DegeneracyStructure and motion (SfM) from two frames is unstable for sideways motion, when both cameras have the same readout speed (or are the same).

If one of the cameras has a global shutter, both structure and motion can be obtained [Ait-Aider&Berry ICCV’09]

If multiple frames are used, rolling shutter structure from motion (SfM) becomes stable again [Hedborg et al. CVPR’12].

26

Page 27: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasPushbroom [Gupta and Hartley PAMI’97]Single line camera that is moving.

27

Page 28: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasPushbroom [Gupta and Hartley PAMI’97]Single line camera that is moving.

Work on scanning LIDARs, e.g. archeological reconstruction work by [Ikeuchi et al.]

[Bosse ICRA’09] e.g. rotating, and bouncing LIDARs.

28

Images from lab of Katsushi Ikeuchi

Page 29: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasCrossed slits [Zomet et al. PAMI’03]

projection rays from 3D points to the image plane are defined as intersections of two ”slit planes”.

29

Page 30: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasCrossed slits [Zomet et al. PAMI’03]

30

pin-hole camera renderinghorizontal/vertical slits

Z rotation of vertical slit X rotation of vertical slit

Page 31: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasCrossed slits [Zomet et al. PAMI’03]

31

standard push-broom mosaicsweeping push-broom mosaic

Page 32: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasCrossed slits [Zomet et al. PAMI’03]

32

standard push-broom mosaicsweeping push-broom mosaic

rendered image pin-hole camera image

Page 33: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

Related scanning camerasCrossed slits [Zomet et al. PAMI’03]

[Geyer et al. OMNIVIS’05] demonstrate that a rolling shutter camera is equivalent to a crossed-slits camera for a pure translation parallel to the image plane. (but not in general)

A crossed-slits camera can thus be seen as a special case of the rolling shutter camera.

33

Page 34: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

SummaryIn rolling shutter geometry, the camera trajectory is best modelled as continuous

There is an ambiguity between structure and sideways motion for two-frame rolling-shutter geometry.

Other types of scanning geometries (push-broom and moving LIDAR) do not have the temporal regularity of rolling shutter geometry.

34

Page 35: Computer Vision on Rolling Shutter Cameras PART II ... · Computer Vision on Rolling Shutter Cameras PART II: Rolling Shutter Geometry Computer Vision Laboratory Dept. of Electrical

ReferencesZomet, Feldman, Peleg, Weinshall, ”Mosaicing New Views: The Crossed-Slits Projection”, PAMI’03

Hartley, Zisserman, ”Multiple View Geometry for Computer Vision”

Geyer, Meingast, Sastry, ”Geometric Models of Rolling Shutter Cameras”, OMIVIS’05

Gupta, Hartley, ”Linear Pushbroom Cameras”, PAMI’97

Ait-Aider, Berry, ”Structure and Kinematics Triangulation from a Rolling Shutter Stereo Rig”, ICCV’09

Bosse, Zlot ”Continuous 3D Scan-Matching with a Spinning 2D Laser”, ICRA’09

Hedborg, Forssén, Felsberg, Ringaby, ”Rolling Shutter Bundle Adjustment”, CVPR’12

35