ee 7700 geometric transformations. bahadir k. gunturk2 geometric transformation translation scale...

EE 7700

Geometric Transformations

Bahadir K. Gunturk 2

Geometric Transformation

translation

scale

rotation & scale

2

1

43

21

b

b

y

x

aa

aa

y

x

Rigid flow

11001

'

'

243

121

y

x

baa

baa

y

x

cossin

sincos

s

s

0

0

Rotation matrix

Scale matrix

Bahadir K. Gunturk 3

Affine Flow

2

1

43

21

b

b

y

x

aa

aa

yy

xx

v

u

Bahadir K. Gunturk 4

Perspective Flow

1 2 3

7 8

'1

a x a y ax

a x a y

4 5 6

7 8

'1

a x a y ay

a x a y

'

'

u x x

v y y

Bahadir K. Gunturk 5

Bahadir K. Gunturk 6

Bahadir K. Gunturk 7

EE 7730

RANSAC: RANdom SAmple Consensus

Bahadir K. Gunturk 9

Outliers

Consider the least squares fit for optical flow:

1 2 1

3 4 2

( , )

( , )

u x y a x a y b

v x y a x a y b

1 1 1 1

1 1 1 2

1

3

4

2

1 0 0 0

0 0 0 1

1 0 0 0

0 0 0 1n n n

n nn

u x y a

v x y a

b

a

u ax y

bx yv

If some of the values are wrong, it will degrade the estimation.

Bahadir K. Gunturk 10

Outliers

It is best not to include outliers in the estimation.

Outliers

Best fit is degraded due to the outliers

Line Fitting Problem: Given (x1,y1), …, (xN,yN); find the line y=ax+b

1 1 1

1N N

y xa

by x

Bahadir K. Gunturk 11

Robust Estimation

Two step process: Classify data points as outliers or inliers Use inliers only to fit a model

Bahadir K. Gunturk 12

RANSAC

Repeat for k times: Randomly choose n points (the smallest number of points

required) from the data. Estimate the parameters using these points. For each data point other than these n points:

Check if the data point is within a threshold, t, distance of current model; if it is, the data point is consistent with current model.

The total number of data points that are consistent is model’s support.

If the support is larger than a predetermined number, d, then there is a good fit. Re-estimate the parameters using these inliers.

Choose the best fit with the smallest fitting error.

Bahadir K. Gunturk 13

RANSAC

Two samplesand their supports

for line-fitting

Bahadir K. Gunturk 14

Example

Find the perspective parameters

from Hartley & Zisserman

1 2 3

7 8

'1

a x a y ax

a x a y

4 5 6

7 8

'1

a x a y ay

a x a y

Bahadir K. Gunturk 15

Example

Apply corner detectors to both images

from Hartley & Zisserman

Bahadir K. Gunturk 16

Example

Find the best match within a search window.

from Hartley & Zisserman

Bahadir K. Gunturk 17

Example

Initial match results

188 matched features in left image pointing to locations of corresponding right image features

from Hartley & Zisserman

Bahadir K. Gunturk 18

Example

Inliers and outliers after RANSAC

89 outliers 99 inliersfrom Hartley & Zisserman

Bahadir K. Gunturk 19

Panoramic Image Reconstruction

Application: Mosaic construction

Find featuresMatch featuresFit parametric model

EE7730

Stereo Vision

Bahadir K. Gunturk 21

Pinhole Camera

Bahadir K. Gunturk 22

Review: Perspective Projection

' ' 'x y f

x y z

Bahadir K. Gunturk 23

Stereoscene pointscene point

optical centeroptical center

image planeimage plane

p p’

p p’

Bahadir K. Gunturk 24

Stereo Constraints

X1

Y1

Z1

O1

Image plane

Focal plane

M

p p’

Y2

X2

Z2O2

Epipolar Line

Epipole

Bahadir K. Gunturk 25

From Geometry to Algebra

O O’

P

pp’

All vectors shown lie on the same plane.

Bahadir K. Gunturk 26

From Geometry to Algebra

O O’

P

pp’

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v

Bahadir K. Gunturk 27

Matrix form of cross product

20

0

0

y z z y z

z x x z z x

x y y z y x

a b a b a a

a b a b a b a a b a b

a b a b a a

( ) 0

( ) 0

a a b

b a b

a×b=|a||b|sin(η)u a=axi+ayj+azk

b=bxi+byj+bzk

Bahadir K. Gunturk 28

The Essential Matrix

( , ,1)[ ( ')] 0 with

' ( ', ',1)

T

T

p u vp t Rp

p u v

' 0Tp Ep

' 0 with Tp Ep E t R Essential matrix

Bahadir K. Gunturk 29

A Simple Stereo System

Zw=0

LEFT CAMERA

Left image:reference

Right image:target

RIGHT CAMERA

Elevation Zw

disparity

Depth Z

baseline

Bahadir K. Gunturk 30

Parallel Cameras

ZT

fZxxTlr

OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T is the stereo baseline

rlxx

TfZ

rlxxd Disparity:

Bahadir K. Gunturk 31

Stereo View

Left View Right View

Disparity

Bahadir K. Gunturk 32

Correlation Approach

For Each point (xl, yl) in the left image, define a window centered at the point

(xl, yl)LEFT IMAGE

Bahadir K. Gunturk 33

Correlation Approach

… search its corresponding point within a search region in the right image

(xl, yl)RIGHT IMAGE

Bahadir K. Gunturk 34

Correlation Approach

… the disparity (dx, dy) is the displacement when the correlation is maximum

(xl, yl)dx(xr, yr)RIGHT IMAGE

Bahadir K. Gunturk 35

Stereo results

Ground truthScene

Data from University of Tsukuba

(Seitz)

Bahadir K. Gunturk 36

Results with window correlation

Estimated depth of field Ground truth

(Seitz)

Bahadir K. Gunturk 37

Results with better method

A state of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

Bahadir K. Gunturk 38

Applications

courtesy of Sportvision

First-down line

Bahadir K. Gunturk 39

ApplicationsVirtual advertising

courtesy of Princeton Video Image