cs6670: computer vision - cornell university · creating a panorama •basic procedure –take a...

60
Lecture 7: Image Alignment and Panoramas CS6670: Computer Vision Noah Snavely What’s inside your fridge? http://www.cs.washington.edu/education/courses/cse590ss/01wi/

Upload: others

Post on 02-Aug-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Lecture 7: Image Alignment and Panoramas

CS6670: Computer VisionNoah Snavely

What’s inside your fridge?http://www.cs.washington.edu/education/courses/cse590ss/01wi/

Page 2: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Projection matrix

(t in book’s notation)

translationrotationprojection

intrinsics

Page 3: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Projection matrix

0

=

(in homogeneous image coordinates)

Page 4: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Questions?

Page 5: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image alignment

Full screen panoramas (cubic): http://www.panoramas.dk/

Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html

2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html

Page 6: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Why Mosaic?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

Slide from Brown & Lowe

Page 7: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Why Mosaic?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

Slide from Brown & Lowe

Page 8: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Why Mosaic?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

– Panoramic Mosaic = 360 x 180°

Slide from Brown & Lowe

Page 9: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Mosaics: stitching images together

Page 10: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Readings

• Szeliski:

– Chapter 6.1: Feature-based alignment

– Chapter 9: Panorama stitching

Page 11: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image alignment

Image taken from same viewpoint, just rotated.

Can we line them up?

Page 12: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image alignment

Why don’t these image line up exactly?

Page 13: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

What is the geometric relationship between these two images?

?

Page 14: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

What is the geometric relationship between these two images?

?

Page 15: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Is this an affine transformation?

Page 16: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Where do we go from here?

affine transformation

what happens when we change this row?

Page 17: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Projective Transformations aka Homographies aka Planar Perspective Maps

Called a homography(or planar perspective map)

projection of 3D plane can be explained by a (homogeneous) 2D transform

Page 18: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Homographies

What happens when the denominator is 0?

Page 19: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Homographies

• Example on board

Page 20: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image warping with homographies

image plane in front image plane below

black area

where no pixel

maps to

Page 21: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Homographies

Page 22: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Homographies

• Homographies …

– Affine transformations, and

– Projective warps

• Properties of projective transformations:

– Origin does not necessarily map to origin

– Lines map to lines

– Parallel lines do not necessarily remain parallel

– Ratios are not preserved

– Closed under composition

Page 23: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

2D image transformations

These transformations are a nested set of groups• Closed under composition and inverse is a member

Page 24: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Questions?

Page 25: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Creating a panorama

• Basic Procedure– Take a sequence of images from the same position

• Rotate the camera about its optical center

– Compute transformation between second image and first

– Transform the second image to overlap with the first

– Blend the two together to create a mosaic

– If there are more images, repeat

Page 26: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Geometric interpretation of mosaics

0

1

2

Page 27: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Geometric Interpretation of Mosaics

• If we capture all 360º of rays, we can create a 360º panorama• The basic operation is projecting an image from one plane to another• The projective transformation is scene-INDEPENDENT

• This depends on all the images having the same optical center

Image 1

Image 2

Optical Center

Page 28: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Projecting images onto a common plane

mosaic PP

Page 29: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image reprojection

• Basic question– How to relate two images from the same camera center?

• how to map a pixel from PP1 to PP2 PP2

PP1

Answer• Cast a ray through each pixel in PP1

• Draw the pixel where that ray intersects PP2

Page 30: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

What is the transformation?

Image 1

Image 2

Optical Center

How do we transform image 2 onto image 1’s projection plane?

image 1 image 2

3x3 homography

image coords(in image 2)

3D ray coords(in camera 2)

image coords(in image 2)

3D ray coords(in camera 1)

image coords(in image 2)

image coords(in image 1)

Page 31: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image alignment

Page 32: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Image alignment

Page 33: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Can we use homography to create a 360 panorama?

Page 34: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Panoramas

• What if you want a 360 field of view?

mosaic Projection Sphere

Page 35: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

– Map 3D point (X,Y,Z) onto sphere

Spherical projection

X

Y

Z

unit sphere

unwrapped sphere

• Convert to spherical coordinates

Spherical image

• Convert to spherical image coordinates

– s defines size of the final image

» often convenient to set s = camera focal length

Page 36: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

f = 200 (pixels)

Spherical reprojection

• Map image to spherical coordinates

– need to know the focal length

input f = 800f = 400

Page 37: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Aligning spherical images

• Suppose we rotate the camera by about the vertical axis– How does this change the spherical image?

Page 38: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Aligning spherical images

• Suppose we rotate the camera by about the vertical axis– How does this change the spherical image?

– Translation by

– This means that we can align spherical images by translation

Page 39: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Unwrapping a sphere

Credit: JHT’s Planetary Pixel Emporium

Page 41: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Different projections are possible

Page 42: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Questions?

• 3-minute break

Page 43: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Computing transformations

• Given a set of matches between images A and B

– How can we compute the transform T from A to B?

– Find transform T that best “agrees” with the matches

Page 44: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Computing transformations

?

Page 45: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Simple case: translations

How do we solve for?

Page 46: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Mean displacement =

Simple case: translations

Displacement of match i =

Page 47: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Another view

• System of linear equations– What are the knowns? Unknowns?

– How many unknowns? How many equations (per match)?

Page 48: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Another view

• Problem: more equations than unknowns– “Overdetermined” system of equations

– We will find the least squares solution

Page 49: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Least squares formulation

• For each point

• we define the residuals as

Page 50: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Least squares formulation

• Goal: minimize sum of squared residuals

• “Least squares” solution

• For translations, is equal to mean displacement

Page 51: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Least squares formulation

• Can also write as a matrix equation

2n x 2 2 x 1 2n x 1

Page 52: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Least squares

• Find t that minimizes

• To solve, form the normal equations

Page 53: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Affine transformations

• How many unknowns?

• How many equations per match?

• How many matches do we need?

Page 54: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Affine transformations

• Residuals:

• Cost function:

Page 55: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Affine transformations

• Matrix form

2n x 6 6 x 1 2n x 1

Page 56: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Homographies

To unwarp (rectify) an image• solve for homography H given p and p’

• solve equations of the form: wp’ = Hp

– linear in unknowns: w and coefficients of H

– H is defined up to an arbitrary scale factor

– how many points are necessary to solve for H?

pp’

Page 57: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Solving for homographies

Page 58: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Solving for homographies

Page 59: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Solving for homographies

Defines a least squares problem:

• Since is only defined up to scale, solve for unit vector

• Solution: = eigenvector of with smallest eigenvalue

• Works with 4 or more points

2n × 9 9 2n

Page 60: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center

Questions?