cs6670: computer vision - cornell university · creating a panorama •basic procedure –take a...
TRANSCRIPT
![Page 1: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/1.jpg)
Lecture 7: Image Alignment and Panoramas
CS6670: Computer VisionNoah Snavely
What’s inside your fridge?http://www.cs.washington.edu/education/courses/cse590ss/01wi/
![Page 2: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/2.jpg)
Projection matrix
(t in book’s notation)
translationrotationprojection
intrinsics
![Page 3: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/3.jpg)
Projection matrix
0
=
(in homogeneous image coordinates)
![Page 4: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/4.jpg)
Questions?
![Page 5: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/5.jpg)
Image alignment
Full screen panoramas (cubic): http://www.panoramas.dk/
Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html
2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html
![Page 6: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/6.jpg)
Why Mosaic?
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
Slide from Brown & Lowe
![Page 7: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/7.jpg)
Why Mosaic?
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV = 200 x 135°
Slide from Brown & Lowe
![Page 8: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/8.jpg)
Why Mosaic?
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV = 200 x 135°
– Panoramic Mosaic = 360 x 180°
Slide from Brown & Lowe
![Page 9: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/9.jpg)
Mosaics: stitching images together
![Page 10: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/10.jpg)
Readings
• Szeliski:
– Chapter 6.1: Feature-based alignment
– Chapter 9: Panorama stitching
![Page 11: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/11.jpg)
Image alignment
Image taken from same viewpoint, just rotated.
Can we line them up?
![Page 12: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/12.jpg)
Image alignment
Why don’t these image line up exactly?
![Page 13: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/13.jpg)
What is the geometric relationship between these two images?
?
![Page 14: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/14.jpg)
What is the geometric relationship between these two images?
?
![Page 15: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/15.jpg)
Is this an affine transformation?
![Page 16: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/16.jpg)
Where do we go from here?
affine transformation
what happens when we change this row?
![Page 17: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/17.jpg)
Projective Transformations aka Homographies aka Planar Perspective Maps
Called a homography(or planar perspective map)
projection of 3D plane can be explained by a (homogeneous) 2D transform
![Page 18: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/18.jpg)
Homographies
What happens when the denominator is 0?
![Page 19: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/19.jpg)
Homographies
• Example on board
![Page 20: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/20.jpg)
Image warping with homographies
image plane in front image plane below
black area
where no pixel
maps to
![Page 21: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/21.jpg)
Homographies
![Page 22: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/22.jpg)
Homographies
• Homographies …
– Affine transformations, and
– Projective warps
• Properties of projective transformations:
– Origin does not necessarily map to origin
– Lines map to lines
– Parallel lines do not necessarily remain parallel
– Ratios are not preserved
– Closed under composition
![Page 23: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/23.jpg)
2D image transformations
These transformations are a nested set of groups• Closed under composition and inverse is a member
![Page 24: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/24.jpg)
Questions?
![Page 25: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/25.jpg)
Creating a panorama
• Basic Procedure– Take a sequence of images from the same position
• Rotate the camera about its optical center
– Compute transformation between second image and first
– Transform the second image to overlap with the first
– Blend the two together to create a mosaic
– If there are more images, repeat
![Page 26: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/26.jpg)
Geometric interpretation of mosaics
0
1
2
![Page 27: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/27.jpg)
Geometric Interpretation of Mosaics
• If we capture all 360º of rays, we can create a 360º panorama• The basic operation is projecting an image from one plane to another• The projective transformation is scene-INDEPENDENT
• This depends on all the images having the same optical center
Image 1
Image 2
Optical Center
![Page 28: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/28.jpg)
Projecting images onto a common plane
mosaic PP
![Page 29: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/29.jpg)
Image reprojection
• Basic question– How to relate two images from the same camera center?
• how to map a pixel from PP1 to PP2 PP2
PP1
Answer• Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2
![Page 30: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/30.jpg)
What is the transformation?
Image 1
Image 2
Optical Center
How do we transform image 2 onto image 1’s projection plane?
image 1 image 2
3x3 homography
image coords(in image 2)
3D ray coords(in camera 2)
image coords(in image 2)
3D ray coords(in camera 1)
image coords(in image 2)
image coords(in image 1)
![Page 31: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/31.jpg)
Image alignment
![Page 32: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/32.jpg)
Image alignment
![Page 33: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/33.jpg)
Can we use homography to create a 360 panorama?
![Page 34: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/34.jpg)
Panoramas
• What if you want a 360 field of view?
mosaic Projection Sphere
![Page 35: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/35.jpg)
– Map 3D point (X,Y,Z) onto sphere
Spherical projection
X
Y
Z
unit sphere
unwrapped sphere
• Convert to spherical coordinates
Spherical image
• Convert to spherical image coordinates
– s defines size of the final image
» often convenient to set s = camera focal length
![Page 36: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/36.jpg)
f = 200 (pixels)
Spherical reprojection
• Map image to spherical coordinates
– need to know the focal length
input f = 800f = 400
![Page 37: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/37.jpg)
Aligning spherical images
• Suppose we rotate the camera by about the vertical axis– How does this change the spherical image?
![Page 38: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/38.jpg)
Aligning spherical images
• Suppose we rotate the camera by about the vertical axis– How does this change the spherical image?
– Translation by
– This means that we can align spherical images by translation
![Page 39: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/39.jpg)
Unwrapping a sphere
Credit: JHT’s Planetary Pixel Emporium
![Page 40: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/40.jpg)
+
+
+
+
Microsoft Lobby: http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski
Spherical panoramas
![Page 41: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/41.jpg)
Different projections are possible
![Page 42: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/42.jpg)
Questions?
• 3-minute break
![Page 43: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/43.jpg)
Computing transformations
• Given a set of matches between images A and B
– How can we compute the transform T from A to B?
– Find transform T that best “agrees” with the matches
![Page 44: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/44.jpg)
Computing transformations
?
![Page 45: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/45.jpg)
Simple case: translations
How do we solve for?
![Page 46: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/46.jpg)
Mean displacement =
Simple case: translations
Displacement of match i =
![Page 47: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/47.jpg)
Another view
• System of linear equations– What are the knowns? Unknowns?
– How many unknowns? How many equations (per match)?
![Page 48: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/48.jpg)
Another view
• Problem: more equations than unknowns– “Overdetermined” system of equations
– We will find the least squares solution
![Page 49: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/49.jpg)
Least squares formulation
• For each point
• we define the residuals as
![Page 50: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/50.jpg)
Least squares formulation
• Goal: minimize sum of squared residuals
• “Least squares” solution
• For translations, is equal to mean displacement
![Page 51: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/51.jpg)
Least squares formulation
• Can also write as a matrix equation
2n x 2 2 x 1 2n x 1
![Page 52: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/52.jpg)
Least squares
• Find t that minimizes
• To solve, form the normal equations
![Page 53: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/53.jpg)
Affine transformations
• How many unknowns?
• How many equations per match?
• How many matches do we need?
![Page 54: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/54.jpg)
Affine transformations
• Residuals:
• Cost function:
![Page 55: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/55.jpg)
Affine transformations
• Matrix form
2n x 6 6 x 1 2n x 1
![Page 56: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/56.jpg)
Homographies
To unwarp (rectify) an image• solve for homography H given p and p’
• solve equations of the form: wp’ = Hp
– linear in unknowns: w and coefficients of H
– H is defined up to an arbitrary scale factor
– how many points are necessary to solve for H?
pp’
![Page 57: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/57.jpg)
Solving for homographies
![Page 58: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/58.jpg)
Solving for homographies
![Page 59: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/59.jpg)
Solving for homographies
Defines a least squares problem:
• Since is only defined up to scale, solve for unit vector
• Solution: = eigenvector of with smallest eigenvalue
• Works with 4 or more points
2n × 9 9 2n
![Page 60: CS6670: Computer Vision - Cornell University · Creating a panorama •Basic Procedure –Take a sequence of images from the same position •Rotate the camera about its optical center](https://reader036.vdocuments.us/reader036/viewer/2022063004/5f89b29d5d1da547d577a89d/html5/thumbnails/60.jpg)
Questions?