image stitching
DESCRIPTION
Image Stitching. Tamara Berg CSE 590 Computational Photography. Many slides from Alyosha Efros & Derek Hoiem. How can we align two pictures?. What about global matching?. How can we align two pictures?. Global matching? But what if Not just translation change, but rotation and scale? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/1.jpg)
Image Stitching
Tamara BergCSE 590 Computational Photography
Many slides from Alyosha Efros & Derek Hoiem
![Page 2: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/2.jpg)
How can we align two pictures?
• What about global matching?
![Page 3: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/3.jpg)
How can we align two pictures?
• Global matching?– But what if
• Not just translation change, but rotation and scale?• Only small pieces of the pictures match?
![Page 4: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/4.jpg)
Keypoint Matching
K. Grauman, B. Leibe
Af Bf
B1
B2
B3A1
A2 A3
Tffd BA ),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 5: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/5.jpg)
Main challenges
• Change in position, scale, and rotation
• Change in viewpoint
• Occlusion
• Articulation, change in appearance
![Page 6: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/6.jpg)
Question
• Why not just take every patch in the original image and find best match in second image?
![Page 7: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/7.jpg)
Goals for Keypoints
Detect points that are repeatable and distinctive
![Page 8: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/8.jpg)
Key trade-offs
More Points More Repeatable
B1
B2
B3A1
A2 A3
Localization
More Robust More Selective
Description
Robust to occlusionWorks with less texture
Robust detectionPrecise localization
Deal with expected variationsMaximize correct matches
Minimize wrong matches
![Page 9: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/9.jpg)
Keypoint Localization
• Goals: – Repeatable detection– Precise localization
K. Grauman, B. Leibe
![Page 10: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/10.jpg)
Which patches are easier to match?
?
![Page 11: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/11.jpg)
Choosing interest points
• If you wanted to meet a friend would you saya) “Let’s meet on campus.”b) “Let’s meet on Green street.”c) “Let’s meet at Green and Wright.”
• Or if you were in a secluded area:a) “Let’s meet in the Plains of Akbar.”b) “Let’s meet on the side of Mt. Doom.”c) “Let’s meet on top of Mt. Doom.”
![Page 12: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/12.jpg)
Choosing interest points
• Corners– “Let’s meet at Green and Wright.”
• Peaks/Valleys – “Let’s meet on top of Mt. Doom.”
![Page 13: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/13.jpg)
Many Existing Detectors Available
K. Grauman, B. Leibe
Hessian & Harris [Beaudet ‘78], [Harris ‘88]Laplacian, DoG [Lindeberg ‘98], [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] Others…
![Page 14: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/14.jpg)
Harris Detector [Harris88]
K. Grauman, B. Leibe
Intuition: Search for local neighborhoods where the image content has two main directions.
![Page 15: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/15.jpg)
Harris Detector – Responses [Harris88]
Effect: A very precise corner detector.
![Page 16: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/16.jpg)
Harris Detector – Responses [Harris88]
![Page 17: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/17.jpg)
So far: can localize in x-y, but not scale
![Page 18: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/18.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),(( )),((11
xIfxIfmm iiii
How to find corresponding patch sizes?
![Page 19: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/19.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 20: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/20.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 21: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/21.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 22: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/22.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 23: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/23.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 24: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/24.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 25: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/25.jpg)
What Is A Useful Signature Function?
• Difference of Gaussian = “blob” detector
K. Grauman, B. Leibe
![Page 26: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/26.jpg)
DoG – Efficient Computation• Computation in Gaussian scale pyramid
K. Grauman, B. Leibe
Original image 41
2
Sampling withstep 4 =2
![Page 27: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/27.jpg)
Results: Lowe’s DoG
K. Grauman, B. Leibe
![Page 28: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/28.jpg)
T. Tuytelaars, B. Leibe
Orientation Normalization
• Compute orientation histogram• Select dominant orientation• Normalize: rotate to fixed orientation
0 2p
[Lowe, SIFT, 1999]
![Page 29: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/29.jpg)
Available at a web site near you…
• For most local feature detectors, executables are available online:– http://robots.ox.ac.uk/~vgg/research/affine– http://www.cs.ubc.ca/~lowe/keypoints/– http://www.vision.ee.ethz.ch/~surf
K. Grauman, B. Leibe
![Page 30: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/30.jpg)
How do we describe the keypoint?
![Page 31: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/31.jpg)
Local Descriptors
• The ideal descriptor should be– Robust– Distinctive– Compact– Efficient
• Most available descriptors focus on edge/gradient information– Capture texture information– Color rarely used
K. Grauman, B. Leibe
![Page 32: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/32.jpg)
Local Descriptors: SIFT Descriptor
[Lowe, ICCV 1999]
Histogram of oriented gradients• Captures important texture information• Robust to small translations / affine deformations
K. Grauman, B. Leibe
![Page 33: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/33.jpg)
What to use when?
Detectors• Harris gives very precise localization but doesn’t
predict scale– Good for some tracking applications
• DOG (difference of Gaussian) provides ok localization and scale– Good for multi-scale or long-range matching
Descriptors• SIFT: good general purpose descriptor
![Page 34: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/34.jpg)
Things to remember• Keypoint detection: repeatable
and distinctive– Corners, blobs– Harris, DoG
• Descriptors: robust and selective– SIFT: spatial histograms of gradient
orientation
![Page 35: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/35.jpg)
Image Stitching• Combine two or more overlapping images to
make one larger image
Add example
Slide credit: Vaibhav Vaish
![Page 36: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/36.jpg)
Panoramic Imaging
• Higher resolution photographs, stitched from multiple images
• Capture scenes that cannot be captured in one frame
• Cheaply and easily achieve effects that used to cost a lot of money
![Page 37: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/37.jpg)
Photo: Russell J. Hewett
Pike’s Peak Highway, CO
Nikon D70s, Tokina 12-24mm @ 16mm, f/22, 1/40s
![Page 38: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/38.jpg)
Photo: Russell J. Hewett
Pike’s Peak Highway, CO
(See Photo On Web)
![Page 39: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/39.jpg)
Photo: Russell J. Hewett
360 Degrees, Tripod Leveled
Nikon D70, Tokina 12-24mm @ 12mm, f/8, 1/125s
![Page 40: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/40.jpg)
Photo: Russell J. Hewett
Howth, Ireland
(See Photo On Web)
![Page 41: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/41.jpg)
Capturing Panoramic Images
• Tripod vs Handheld• Help from modern cameras• Leveling tripod• Or wing it
• Exposure• Consistent exposure between frames• Gives smooth transitions• Manual exposure
• Caution• Distortion in lens (Pin Cushion, Barrel, and Fisheye)• Motion in scene
• Image Sequence• Requires a reasonable amount of overlap (at least 15-30%)• Enough to overcome lens distortion
![Page 42: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/42.jpg)
Photo: Russell J. Hewett
Handheld Camera
Nikon D70s, Nikon 18-70mm @ 70mm, f/6.3, 1/200s
![Page 43: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/43.jpg)
Photo: Russell J. Hewett
Handheld Camera
![Page 44: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/44.jpg)
Photo: Russell J. Hewett
Les Diablerets, Switzerland
(See Photo On Web)
![Page 45: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/45.jpg)
Photo: Russell J. Hewett & Bowen Lee
Macro
Nikon D70s, Tamron 90mm Micro @ 90mm, f/10, 15s
![Page 46: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/46.jpg)
Photo: Russell J. Hewett & Bowen Lee
Side of Laptop
![Page 47: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/47.jpg)
Photo: Russell J. Hewett
Ghosting and Variable Intensity
Nikon D70s, Tokina 12-24mm @ 12mm, f/8, 1/400s
![Page 48: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/48.jpg)
Photo: Russell J. Hewett
![Page 49: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/49.jpg)
Photo: Bowen Lee
Ghosting From Motion
Nikon e4100 P&S
![Page 50: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/50.jpg)
Photo: Russell J. Hewett Nikon D70, Nikon 70-210mm @ 135mm, f/11, 1/320s
Motion Between Frames
![Page 51: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/51.jpg)
Photo: Russell J. Hewett
![Page 52: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/52.jpg)
Photo: Russell J. Hewett
Gibson City, IL
(See Photo On Web)
![Page 53: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/53.jpg)
Photo: Russell J. Hewett
Mount Blanca, CO
Nikon D70s, Tokina 12-24mm @ 12mm, f/22, 1/50s
![Page 54: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/54.jpg)
Photo: Russell J. Hewett
Mount Blanca, CO
(See Photo On Web)
![Page 55: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/55.jpg)
Image Stitching Algorithm Overview
1. Detect keypoints2. Match keypoints3. Estimate homography with matched
keypoints (using RANSAC)4. Project onto a surface and blend
![Page 56: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/56.jpg)
Image Stitching Algorithm Overview
1. Detect keypoints (e.g., SIFT)2. Match keypoints
![Page 57: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/57.jpg)
Computing homography
If we have 4 matched points we can compute homography H
Hxx '
987
654
321
hhhhhhhhh
H
![Page 58: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/58.jpg)
Computing homography
Assume we have matched points with outliers: How do we compute homography H?
Automatic Homography Estimation with RANSAC
![Page 59: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/59.jpg)
RANSAC: RANdom SAmple ConsensusScenario: We’ve got way more matched points than needed to fit the parameters, but we’re not sure which are correct
RANSAC Algorithm• Repeat N times
1. Randomly select a sample– Select just enough points to recover the parameters (4)2. Fit the model with random sample3. See how many other points agree
• Best estimate is one with most agreement– can use agreeing points to refine estimate
![Page 60: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/60.jpg)
Automatic Image Stitching
1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched points and RANSAC
4. Project each image onto the same surface and blend
![Page 61: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/61.jpg)
RANSAC for Homography
Initial Detected Points
![Page 62: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/62.jpg)
RANSAC for Homography
Final Matched Points
![Page 63: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/63.jpg)
RANSAC for Homography
![Page 64: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/64.jpg)
Blending to remove artifacts
• Burt & Adelson 1983
![Page 65: Image Stitching](https://reader035.vdocuments.us/reader035/viewer/2022062815/56816958550346895de1072d/html5/thumbnails/65.jpg)
Further reading
Harley and Zisserman: Multi-view Geometry book• DLT algorithm: HZ p. 91 (alg 4.2), p. 585• Normalization: HZ p. 107-109 (alg 4.2)• RANSAC: HZ Sec 4.7, p. 123, alg 4.6• Tutorial:
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdf
• Recognising Panoramas: Brown and Lowe, IJCV 2007