automated image stitching using sift feature matching
DESCRIPTION
article in image stitching methodTRANSCRIPT
![Page 1: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/1.jpg)
Automated Image Stitching Using SIFT Feature Matching
Ryan CrawfordComputer Vision EGGN 512
May 1st, 2012
![Page 2: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/2.jpg)
Outline
• Introduction• Finding & Matching Features• Verifying Matches• Compositing Images• Results• Errors & Limitations• Reducing Seams• Conclusion
![Page 3: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/3.jpg)
Introduction
• Algorithms for image stitching (or “mosaicking”) are some of the oldest computer vision problems
• Goal: Take several overlapping pictures of a scene and then let the computer determine the best way to combine them into a single image
![Page 4: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/4.jpg)
Applications
• Image panoramas• WAAS imagery1
• Satellite imagery• Radiology• Microscopy• Cell phone applications
[1] J. Prokaj and G. Medioni, “Accurate Efficient Mosaicking for Wide Area Aerial Surveillance”, University of Southern California Computer Vision, Los Angeles, CA, WACV 2012 pp. 273-280.
![Page 5: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/5.jpg)
Process1. Collect a set of overlapping images2. Choose a reference image3. For each image “x”:
1. Find feature matches between reference and x2. Determine transform from x to reference3. Transform x and place both on composite
surface4. Composite becomes new reference
4. Run aestethic algorithms (reduce seams, adjust for lighting, etc.)
![Page 6: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/6.jpg)
SIFT
• Scale Invariant Feature Transform• Find and match features between two images• Works very well even under rotation and
scaling changes between images• Approach2:
– Create a scale space of images: progressively blur images with a Gaussian
– Take difference between images (DoG)– Find the local extrema in this scale space
![Page 7: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/7.jpg)
SIFT cont.
– Choose keypoints (dependent on specified threshold)
– For each keypoint create a 16x16 window and find histograms of gradient directions
– Combine these into a feature vector (128 dimensions)
– Implemented with VL SIFT Matlab coding3
[2] Lowe, David G. "Distinctive Image Features from Scale-Invariant Keypoints." International Journal of Computer Vision 60.2 (2004): 91-110. Print.
![Page 8: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/8.jpg)
Matching Detected Features
• Use vl_sift to find features in each image– Can limit number of features detected with
threshold specifications• Use vl_ubcmatch to match features
between two images– Candidate matches are found by examining
the Euclidian distance between keypointfeature vectors
[3] Vedaldi, A., and B. Fulkerson. "VLFeat: An Open and Portable Library of Computer Vision Algorithms." 2008. Web. 1 Apr. 2012. <http://www.vlfeat.org>.
![Page 9: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/9.jpg)
Example
4709 Features Detected 8485 Features Detected
![Page 10: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/10.jpg)
Example
324 Matches detected, but there are some outliers
![Page 11: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/11.jpg)
Verifying Matches
• Used a coarse Hough space method4
– 4-dimensional space: x,y,scale,θ– Each match “votes” for a pose– The bin with the highest number of votes is
the mostly likely transformation between image 1 and image 2
– Used an affine transformation (rotation, scaling, translation) Can approximate small, out of plane rotations
[4] Hoff, William. "SIFT-Based Object Recognition." Colorado School of Mines, Golden, Colorado. 15 Apr. 2012. Lecture.
![Page 12: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/12.jpg)
Derived affine transformation:1.0e+003 *0.0010 -0.0000 -2.1750-0.0001 0.0010 0.2636
0 0 0.0010
Largest bin corresponds to 223 features
Outliers are filtered out
![Page 13: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/13.jpg)
Alternate Methods
• Other matching methods include:– RANSAC
• Not suitable if there are a lot of outliers– Correlation-Based
• Computationally expensive– Minimum variance of intensity estimate5
• Uses reference coordinate system rather than reference image, robust to lens distortion
[5] Sawhney, H.S.; Kumar, R.; , "True multi-image alignment and its application to mosaicing and lens distortion correction," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.21, no.3, pp.235-243, Mar 1999.
![Page 14: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/14.jpg)
Compositing Images
• Once we have the transformation from the first image to the reference image, we need to place the images on a composite surface
• I used a planar surface– Affine transformation
• Used Matlab’s imtranform function– Option to output “xdata” and “ydata” – Scalars corresponding to offset in x or y (positive
or negative)
![Page 15: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/15.jpg)
Compositing Images Cont.
• 4 Possible scenarios for where image should go: UL, LL, LR, UR6
• Create a blank canvas for each image that is big enough to hold reference and transformed image
• Base location on xdata and ydata• Use Matlab’s imsubtract and imadd
functions to combine the two canvases[6] Michael Carroll and Andrew Davidson. “Image Stitching with MatLab”. University of Louisiana State University. Department of Electrical and Computer Engineering.
![Page 16: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/16.jpg)
Results
![Page 17: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/17.jpg)
![Page 18: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/18.jpg)
![Page 19: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/19.jpg)
Purely in-plane rotation
![Page 20: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/20.jpg)
![Page 21: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/21.jpg)
Errors & Limitations
• Mis-registration can cause blurring• Repeated similar features can throw off
the matching algorithm• Moving objects or people can cause
“ghosting”• Used a planar compositing surface
– Cylindrical would work better for large out of plane camera rotations7
[7] Szeliski, Richard. "Image Stitching." Computer Vision: Algorithms and Applications. London: Springer, 2011. 375-406. Print.
![Page 22: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/22.jpg)
Repeated Similar features
The windows in the buildings are all very similar, so it’s hard to match features effectively in this image
![Page 23: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/23.jpg)
Ghosting
The person shown was in one frame of the picture and not the other, resulting in a “ghost”
![Page 24: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/24.jpg)
Other Examples
![Page 25: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/25.jpg)
Reducing Seams
• There are several ways to reduce the visible seams in mosaics7:– Feathering (weighted average)– LaPlacian Blending– Gradient-Domain Image Stitching– Regions of Differences (ROD)
• I tried averaging and LaPlacian blending but was unsuccessful in reducing the visible seams
![Page 26: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/26.jpg)
Conclusion• My algorithm works well as long as the out of
plane rotation is not too big• Similar repeated features caused the
matching to fail• Ghosting occurs when there are moving
objects• Future work: Try other matching algorithms,
Implement seam reduction, use a cylindrical composite surface, stitch together 10+ images
![Page 27: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/27.jpg)
QUESTIONS?
![Page 28: Automated Image Stitching Using SIFT Feature Matching](https://reader033.vdocuments.us/reader033/viewer/2022051001/5695d2621a28ab9b029a3989/html5/thumbnails/28.jpg)
References[1] J. Prokaj and G. Medioni. “Accurate Efficient Mosaicking for Wide Area Aerial Surveillance”, University of Southern California Computer Vision, Los Angeles, CA, WACV, 2012, pp. 273-280. Print.
[2] Lowe, David G. "Distinctive Image Features from Scale-Invariant Keypoints." International Journal of Computer Vision 60.2, 2004, pp. 91-110. Print.
[3] Vedaldi, A., and B. Fulkerson. "VLFeat: An Open and Portable Library of Computer Vision Algorithms." 2008. Web. 1 Apr. 2012. <http://www.vlfeat.org>.
[4] Hoff, William. "SIFT-Based Object Recognition." Colorado School of Mines, Golden, Colorado. 15 Apr. 2012. Lecture.
[5] Sawhney, H.S., and Kumar, R. "True multi-image alignment and its application to mosaicing and lens distortion correction," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol.21, no.3, Mar 1999, pp.235-243. Print.
[6] Michael Carroll and Andrew Davidson. “Image Stitching with MatLab”. University of Louisiana State University. Department of Electrical and Computer Engineering.
[7] Szeliski, Richard. "Image Stitching." Computer Vision: Algorithms and Applications. London: Springer, 2011, pp. 375-406. Print.