scale invariant feature transform tom duerig. why do we care about matching features? object...
Post on 20-Dec-2015
213 views
TRANSCRIPT
![Page 1: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/1.jpg)
Scale Invariant Feature Transform
Tom Duerig
![Page 2: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/2.jpg)
Why do we care about matching features?
Object Recognition Wide baseline matching Tracking/SFM
![Page 3: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/3.jpg)
We want invariance!!!
Good features should be robust to all sorts of nastiness that can occur between images.
![Page 4: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/4.jpg)
Types of invariance
Illumination
![Page 5: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/5.jpg)
Types of invariance
Illumination Scale
![Page 6: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/6.jpg)
Types of invariance
Illumination Scale Rotation
![Page 7: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/7.jpg)
Types of invariance
Illumination Scale Rotation Affine
![Page 8: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/8.jpg)
Types of invariance
Illumination Scale Rotation Affine Full Perspective
![Page 9: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/9.jpg)
How to achieve illumination invariance
The easy way (normalized)
Difference based metrics (random tree, Haar, and sift)
![Page 10: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/10.jpg)
How to achieve scale invariance
Pyramids – Divide width and height by 2– Take average of 4 pixels for each pixel (or
Gaussian blur)– Repeat until image is tiny– Run filter over each size image and hope its
robust
Scale Space (DOG method)
![Page 11: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/11.jpg)
Pyramids
![Page 12: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/12.jpg)
How to achieve scale invariance
Pyramids Scale Space (DOG method)
– Pyramid but fill gaps with blurred images– Like having a nice linear scaling without the
expense– Take features from differences of these images– If the feature is repeatably present in between
Difference of Gaussians it is Scale Invariant and we should keep it.
![Page 13: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/13.jpg)
Differences Of Gaussians
![Page 14: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/14.jpg)
Rotation Invariance
Rotate all features to go the same way in a determined manner
Take histogram of Gradient directions (36 in paper for 1 every 10 degrees)
Rotate to most dominant (maybe second if its good enough, sub-Bin accuracy)
![Page 15: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/15.jpg)
Rotation Invariance
![Page 16: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/16.jpg)
Affine Invariance
Easy way: Warp your training and hope Fancy way: design your feature itself to be
robust against affine transformations (SIFT method)
![Page 17: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/17.jpg)
Actual SIFT features
Remember the gradient histograms we used for rotation invariance?
Same theory, except keep N2 histograms (4 shown, 16 used)
Note, use weighted contributions to avoid edge nastiness
![Page 18: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/18.jpg)
SIFT algorithm overview
Get tons of points from maxima+minima of DOGS
Threshold on simple contrast (low contrast is generally less reliable than high for feature points)
Threshold based on principal curvatures (technical term is linyness)
![Page 19: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/19.jpg)
SIFT algorithm overview
Gradient, histogram, Rotate Take old gradient, histogram regions using
Gaussian weighting Hand off beautiful robust feature to someone
who cares (i.e. object recognizer, navigation software (SFM), or stereo matching algorithm)
![Page 20: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/20.jpg)
Actual SIFT stage output
![Page 21: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/21.jpg)
MSER (a one minute survey)
Maximally Stable Extremal Regions Go through thresholds, grab regions which
stay nearly the same through a wide range of thresholds (connected components then bounding ellipses)
Keep those regions descriptors as features As regions are illumination based warp
ellipse to circle for affine invariance
![Page 22: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/22.jpg)
Actual MSER output
![Page 23: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/23.jpg)
Initial image
![Page 24: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/24.jpg)
Threshold
![Page 25: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/25.jpg)
Threshold
![Page 26: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/26.jpg)
Threshold
![Page 27: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/27.jpg)
Ears and Square (note: not actual algorithm output)
![Page 28: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/28.jpg)
How to use these features?
Distance could be L2 norm on histograms Match by (nearest neighbor distance)/(2nd
nearest neighbor distance) ratio Object recognize with Hough of pose of
points (ie, these three should be in line on object, gee… they are that’s the object all right)
![Page 29: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/29.jpg)
But where does the magic end?
SIFT is relatively expensive (computationally) and copyrighted (the other type of expensive)
MSER doesn’t work well with images with any motion blur
Interesting alternatives: – GLOH (Gradient Location and Orientation Histogram)
larger initial descriptor + PCA
– SURF (Speeded Up Robust Features) possibly faster AND more robust?
![Page 30: Scale Invariant Feature Transform Tom Duerig. Why do we care about matching features? Object Recognition Wide baseline matching Tracking/SFM](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d4e5503460f94a2e51c/html5/thumbnails/30.jpg)
Credits + References
MSER image from J. Matas, O. Chum, M. Urban, T. Pajdla “Robust Wide Baseline Stereo from Maximally Stable Extremal Regions”, BMVC, 2002
SIFT keypoint images from F. Estrada & A Jepson & D. Fleet ‘s SIFT tutorial, 2004
GLOH “A performance evaluation of local descriptors”, Krystian Mikolajczyk and Cordelia Schmid, IEEE tran. On Pattern Analysis and Machine Intelligence, pp 1615—1630, 2005
SURF “SURF: Speeded Up Robust Features”, Herbert Bay, Tinne Tuytelaars and Luc Van Gool Proceedings of the 9th European Conference on Computer Vision 2006