![Page 1: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/1.jpg)
Bag of Features Approach: recent work, using geometric information
![Page 2: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/2.jpg)
Problem
• Search for object occurrences in very large image collection
![Page 3: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/3.jpg)
2 sub problems
• Object Category Recognition and Specific Object Recognition
![Page 4: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/4.jpg)
Motivation
• Look for product information• Look for similar products
![Page 5: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/5.jpg)
Related work on large scale image search
• Most systems build upon the BoF framework [Sivic & Zisserman 03]– Large (hierarchical) vocabularies [Nister Stewenius 06]– Improved descriptor representation [Jégou et al 08, Philbin et
al 08]– Geometry used in index [Jégou et al 08, Perdoc’h et al 09]– Query expansion [Chum et al 07]– …
• Efficiency improved by:– Min-hash and Geometrical min-hash [Chum et al. 07-09]– Compressing the BoF representation [Jégou et al. 09]
![Page 6: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/6.jpg)
Local Features - SIFT
![Page 7: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/7.jpg)
Creating a visual vocabulary1 2
3 4
![Page 8: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/8.jpg)
Inverted Index
Index construction Searching
![Page 9: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/9.jpg)
Use geometry
• Possible directions:– Change/optimize spatial verification stage– Insert a new geometric information to the index• Ordered BOF• Bundled features• Visual phrases
– Change the searching algorithm
![Page 10: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/10.jpg)
Survey for today
• Spatial Bag-of-features [Cao, CVPR2010]• Image Retrieval with Geometry-Preserving
Visual Phrases [Zhang Jia Chen, CVPR2011]• Smooth Object Retrieval using a Bag of
Boundaries [Arandjelovi Zisserman, ICCV2011]
![Page 11: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/11.jpg)
Spatial BOF
• Basic idea:
![Page 12: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/12.jpg)
Spatial BOF
• Constructing linear and circular ordered bag-of-features:
![Page 13: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/13.jpg)
Spatial BOF
• Translation invariance:
![Page 14: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/14.jpg)
Spatial BOF
• Pros:– Gets better performance than BOF+RANSAC for large scale
dataset*– Same format as standard BOF
• Cons:– Is dataset dependent because of need of training
• Do not present the results for large scale dataset with transfer learning from another dataset
• Future work– Check it with cross training for large dataset. Otherwise, it
is not worth working further.
![Page 15: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/15.jpg)
Geometry-Preserving Visual Phrases
• Basic idea:
![Page 16: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/16.jpg)
Geometry-Preserving Visual Phrases
• Representation– Quantize image to 10x10 grid– Histogram of GVPs of length k– GVP dictionary size is “choose k from N visual
words”
![Page 17: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/17.jpg)
Geometry-Preserving Visual Phrases
• Pros:– Outperforms BOV + RANSAC
• Cons:– Only translation invariant because of memory
• Future work
![Page 18: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/18.jpg)
BOF for smooth objects
Idea:
The information used for retrievalQuery object
Segment Gradient
![Page 19: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/19.jpg)
BOF for smooth objects
Results:
![Page 20: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/20.jpg)
BOF for smooth objects
Segmentation phase
• Over segmentation with super-pixels• Classification of super-pixels:• 3208 feature vector (median(Mag(Grad)), 4 bits, color
histogram, BOF)• SVM
• Post-processing
![Page 21: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/21.jpg)
BOF for smooth objects
Boundary description phase:• Sample points on the boundary• Calculate HoG at each point in 3 scales
340 dimensional
L2 normalized vector
* The descriptor is not rotation invariant
![Page 22: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/22.jpg)
BOF for smooth objects
Retrieval procedure:• Boundary descripors are quantized (k=10k)• Standard BOF scheme*• Spatial verification for top 200 with loose
affine homography (errors up to 100pixs)
* No spatial information is recorded in the histogram
![Page 23: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/23.jpg)
BOF for smooth objects
• Pros:– Solves the smooth object retrieval problem– Fast
• Cons:– Is dataset dependent because of need of training– Limited to objects with “solid” materials –
segmentation has to catch the object’s boundary• Future work– Eliminate the training step
![Page 24: Bag of Features Approach: recent work, using geometric information](https://reader035.vdocuments.us/reader035/viewer/2022062516/56649d2d5503460f94a044af/html5/thumbnails/24.jpg)
Summary
• There is an active research in the field of CBIR to exploit geometry information.
• Each method with its limitations• Still no widely accepted solution– Like spatial verification with RANSAC