![Page 1: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/1.jpg)
Total Recall: Automatic Query Expansion with a Generative Feature Model for
Object Retrieval
O. Chum, et al.
presented by Yangzihao Wang
![Page 2: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/2.jpg)
Problem to Solve
Given a query image of an object, retrieve all instances of that object in a large (1M+) image database.
Slide Credit: Brandon Smith
![Page 3: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/3.jpg)
What is Total Recall?● Precision-Recall Curve
● Average Precision○ area under the precision-recall curve
![Page 4: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/4.jpg)
Approach
figure credit: Relja Arandjelovic
![Page 5: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/5.jpg)
Real-time Object Retrieval
● Generate descriptor○ Hessian-Affine Region+SIFT
● Quantization○ Approx k-means based on random trees○ forming visual words
● Search Engine○ inverted file(maps words to docs in which they
occur)○ tf-idf weighting scheme
![Page 6: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/6.jpg)
Main Contributions
figure credit: Relja Arandjelovic
![Page 7: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/7.jpg)
Spatial Verification
It is vital for query-expansion that we do not…● …expand using false positives, or● …use features which occur in the result
image, but not in the object of interest
figure credit: Ondrej Chum et al’s slides for ICCV ‘07
![Page 8: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/8.jpg)
Spatial Verification
figure credit: Michael Isard
● Use hypothesize and verify procedure to estimate homography between query and target
● > 20 inliers = spatially verified result
![Page 9: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/9.jpg)
● Evaluate the original query Q0● Construct latent model● Issue new query/queries
Query Expansion
![Page 10: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/10.jpg)
Generative ModelQuery Expansion Baseline● Find top 5 (unverified) results from original
query● Average the term-frequency vectors● Requery once● Append these results
![Page 11: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/11.jpg)
Generative ModelTransitive Closure Expansion● Create a priority queue based on number of inliers● Get top image in the queue● Find region corresponding to original query● Use this region to issue a new query● Add new verified results to queue● Repeat until queue is empty
![Page 12: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/12.jpg)
Generative ModelAverage Query Expansion● Obtain top (m < 50) verified results of
original query● Construct new query using average of these
results● Requery once● Append these results
![Page 13: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/13.jpg)
Generative ModelRecursive Avg Query Expansion● Improvement of average query expansion method● Recursively generate queries from all verified results
returned so far● Stop once 30 verified images are found or once no new
images can be positively verified
![Page 14: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/14.jpg)
Generative ModelMulti-resolution Expansion● For each verified result of original query
calculate relative change in resolution required to project verified region onto query region
![Page 15: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/15.jpg)
Generative ModelMulti-resolution Expansion● Place results in 3 bands:(0,⅘), (⅔, 3/2), (5/4, inf) ● Construct average query for each band● Execute independent queries● Merge results
![Page 16: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/16.jpg)
Experiments
Datasets:● Oxford dataset: 5k high res. with ground
truth labels of Oxford famoust landmarks● Flickr1 (high-res popular tag)● Flickr2 (medium-res popular tag)● For experiment, using:
○ D1: Oxford+Flickr1○ D2: Oxford+Flickr1+Flickr2
![Page 17: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/17.jpg)
![Page 18: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/18.jpg)
ExperimentsEvaluation:● Precision-Recall Curve
● Average Precision○ area under the precision-recall curve
![Page 19: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/19.jpg)
Experiments
Slide credit: Ondrej Chum
![Page 20: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/20.jpg)
Experiments
Slide credit: Ondrej Chum
![Page 21: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/21.jpg)
Experiments
ori = original queryqeb = query expansion baselinetrc = transitive closure expansionavg = average query expansionrec = recursive average query expansionsca = multiple image resolution expansion
Slide credit: Ondrej Chum
![Page 22: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/22.jpg)
Experiments
![Page 23: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/23.jpg)
Experiments
![Page 24: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/24.jpg)
Experiments
![Page 25: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/25.jpg)
ConclusionHave successfully ported methods from text retrieval to the visual domain:● Visual words enable posting lists for efficient
retrieval of specific objects
● Spatial verification improves precision
● Query expansion improves recall, without drift
![Page 26: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/26.jpg)
Future Works
How to deal with confuser?
How to improve spatial verification?● compare with previous verified images● add spatial context outside the original
region
![Page 27: Object Retrieval with a Generative Feature Model for Total Recall ...€¦ · Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval O. Chum,](https://reader035.vdocuments.us/reader035/viewer/2022071018/5fd17358dd40551ab220213a/html5/thumbnails/27.jpg)
Question?