ensemble of exemplar-svm for object detection and beyond
TRANSCRIPT
![Page 1: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/1.jpg)
Ensemble of Exemplar-SVMs for Object Detection and Beyond
Tomasz MalisiewiczSeptember 20, 2011
Computer Vision Reading Group@MIT
Tomasz Malisiewicz, Abhinav Gupta and Alexei A. Efros. “Ensemble of Exemplar-SVMs for Object Detection and Beyond.” In ICCV, 2011.
![Page 2: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/2.jpg)
Overview
• Motivation and Related Work
• Learning Exemplar-SVMs
• Results
• PASCAL VOC Object Detection Results
• Transfer and Prediction
![Page 3: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/3.jpg)
![Page 4: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/4.jpg)
![Page 5: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/5.jpg)
Discriminative Object Detectors
Dalal and Triggs 2005
Linear SVM on HOGHard-Negative MiningSliding Window Detection
DT
![Page 6: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/6.jpg)
Discriminative Object Detectors
Dalal and Triggs 2005
Linear SVM on HOGHard-Negative MiningSliding Window Detection
DT, Felzenszwalb et al. 2010
PartsMixtures
LDPM
![Page 7: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/7.jpg)
Discriminative Object Detectors
Dalal and Triggs 2005
Linear SVM on HOGHard-Negative MiningSliding Window Detection
DT, Felzenszwalb et al. 2010
PartsMixtures
LDPM
Parametric: A fixed number of models per category
![Page 8: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/8.jpg)
Nearest Neighbor Approaches
• Non-parametric: keep all the data around
• Enables Label Transfer
• However
• No learning implies results depend on features and distance metric
• Not shown to compete with discriminatively-trained LDPM on Pascal
![Page 9: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/9.jpg)
Per-Exemplar Methods
• NN-method, where each exemplar has its own distance “similarity” function
• Better than using a single similarity measure across all exemplars
Frome et al. 2007, Malisiewicz et al. 2008
![Page 10: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/10.jpg)
Exemplar-SVMs
• Combine
• Effectiveness of discriminatively-trained object detectors
• Explicit correspondence of Nearest Neighbor approaches
![Page 11: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/11.jpg)
Exemplar-SVMs
• Learn a separate linear SVM for each instance (exemplar) in the dataset (PASCAL VOC)
![Page 12: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/12.jpg)
Exemplar-SVMs
• Learn a separate linear SVM for each instance (exemplar) in the dataset (PASCAL VOC)
• Each Exemplar-SVM is trained with a single positive instance
![Page 13: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/13.jpg)
Exemplar-SVMs
• Learn a separate linear SVM for each instance (exemplar) in the dataset (PASCAL VOC)
• Each Exemplar-SVM is trained with a single positive instance
• Each Exemplar-SVM is more defined by “what it is not” vs. “what it is similar to”
![Page 14: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/14.jpg)
Exemplar-SVMs
• Because each Exemplar-SVM is defined by a single positive instance, we can use different features for each exemplar
![Page 15: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/15.jpg)
Exemplar-SVMs
• Because each Exemplar-SVM is defined by a single positive instance, we can use different features for each exemplar
7x4 HOG 4x8 HOG
• Adapt features to each exemplar’s aspect ratio
![Page 16: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/16.jpg)
Exemplar-SVMsExemplar E’s Objective Function:
h(x) = max(1-x,0) “hinge-loss”
![Page 17: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/17.jpg)
Exemplar-SVMs
Exemplar represented by ~100 HOG Cells (~3,100 features)
Exemplar E’s Objective Function:
h(x) = max(1-x,0) “hinge-loss”
![Page 18: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/18.jpg)
Exemplar-SVMs
Windows from images not containing any in-class instances (~2,000 images x ~10,000 windows/image = ~2M negatives )
Exemplar represented by ~100 HOG Cells (~3,100 features)
Exemplar E’s Objective Function:
h(x) = max(1-x,0) “hinge-loss”
![Page 19: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/19.jpg)
Large-scale training
• Each exemplar performs its own hard negative mining
• Solve many convex learning problems
• Parallel training on cluster
CPU1 CPU2 CPUN
Ex1 Ex2 ExN...
![Page 20: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/20.jpg)
Exemplar-SVM Calibration
“Leave All But One Out Calibration”
![Page 21: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/21.jpg)
Exemplar-SVM Calibration
1) Apply ExemplarSVM to held-out negative
images and all positive images
“Leave All But One Out Calibration”
![Page 22: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/22.jpg)
Exemplar-SVM Calibration
1) Apply ExemplarSVM to held-out negative
images and all positive images
“Leave All But One Out Calibration”2) Fit sigmoid to
responses [Platt 1999]
![Page 23: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/23.jpg)
Exemplar-SVM Calibration
1) Apply ExemplarSVM to held-out negative
images and all positive images
“Leave All But One Out Calibration”2) Fit sigmoid to
responses [Platt 1999]
![Page 24: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/24.jpg)
Ensemble of Exemplar-SVMsExemplars
Image + Detections
![Page 25: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/25.jpg)
Ensemble of Exemplar-SVMs
Learn an exemplar co-occurence matrix
Exemplars
Image + Detections
![Page 26: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/26.jpg)
Qualitative Results
• Let’s take a look at some Exemplar-SVM results in PASCAL VOC dataset
![Page 27: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/27.jpg)
Exemplar w Averaged Detections
![Page 28: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/28.jpg)
Exemplar w Averaged Detections
![Page 29: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/29.jpg)
Exemplar w Averaged Detections
Average of first 10
detections
Average of first 20
detections
![Page 30: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/30.jpg)
![Page 31: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/31.jpg)
![Page 32: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/32.jpg)
![Page 33: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/33.jpg)
![Page 34: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/34.jpg)
EvaluatingExemplar-SVMs
• Nearest Neighbor
• No Learning
• Per-Exemplar Distance Functions
• Learning in distance-to-exemplar space [Malisiewicz et al. 2008]
• Exemplar-SVMs
![Page 35: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/35.jpg)
Comparison of 3 methods
!"#$%&'()*+,
--
!"#$%&'( ! ./%0102#3#435/6708(/$0.#737#3
9:;<
*Learned Distance Function
*
![Page 36: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/36.jpg)
Comparison of 3 methods
!"#$%&'()*+,
--
!"#$%&'( ! ./%0102#3#435/6708(/$0.#737#3
9:;<
*Learned Distance Function
*
![Page 37: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/37.jpg)
Comparison of 3 methods
!"#$%&'()*+,
--
!"#$%&'( ! ./%0102#3#435/6708(/$0.#737#3
9:;<
*Learned Distance Function
*
![Page 38: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/38.jpg)
Comparison of 3 methods
!"#$%&'()*+,
--
!"#$%&'( ! ./%0102#3#435/6708(/$0.#737#3
9:;<
*Learned Distance Function
*
![Page 39: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/39.jpg)
Quantitative: PASCAL VOC 2007 dataset
• A standard computer vision object detection benchmark
• 20 object categories
• Machine performance is far below human
![Page 40: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/40.jpg)
PASCAL VOC 2007 Object Category Detection Results
![Page 41: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/41.jpg)
Object Category Detection
NN + Cal 0.110
DFUN + Cal 0.155
Exemplar-SVMs + Cal 0.198
Exemplar-SVMs + Co-occ 0.227
DT* 0.097
LDPM** 0.266
mAP on PASCAL VOC 2007 detection task
*Dalal et al. 2005 **Felzenszwalb et al. 2010
![Page 42: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/42.jpg)
Beyond Object Category Detection
• Based on the idea of label transfer, ExemplarSVMs can be used for tasks which go beyond object category detection
![Page 43: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/43.jpg)
Task 1: Geometry Transfer
![Page 44: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/44.jpg)
![Page 45: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/45.jpg)
![Page 46: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/46.jpg)
![Page 47: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/47.jpg)
![Page 48: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/48.jpg)
![Page 49: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/49.jpg)
Task 1: Evaluation on Buses
• 43.0% Hoiem et al. 2005
• 51.0% Category-SVM* + NN
• 62.3% Exemplar-SVMs
• measure pixelwise accuracy on the 3-class geometric-labeling problem: “left,” “front,” “right”-facing
*Felzenszwalb et al. 2010
![Page 50: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/50.jpg)
Detector w
Appearance
Exemplar
Task II: Person Prediction
![Page 51: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/51.jpg)
Detector w
Appearance
Exemplar
Meta-dataPerson
Task II: Person Prediction
![Page 52: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/52.jpg)
Detector w
Appearance
Exemplar
PersonMeta-data
Person
Task II: Person Prediction
![Page 53: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/53.jpg)
![Page 54: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/54.jpg)
![Page 55: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/55.jpg)
Task II: Evaluation
![Page 56: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/56.jpg)
More Transfer Examples
![Page 57: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/57.jpg)
![Page 58: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/58.jpg)
![Page 59: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/59.jpg)
3D Model Transfer
Manually align 3D model from Google 3D Warehouse with a subset of PASCAL
VOC “chair” exemplars
![Page 60: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/60.jpg)
![Page 61: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/61.jpg)
![Page 62: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/62.jpg)
Conclusion
![Page 63: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/63.jpg)
Conclusion• ExemplarSVMs can be used for recognition, label
transfer, and complementary object prediction
![Page 64: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/64.jpg)
Conclusion
!"# !!!
"#$%&'()*+,-./ "#$%&'()*+,-.0 "#$%&'()*+,-.1-232'45647.+,-
• Large-scale negative mining is the key to learning a good ExemplarSVM
• ExemplarSVMs can be used for recognition, label transfer, and complementary object prediction
![Page 65: Ensemble of Exemplar-SVM for Object Detection and Beyond](https://reader034.vdocuments.us/reader034/viewer/2022052506/557cce8cd8b42a7e5b8b4f6c/html5/thumbnails/65.jpg)
Thank You
Questions?