![Page 1: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/1.jpg)
3D object categorization, detection, and viewpoint classification
Final Presentation
Min SunEECS, University of Michigan at Ann Arbor
Mentor: Gary Bradski
![Page 2: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/2.jpg)
2
Goal: Viewpoint ClassificationGoal: Detection and Categorization
![Page 3: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/3.jpg)
Approaches
• Discriminative Codeword (Random Forest), using Random Forest
• Hough voting for each viewing region
Mouse, V1
Mouse, V2
![Page 4: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/4.jpg)
First Step: System implementation
• Using OpenCV and octave to re-implement the system
• Old system in Matlab: slow and not open source
• New system is fast and open source
• Speed-up detection from ~2 minutes to ~5 seconds for single object class
• Create a ROS node (rf_detector) to recognize object online
![Page 5: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/5.jpg)
Challenges
• Need good shape descriptor for objects with less texture
• Need to have a multi-class object detector to detect multiple object classes at the same time
![Page 6: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/6.jpg)
Second Step: system upgrade
• Exploring different shape features:
1. Histogram Oriented Gradients (opencv)
2. Geometric Blur (geometric_blur in ROS)
3. Berkeley natural boundary (Nb) detector
Berkeley (Nb) OpenCV
![Page 7: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/7.jpg)
Conclusion
• Hog has similar performance as Geometric blur+natural boundary detector(Nb)
• It takes 3 minutes to compute Natural boundary(Nb) for each image
• Hog is fast and almost the best
Recall Mouse Stapler
Gb+Nb 28% 37%
Hog+Nb 25% 45%
Hog 30% 35%
![Page 8: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/8.jpg)
Second Step: system upgrade
• Multi-class Random Forest
Mouse
Stapler
Mug & Mouse
Mug
![Page 9: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/9.jpg)
Third Step: 3d information
• Using stereo depth to sample image patches corresponding to fix physical size to avoid scale search
• Using Dan’s shape spectral and spin image descriptors in descriptor_3d (ROS pkg)
• Combine both Hog and 3d descriptors
![Page 10: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/10.jpg)
Data collection
• Table top object classes: mice, staplers, and mugs
• Collect aligned images and dense stereo point clouds
![Page 11: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/11.jpg)
Multiple Views
![Page 12: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/12.jpg)
Third Step: Results
mouse
staplermug
mouse stapler mug
![Page 13: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/13.jpg)
Mouse
![Page 14: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/14.jpg)
Stapler
![Page 15: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/15.jpg)
Mug
![Page 16: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/16.jpg)
Third Step: Comparison
Average Precision
ClassificationAccuracy
spin 0.213 0.2
shape 0.138 0.4
hog 0.635 0.73
hog+spin 0.612 0.7
Hog+shape 0.67 0.72
![Page 17: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/17.jpg)
Working system
• Texture_light_on_off node aligns images w/o texture light and dense stereo point clouds
• Table top object detector (t2obj) segments out the point clouds of table top objects
• Finally, rf_detector recognizes object locations, classes, and viewpoints.
![Page 18: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/18.jpg)
Results: miceRecognition Table top segmentation
![Page 19: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/19.jpg)
Results: mugsRecognition Table top segmentation
![Page 20: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/20.jpg)
Results: staplersRecognition Table top segmentation
![Page 21: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/21.jpg)
ResultsRecognition Table top segmentation
![Page 22: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/22.jpg)
ResultsRecognition Table top segmentation
![Page 23: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/23.jpg)
• Train on 3d+image, Test on image only
Use image patches of fix physical size to detect objects and infer the 3d position of the supporting image patches -> Object Pop-Up
• Vote for object center directly in 3d
Make the model fully rotational invariant and more compact
Future work
![Page 24: 3D object categorization, detection, and viewpoint · PDF file3D object categorization, detection, and viewpoint classification ... seconds for single object class •Create a ROS](https://reader031.vdocuments.us/reader031/viewer/2022021823/5aa2ab6a7f8b9aa0108d839d/html5/thumbnails/24.jpg)
Thank you