bovdw: bag-of-visual-and-depth- words for gesture recognition all rights reserved hubpa© human pose...
TRANSCRIPT
![Page 1: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/1.jpg)
BoVDW:Bag-of-Visual-and-Depth-Words for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Antonio Hernández-Vela1,2
Miguel Ángel Bautista1,2
Xavier Perez-Sala2,3
Victor Ponce Lopez1,2
Xavier Baro2,4
Oriol Pujol1,2
Cecilio Angulo3
Sergio Escalera1,2
1Dept. Applied Mathematics and Analysis, Universitat de Barcelona
2Computer Vision Center3CETpD, Universitat Politècnica de Catalunya
4EIMT, Universitat Oberta de Catalunya
![Page 2: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/2.jpg)
1. Introduction2. Methodology3. Results4. Conclusion
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Outline
![Page 3: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/3.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Bag of (Visual) WordsIntroduction Methodology Results Conclusion
![Page 4: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/4.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Bag of Visual and Depth WordsIntroduction Methodology Results Conclusion
• In this work, we propose:
• Bag of Visual and Depth Words (BoVDW).
• A new depth descriptor.
• Comparison with state-of-the-art descriptors.
• Gesture recognition framework.
![Page 5: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/5.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Standard pipelineIntroduction Methodology Results Conclusion
Point detection Point description Vocabulary & Representation Classification
![Page 6: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/6.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Introduction Methodology Results Conclusion
Point detection Point description Vocab. & Represent. Classification
• Spatio-Temporal Interest Points (STIPs) [1]
[1] I. Laptev, "On Space-Time Interest Points", (2005) in International Journal of Computer Vision, vol 64, number 2/3, pp.107-123.
Temporal extension of the Harris operator
![Page 7: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/7.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Introduction Methodology Results Conclusion
Point detection Point description Vocab. & Represent. Classification
• Viewpoint Feature Histogram (VFH)[2]
[2] Rusu, R.B et al., "Fast 3D recognition and pose using the Viewpoint Feature Histogram", IROS, 2010
Figs. credit to [2] and Aitor Aldoma
Histogram of angles between surface normals and viewpoint direction
• Camera Roll Histogram (CRH)
Invariant to rotations in the roll axis of the camera!
![Page 8: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/8.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Introduction Methodology Results Conclusion
Point detection Point description Vocab. & Represent. Classification
• Concatenation of VFH and CRH
![Page 9: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/9.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Introduction Methodology Results Conclusion
Point detection Point description Vocab. & Represent. Classification
• Vocabulary building K-means clustering
• Spatio-temporal pyramids
…
Final histogram: Concatenation of 8+1
histograms
![Page 10: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/10.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Introduction Methodology Results Conclusion
Point detection Point description Vocabulary Classification
• K-nearest neighbor classifier
• Distance function:
• Histogram intersection:
![Page 11: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/11.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Chalearn datasetIntroduction Methodology Results Conclusion
• RGB-D video sequences.
• Organised in 20 batches:
• 47 sequences of 1-5 gestures each.
• Gestures from certain lexicon.
• Same actor.
•One-shot learning problem:
• Just 1 training sample for gesture.
![Page 12: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/12.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
ResultsIntroduction Methodology Results Conclusion
• Evaluation measurement: Levenshtein distance
Series1
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45
0.3452
0.4144
0.331400000000001
0.4021
0.306400000000001
VFHCRHVFHHOGHOFHOFHOG
Depth
RGB
Mean Levenshtein distance
(α= 1)
(α= 0)
![Page 13: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/13.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Results (Late fusion)Introduction Methodology Results Conclusion
Late fusion approach Mean Lev. Dist.
HOGHOF/VFHCRH 0.2714
HOG/HOF/VFHCRH 0.2662
Batch number
Mea
n Le
vens
htei
n di
st.
(α= 0.8)
![Page 14: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/14.jpg)
BoVDW for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
ConclusionIntroduction Methodology Results Conclusion
• We have presented:
• BoVDW approach for gesture recognition.
• VFHCRH, a new depth descriptor.
• Comparison of state-of-the-art descriptors .
• Analysis of Late fusion of RGB and Depth information.
• Future work:
• Test other methodologies for spatial coherence.
• Improve continuous gesture detection.
![Page 15: BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649cf35503460f949c1298/html5/thumbnails/15.jpg)
BoVDW:Bag-of-Visual-and-Depth-Words for Gesture Recognition
All rights reserved HuBPA©
Human Pose Recovery and Behavior Analysis
Antonio Hernández-Vela1,2
Miguel Ángel Bautista1,2
Xavier Perez-Sala2,3
Victor Ponce Lopez1,2
Xavier Baro2,4
Oriol Pujol1,2
Cecilio Angulo3
Sergio Escalera1,2
1Dept. Applied Mathematics and Analysis, Universitat de Barcelona2Computer Vision Center
3CETpD, Universitat Politècnica de Catalunya4EIMT, Universitat Oberta de Catalunya
Thank you!Questions?