single person pose recognition and tracking
DESCRIPTION
Slides for my Master Thesis defense at TUDelft.TRANSCRIPT
![Page 1: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/1.jpg)
Single Person Pose Recognition and Tracking
Defender: Javier Barbadillo AmorInformation and Communication Theory (ICT) GroupDelft University of Technology
Committee: Dr. Alan HanjalicDr. Emile. A. HendriksPhD. Feifei HuoDr. Pavel Paclik
25-06-2010
![Page 2: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/2.jpg)
2
![Page 3: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/3.jpg)
3
Outline:
Introduction Single Person Pose Recognition and Tracking System
Theory
The goal of this researchImprove Body Parts Detection and Pose recognition
Experiments and Results• Improved Hand Detection• Detection of a new class: Non-Pose Classification
Conclusions
Future Work
![Page 4: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/4.jpg)
4
Introduction
![Page 5: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/5.jpg)
5
Single Person Pose Recognition and Tracking System
• Real time • One single camera• Game control with detected poses
![Page 6: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/6.jpg)
Theory: Background Subtraction by Mixture of Gaussians
• Compare the current frame with a model of the background.• Obtain a binary image with the foreground pixels
6
![Page 7: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/7.jpg)
7
Theory: Background Subtraction by Mixture of Gaussians
• History of pixel intensity values:
• An intensity value belongs to a Gaussian Distribution if it is within [-2.5σ, 2.5σ]
• Each pixel is modeled by K Gaussians.
![Page 8: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/8.jpg)
Theory: Particle Filter for tracking the torso and head
8
• Torso and head region detection
• Hand Detection
![Page 9: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/9.jpg)
Theory: Particle filter for tracking torso and head
9
•N particles are generated
• Weights π assigned according to measured probability.
• Father particles spread into G sons
![Page 10: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/10.jpg)
10
• Sample sets of particles are generated for 3 states: x, y and Scale
• The probability of the state of the torso is given by
Theory: Particle filter for tracking torso and head
Primitive for torso and head
![Page 11: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/11.jpg)
11
• Hand Detection with general skin color model
Theory: General skin color detection
![Page 12: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/12.jpg)
• Relative distances between hands and torso center.• Angles of the hands with the torso center.• r, l and t stand for right, left and torso.
12
Theory: Feature extraction
![Page 13: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/13.jpg)
• Incoming observations = 6-feature-set• Classifier decides one Pose class.• Each Pose number is a different action in the game
13
Theory: Pose Classification
![Page 14: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/14.jpg)
The goal of this research• Improve the system performance
– Hand detection: fails for short sleeves and “skin color clothes”– Pose recognition: detect Non-Poses
14
Hands detected in the forearm The 9 Predefined Poses
![Page 15: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/15.jpg)
15
Experiments and Results
![Page 16: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/16.jpg)
Skin color detection combined with human blob silhouette for hand detection
16
• Preliminary hand position is obtained from the center of gravity of the biggest skin blobs.
•First, general skin color detection is applied using this mask:
![Page 17: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/17.jpg)
17
Skin color detection combined with human blob silhouette for hand detection
We check if the blob is:
- Below the heep
- Between the heep and the shoulder
- Over the shoulder
![Page 18: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/18.jpg)
18
Skin color detection combined with human blob silhouette for hand detection
•Al the cases where people are wearing short sleeves or “skin color clothes” are correct now.
![Page 19: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/19.jpg)
• Non-Pose classification
19
DEFINITION: Everything different from a predefined Pose
More features Needed!
![Page 20: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/20.jpg)
• Clear Non-Pose: poses where one or both hand positions are in between positions corresponding to Poses
20
Non-Pose classification
![Page 21: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/21.jpg)
21
Non-Pose classification• 17 videos from 17 different people were recorded.• Features extracted from each frame by processing the videos. • Multiple labeling with PRSD Studio, Matlab.
![Page 22: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/22.jpg)
22
Non-Pose classification
Initial Dataset Improved Dataset
![Page 23: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/23.jpg)
23
Non-Pose classification
• Experiments with Initial Dataset
First approach:
Second approach:
![Page 24: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/24.jpg)
• “Leave one person out” method for realistic results.• Tested on Parzen, K-Nearest-Neighbor and Gaussian classifiers.
24
Non-Pose classification: Initial Dataset
ROC curve
![Page 25: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/25.jpg)
• Mean error from LOPO: errors from all people summed and divided by the number of people.
25
•Best results detecting Poses are for K-Nearest-Neighbor, in general.•Parzen and Gaussian are considaribily worse.
Non-Pose classification: Initial Dataset
![Page 26: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/26.jpg)
K-Nearest-Neighbor
26
![Page 27: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/27.jpg)
• Parzen classifier shows more interesting results for a particular person (Hasan).
27
Results for Hasan´s samples as Test, from a single experiment.
•Pose 9 has 404 samples in total.
-120 from Hasan (Test)-171 from Saleem (Training)
Is there any relation?
Non-Pose classification: Initial Dataset
![Page 28: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/28.jpg)
• Two correct ways of performing the same Pose result in quite different features.• Errors in Parzen give us an idea on how to improve even more K-NN performance.
28
Non-Pose classification: Initial Dataset
![Page 29: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/29.jpg)
29
10-NN 5-NN
•All the samples from Carmen are missclassified as Non-Poses.
Non-Pose classification: Initial Dataset
Cascade of detector and classifier
![Page 30: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/30.jpg)
• Results are much better with this approach than with the cascade. Single Pose classes seem to be better modelled than the whole Poses class with K-Nearest-Neighbor.
30
Second approach:
Non-Pose classification: Initial Dataset
![Page 31: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/31.jpg)
31
• Experiments with Improved Dataset
– All classes have more than 100 samples
•For 10-NN the error on Poses decreased 1.5% and the error on Non-Poses decreased 3%.•Having more samples from singles Poses makes the whole Poses class more robust.
Mean errors for the detector trained on Poses.
Non-Pose classification: Improved Dataset
![Page 32: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/32.jpg)
Mean errors for detector trained on Non-Poses.
32
•Training on Non-Poses doesn´t improve detection.
•Non-Poses are more difficult to model than Poses.
Non-Pose classification: Improved Dataset
![Page 33: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/33.jpg)
33
•Now, Carmen´s samples of Pose 3 are correctly detected as Poses.•Pose 3 class is more compact.
Initial Dataset Improved Dataset
Non-Pose classification
![Page 34: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/34.jpg)
34
Non-Pose classification
Decreased from 2% to 0%
Increased from 0% to 1%!!!
•More samples of poses 3 and 4 improved Detection on Poses and Non-Poses detection, but didn´t improve classification of the Pose classes.
![Page 35: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/35.jpg)
Conclusions• The Improved hand detection is a simple method but robust, and solves the problem of
wrong detection for short sleeves.
• Non-Pose class is difficult to model because it overlaps with Poses and it is not a compact class. Anyway, almost 80% of Non-Poses can be detected.
• Having a good dataset might improve results drastically.– Samples must represent different people and ways of performing poses– Samples of wrong hand detections increase the error rate
• The K-Nearest-Neighbor is the best method for modelleing this Pose classes.
• The more restrictive the system is, the better results will be: Comprimise Solution
35
![Page 36: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/36.jpg)
Future Work
• Make a new Dataset with the improved hand detection.
• Add a new feature for detecting more Non-Poses, e.g., face detection.
• Elbow detection.
36
![Page 37: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/37.jpg)
37
I appreciate your attention
Questions?
![Page 38: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/38.jpg)
Initial Dataset
38
![Page 39: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/39.jpg)
Improved Dataset
39
![Page 40: Single person pose recognition and tracking](https://reader033.vdocuments.us/reader033/viewer/2022061220/54858e53b4af9faf0d8b4e5d/html5/thumbnails/40.jpg)
Spatial Game
40