emblem detection by tracking facial features · yuchi huang, cbim, rutgers university dimitris...
TRANSCRIPT
![Page 1: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/1.jpg)
Emblem Detection by Emblem Detection by
Tracking Facial FeaturesTracking Facial Features
AtulAtul KanaujiaKanaujia, , CBIM, Rutgers UniversityCBIM, Rutgers University
YuchiYuchi Huang, Huang, CBIM, Rutgers UniversityCBIM, Rutgers University
DimitrisDimitris Metaxas, Metaxas, CBIM, Rutgers UniversityCBIM, Rutgers University
![Page 2: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/2.jpg)
Understanding NonUnderstanding Non--verbal verbal
CommunicationCommunication
►► Emblems Emblems –– An event or movement that symbolizes An event or movement that symbolizes an Ideaan Idea�� Head Nodding, Shaking and Head TiltingHead Nodding, Shaking and Head Tilting
�� “Thieves” use fewer head movements, gestures and “Thieves” use fewer head movements, gestures and more selfmore self--touching.touching.
Interviewing Criminal Investigations
![Page 3: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/3.jpg)
Understanding NonUnderstanding Non--verbal verbal
CommunicationCommunication
•• Facial Expressions Facial Expressions
Analysis and SynthesisAnalysis and Synthesis
-- 6 Universal Facial 6 Universal Facial
ExpressionsExpressions i.e. Anger, Joy, i.e. Anger, Joy,
Disgust, Surprise, Sadness Disgust, Surprise, Sadness
and Fear.and Fear.
•• Eye GesturesEye Gestures
-- Blinking (Drowsiness) Blinking (Drowsiness)
-- Gaze (Interest in Conversation)Gaze (Interest in Conversation)
![Page 4: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/4.jpg)
Detecting Emblems by Tracking Detecting Emblems by Tracking
Facial FeaturesFacial Features
►► Extend Active Shape Models to Extend Active Shape Models to handle Localized shape handle Localized shape deformation.deformation.�� Expressions and AU detectionExpressions and AU detection
►► Track facial Features across large Track facial Features across large head movement.head movement.
►► Use only 2D Shape tracking.Use only 2D Shape tracking.
►► Demonstrate algorithm on Demonstrate algorithm on Emblem Detection Emblem Detection –– Head Head Nodding, Head Shaking, eye Nodding, Head Shaking, eye blinking.blinking.
![Page 5: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/5.jpg)
Why Active Shape Models ?Why Active Shape Models ?
Active Appearance Model + Lesser Land marks needed. + More Robust to textured Images
- Takes Longer to converge - Large influence due to changes in Appearance and Illumination
Active Shape Models+ Larger Capture Range Compared to AAM ( Search along profile) + Faster convergence + Less affected by Illumination variations.
- Does not use grey level information - Needs finely located landmarks - Poor Performance with textured background
Courtesy: Tim Cootes et al. Comparing Active Shape Models with Active Appearance Models
![Page 6: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/6.jpg)
Shape Subspace for Localized Shape Subspace for Localized
DeformationDeformation• Accurate tracking of Facial expressions largely depends on the characteristics of the shape basis vector.• Localized Shape Representation using Local Factor Analysis (LFA), Local ICA, Non-negative Matrix Factorization (NMF).• NMF – Factorizes the observed shapes into Non-Negative Linear combination of basis vectors.
Varying Control Parameters
Holistic PCA basis
Localized NMF basis
![Page 7: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/7.jpg)
NMF Basis VectorNMF Basis Vector
![Page 8: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/8.jpg)
NMF NMF vsvs PCAPCA
(Qualitative Comparison)(Qualitative Comparison)
NMF Basis
PCA Basis
![Page 9: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/9.jpg)
NMF NMF vsvs PCA PCA
(Quantitative Comparison)(Quantitative Comparison)
►► Prediction accuracy on Prediction accuracy on CohnCohn--KanadeKanade database database on Facial Expressions.on Facial Expressions.
�� PCA vectors ranged from PCA vectors ranged from 35 35 –– 45 (captured 98% 45 (captured 98% variance)variance)
�� Trained on specific Trained on specific emotionemotion
�� NMF with 40 basis NMF with 40 basis vectors gave consistently vectors gave consistently better results.better results.
![Page 10: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/10.jpg)
Handling Large Head MovementHandling Large Head Movement
• Aspect changes cause Shapes to lie on a Non-Linear Manifold.• Linear Subspaces cannot model non-linearities. • Use Multiple ASM models to learn shapes from different viewpoints.
![Page 11: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/11.jpg)
Classifying poses using SIFT DescriptorsClassifying poses using SIFT Descriptors
Histogram of Gradients
• Train a Multi-Category Classifier to recognize head pose
- Trained on ~2000 Images, ~400images of each pose.- Images aligned by nose tip.- ~93% accuracy
Locally Normalized Orientation Bins
50x50, Rescaled Image 5x5 Pixel cells, 4x4 block
800 Dimensional Vector
Confusion Matrix
![Page 12: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/12.jpg)
Tracking the FeaturesTracking the Features
►► Active Shape Model search cannot be used at every frame. Active Shape Model search cannot be used at every frame.
►► Tracking using SSID point trackerTracking using SSID point tracker�� Distorts the facial feature shapesDistorts the facial feature shapes
►► Constrain the shape to lie within the NMF subspace at Constrain the shape to lie within the NMF subspace at every frame. every frame.
ConstrainShape
Tracker Results
![Page 13: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/13.jpg)
Tracking Head Rotations(25 FPS)Tracking Head Rotations(25 FPS)
![Page 14: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/14.jpg)
Detecting Head NoddingDetecting Head Nodding
Detect Nodding and Shaking by tracking Y and X co-ordinates of the Nose Tip
![Page 15: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/15.jpg)
Detecting Eye BlinkingDetecting Eye Blinking
Detecting Eye Blinking using Template Matching.
ROC curves obtained by varying the thresholds in the detection algorithm – Head Nodding ROC area – 96.24% – Head Shaking ROC area – 94.7%– Eye Blinking ROC area – 95.46%
![Page 16: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/16.jpg)
Tracking with Glasses and Varying Tracking with Glasses and Varying
IlluminationIllumination
![Page 17: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/17.jpg)
Tracking Facial ExpressionsTracking Facial Expressions
Joy Surprise
![Page 18: Emblem Detection by Tracking Facial Features · Yuchi Huang, CBIM, Rutgers University Dimitris Metaxas, CBIM, Rutgers University. Understanding Non -verbal Communication](https://reader035.vdocuments.us/reader035/viewer/2022070917/5fb79fe6ac200602ec326b99/html5/thumbnails/18.jpg)
Real time Online demonstrationReal time Online demonstration
Day: Wednesday Timings: 8:30 AM to 4:30 PM Location: Kimmel Center, Room 906