Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression

Recognition

Xiang Yu1, Shaoting Zhang1, Yang Yu1, Norah Dunbar2, Matthew Jensen2, Judee K. Burgoon3,

Dimitris N. Metaxas1

1CBIM, Rutgers Univ., NJ2Oklahoma University, Ok 3Univ. of Arizona, AZ

IntroductionOur goal

• Predict whether an interactant is truthful or deceptive by analyzing interactional synchrony

• Propose a computational framework to do so

IntroductionWhat is interactional synchrony?

• Interpersonal communication is contingent upon some form of mutual adaptation such as: – Accommodation– Interpersonal coordination– Matching– Mirroring– Compensation– Divergence– Complementarity

IntroductionWhy investigating interactional synchrony?

• Practitioners have suggested using interactional synchrony for detecting deception:– with terrorists (Turvey, 2008)– in FBI interviews (Navarro, 2003)– and in police investigations (Kassin et al., 2007)

• Assumption: interviews with deceivers are less synchronous than interviews with truth tellers.

• However, few systematic studies of coordination, synchrony or reciprocity have examined the effects of synchrony on deception.

IntroductionRelevant techniques

• Face Tracking - Active Shape Model (Cootes et al. CVIU 1995) - Active Appearance Model (Cootes et al. ECCV 1998) - Constrained Local Model (Cristinacce and Cootes, BMVC 2006)

• Gesture and Expression recognition - Head nodding, shaking, HMM(Kapoor and Picard PUI 2001) - Gesture recognition, CRF (Morency et al. CVPR 2007) - Image based expression recognition (Pantic et al. PAMI 2000) - Video based (Black and Yacoob, IJCV1997; Cohen et al. CVIU 2003)

Proposed MethodSystem framework

Module 1: Robust Facial Tracking

• Robust facial tracking is the foundation of interactional synchrony analysis

• Challenges:– Partial occlusion– Multiple poses– Poor lighting conditions

Active Shape Models• Active Shape Model

Shape vector: Shape matrix: PCA: Covariance matrix: Shape representation:

Active Shape Models

• Using KLT tracker to estimate shape locally.• Optimize shape , not only similar to , but also

follows the shape distribution:

• Alternately optimize and , until they converge.

Limitations

• Partial occlusion:– Bayesian inference [Y. Zhou CVPR’03]– Pictorial structures [P. Felzenszwalb IJCV’05]– Sparse outliers [F. Yang FG’11]

• Multiple poses:– Mixture of Gaussian distribution [T. Cootes IVC’99]– Kernel PCA [S. Romdhani BMVC’07]– Hierarchical multi-state [Y. Tong PR’07]

Face Tracking: Handle Occlusions

• We explicitly model the large errors as a sparse vector :

sparsity number of

Face Tracking: Handle Multi-pose

• Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:

≈...

Face Tracking: Handle Multi-pose

• Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:

sparsity number of sparsity number of

Synthetic data

• Shape optimization from side pose with outliers. Blue line: detection results. Red line: fitting result.

E-ASM Sparse shape registration(Handle gross error)

Our method(Handle gross error

and multi-pose)

Quantitative Comparison

• The point error by pixel

Occlusion by Hat

Extended ASM Sparse Shape Registration Our method

Occlusion by Scarf

Extended ASM Sparse Shape Registration Our method

Multiple Pose

Sparse shape RegistrationOur Method

Multiple Pose with Occlusions

Sparse shape RegistrationOur Method

Face Tracking Results

• Tracking result from CMC dataset

Module 2 and 3: Expression Detection and Head Pose

Demo of the successful detection of events (both are smiling and with the same head pose)

Module 2: Expression Recognition

• Use the relative intensity order of facial expressions to learn a ranking model (RankBoost) for recog. and intensity estimation

• 6 universal facial expressions (i.e., fear, anger, sadness, disgust, happiness, surprise)

• Trained ranking model gives a score for the prob. of smiling in real time.

Expression Recognition(1)

• Facial feature representation.

Expression Recognition(2)

• Ordinal pair-wise data organization.

Expression Recognition(3)

• Ranking Model. - sparse Rankboost (weak classifier) lost function:

- Adaboost strong classifier

Smile Detection

Illustration of synchrony for a smiling event (both the subject and the interviewer are smiling at about the same time instant)

Smile Synchrony

Left: smiling scores curves for both the subject (in blue) and interviewer (in red). Here we see that there is synchrony.Right: Snapshots from the actual tracked footage illustrating smiling synchrony.

Module 3: Head Pose Module: Nodding

• A 3D head nodding sequence. The green lines show head pitch angle. The black lines show head yaw angle.

• Head pose pitch curve along time axis. The pattern of the blue plot is characteristic of head nodding

Head Pose Module: Shaking• A 3D head shaking

sequence. The green lines show head pitch angle. The black lines show head yaw angle.

• head pose yaw curve. The pattern of the red plot is characteristic of a head shake.

Head Pitch Synchrony (nodding)

head pitch curves for the subject (in blue) and the interviewer (in red). The patterns are characteristic of head nodding. Here we see that there is synchrony (both plots show nodding pattern).

Head Yaw Dissynchrony (shaking)

head yaw curves for the subject (in blue) and the interviewer (in red). The distinct pattern of the red plot is characteristic of a head shake. This head shaking event is NOT in synchrony.

Module 4: Synchrony Feature ExtractionCross correlation

• Generate higher level synchrony feature from Lower level features

• Correlation based strategy

• Synchrony definition. – If event A is detected for subject X at

time T and for subject Y at time T ± w we say we have synchrony

• Some attributes can be extracted from detected Synchrony:– 1. Smile_Hit_Subject_Rate_S(x): Rate of

smiling synchrony (led by subject and followed by interviewer) in segment (x)

– 2. Smile_Hit_Interviewer_Rate_S(x): Rate of smiling synchrony (led by interviewer and followed by subject)

Smile_Hit_Subject

Smile

Time

Yes

No

SubjectInterviewer

Smile_Hit_Interviewer

Smile

Time

Yes

No

SubjectInterviewer

Synchrony Feature Extractionhit-miss rate

Synchrony Feature Extractionhit-miss rate

• Some attributes can be extracted from detected Synchrony:– 3. Smile_Miss_Subject_Rate_S(x):

The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)

– 4.Smile_Miss_Interviewer_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)

Smile_Miss_Subject

Smile_Miss_InterviewerSmile

Time

Yes

No

SubjectInterviewer

Smile

Time

Yes

No

SubjectInterviewer

Module 5: Feature Selection and Classification

• Feature Selection, Genetic Algorithm.• Classification Two-class problem:

Truthful vs. Deceptive

Three-class problem: Truthful, Sanctioned and Unsanctioned cheating

Experimental Databases

• Computer-Mediated Communication Dataset (CMC)

• Face to Face Communication Dataset (FtF)

Evaluation of Synchrony feature

Evaluation of Two-class Classification

• Confusion Matrices

• Performance

Evaluation of Three-class Classification

• Confusion Matrices

• Performance

Conclusions

• We investigated how the degree of synchrony effects the result of deception detection.

• Automatic methods provide an important way to evaluate synchrony other than manual coding

• Some observations:– Different lower level features contribute differently

to the deception detection.–Modalities have subtle influence in detecting

deception. (CMC vs. FtF)

Thanks!