Recognition, Analysis and Synthesis of Gesture

Expressivity

George Caridakis

IVML-ICCS

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overview

Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overview

Video sequences varying in activation-evaluation

Head and hand tracking

Hand Coordinates/Arm configuration

Synthetic expressive gesture reconstruction

Gesture expressivity

features extractionHMM classifier

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus mint-IVML

• 7 subjects

• 7 gesture classes

• 20 gesture variations (3 quadrants)

• 20’ minutes – 30000 frames

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus EmoTV

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus GEMEP (on going…)

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Head detection

Detect candidate facial areas Validate using skin probability Conclude on number of persons

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Hand Detection

•Skin probability

•Thresholding & Morphology Operations

•Distance Transform

•Frame difference

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Tracking

Scoring system based on:• Skin region size

• Distance wrt the previous position

• Optical flow alignment

• Spatial constraints

Thresholding scores Periodical re-initialization

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Head & Hand Tracking

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

HMM parameters for gestures

States are head and hands coordinates• XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL

6 output states Bakis left-to-right models Continuous output distribution

• 3 Gaussian mixtures

Arbitrary training initial estimation of transition probabilities

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Recognition via HMM (Why HMMs?)

Stochastic models fit the nature of the gestures

Fast convergence due to effective training algorithms

Sufficient modeling of the temporal aspect of gestures

Continuous HMMs suitable for gesture-level classification

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

HMM overview

λ2=(A 2, B 2, π )

λ1=(A 1, B 1, π )

λM =(A M , B M , π )

Select Maxim un

Feature VectorSequence υ*=a rgm ax[P (O |λ υ)]

υ

P(O | λ1 )

P(O | λ2 )

P(O | λM )

.

.

.

.

.

.

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Recognition via HMM

0 10 20 30 40 50 60 70 80 90-300

-250

-200

-150

-100

-50

0

50

Xf - X

lh

Xf - X

rh

Xlh

- Xrh

0 10 20 30 40 50 60 70 80 90-400

-300

-200

-100

0

100

200

Yf - Y

lh

Yf - Y

rh

Ylh

- Yrh

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Results

Gesture Attention Clapping ExplainOne

hand Oh my

god Wave Go away

Unclassified

Attention 90.9 0 0 0 0 0 9.1 0

Clapping 0 75 25 0 0 0 0 0

Explain 0 0 60 0 0 0 0 40

One hand 0 0 0 83.3 0 16.7 0 0

Oh my god 0 0 8.3 0 83.3 0 08.3

Wave 25 0 0 0 0 62.5 12.5 0

Go away 25 0 0 0 0 0 75 0

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Expressivity features analysis

Overall activation Spatial extent Temporal Fluidity Power/Energy Repetitivity

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overall activation

Considered as the quantity of movement during a conversational turn

Computed as the sum of the motion vectors’ norm

n

i

ilirOA0

|)(||)(|

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Spatial extent

Modeled by expanding or condensing the entire space in front of the agent that is used for gesturing

Calculated as the maximum Euclidean distance of the position of the two hands

The average spatial extent is also calculated for normalization reasons

max(| ( ( ) ( )) |)SE d r i l i

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Temporal

The temporal parameter of the gesture determines the speed of the arm movement of a gesture’s meaning carrying stroke phase and also signifies the duration of movements (e.g., quick versus sustained actions)

OATE

duration

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Fluidity

Differentiates smooth/graceful from sudden/jerky ones. This concept seeks to capture the continuity between movements, the arms’ trajectory paths as well as the acceleration and deceleration of the limbs

To extract this feature from the input image sequences we calculate the sum of the variance of the norms of the motion vectors

0

var(| ( ) |) var(| ( ) |)n

i

FL r i l i

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Power/Energy

The power is actually identical with the first derivative of the motion vectors calculated in the first steps

| ( ) | | ( ) |d r x d l xPO

dx dx

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Results of expressivity analysis

EF variation Overall Activation

Spatial Extent

Temporal Fluidity Power/Energy

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Expressive synthesis

A system that mimics user’s behaviour through the analysis of facial and gesture signals and expressivity

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Synthesis

Greta Platform BAP calculation

• Plane assumption

• Inverse kinematics

• Manual adaptation

Expressivity features variations implemented in Greta’s BAP interpolation

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Synthesis Results