full-body motion analysis for animating expressive, socially-attuned agents elisabetta bevacqua...
TRANSCRIPT
Full-body motion analysis for animating expressive,
socially-attuned agents
Elisabetta Bevacqua Paris8
Ginevra Castellano DIST
Maurizio Mancini Paris8
Chris Peters Paris8
People involved
• DIST
- full-body movement and gesture analysis
• Paris8 - Agent processing and behavior
Overview
• Scenario: agent that senses, interprets and copies a range of full-body movements from a person in the real world
• System able to - acquire input from a video camera - process information related to the expressivity of
human movement - generate copying behaviours
• Towards a system that recognizes emotions of users from human movement and an expressive agent that shows empathy to them
General framework
• E. Bevacqua, A. Raouzaiou, C. Peters, G. Caridakis, K. Karpouzis, C. Pelachaud, M. Mancini, Multimodal sensing, interpretation and copying of movements by a virtual agent, PIT 2006.
• Encompasses domains of:– Sensing– Interpretation– Planning– Generation
The application
• From human motion to behaviour generation of expressive agents
• Full-body motion analysis of a dancer - real and virtual world
• Agent’s response to expressive human motion descriptors
- quantity of motion - contraction/expansion
• Copying behaviour
Part 1. Sensing and analysis
• Real world Analysis– Computer vision
techniques– Facial analysis– Gesture analysis– Full-body analysis
• Ambition: ‘switchable’ sensing– Real-world and virtual environment– Bridge gap between ECA and embedded virtual agents
Full-body analysis• Expressive cues from human full-body movement
– Real motion– Virtual motion
• Global indicators• EyesWeb Expressive Gesture Processing Library*
– MotionAnalysis: motion trackers (e.g., LK), movement expressive cues (QoM, CI, ...).
– TrajectoryProcessing: processing of 2D (physical or abstract) trajectories (e.g., kinematics, directness, …)
– SpaceAnalysis
*Camurri, A., Mazzarino, B. and Volpe, G., Analysis of Expressive Gesture: The Eyesweb Expressive Gesture Processing Library, in A. Camurri, G.Volpe (Eds.), “Gesture-based Communication in Human-Computer Interaction ”, LNAI 2915, Springer Verlag, 2004.
SMI and Quantity of Motion
tSilhouetteitSilhouettetSMIn
i
0
• Quantity of Motion is an approximation of the amount of detected movement, based on Silhouette Motion Images
QoM = Area(SMI[t, n])/Area(Silhouette[t])
• A measure, ranging from 0 to 1, of how the dancer’s body uses the space surrounding it
• It can be calculated using a technique related to the bounding region, i.e., the minimum rectangle surrounding the dancer’s body: the algorithm compares the area covered by this rectangle with the area currently covered by the silhouette
Contraction Index
Full-body analysis: examples in the real world and in the virtual
environment (I)
• Analysis of quantity of motion and contractionindex with EyesWeb
(G. Castellano, C. Peters, Full-body analysis of real and virtual human motion for animating expressive agents, HUMAINE Presentation, Athens 2006)
• Real world and virtual environment
• Switchable sensing: analysis algorithms capable of - handling input from real-world video stream and from
virtual data - providing similar results
Full-body analysis: examples in the real world and in the virtual
environment (II)
Comparison of metrics: contraction indexContraction Index Real Dancer
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0 100 200 300 400 500 600 700 800
Frames
Contraction Index Virtual Dancer
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0 100 200 300 400 500 600 700 800
Frames
Comparison of metrics: quantity of movement
Quantity of Motion Real Dancer
0,00
0,05
0,10
0,15
0,20
0,25
0 100 200 300 400 500 600 700 800
Frames
Quantity of Motion Virtual Dancer
0,00
0,05
0,10
0,15
0,20
0,25
0 100 200 300 400 500 600 700 800
Frames
Part 2. Interpretation and Behaviour
• Ideal goal: What do we use the expressive cues for?– Planning how to
behave according to users’ quality of gesture
• In this work: Copying dancer’s quality of gesture
Analysis of gesture data
• Full-body analysis of a dancer• Manual segmentation of dancer’s gestures
• Mean value of the quantity of motion and the contraction index of the dancer for each gesture
CI & QoM Copying
• Greta performs one gesture type (same shape) but copies the gesture quality of movement of the dancer
• Greta uses expressivity parameters to modulate the quality of her gestures
• Mapping expressive cues to expressivity parameters:
» CI Spatial extent» QoM Temporal extent
Parameters scaling
Copying: an example
Video of dancer moving and virtual agent performing gestures copying quality of the dancer motion
DEMO!
Facial expressions (1)
• Show emotional facial expressions depending on users’ quality movement
• Study the relation between quality of movement and emotion
• Example: Link QoM and CI to threat:
Facial expressions (2)
• Example: Link QoM and CI to empathy:
Future
• Preliminary work• Validation both for analysis and synthesis
– Perceptive tests to study how users associate an emotional label to an expressive behaviour
• Towards a virtual agent able to recognize users’ emotions from their movement and to show empathy
• Real-time system with continuous input