toward a unified scripting language : lessons learned from developing cml and aml

1

Toward a Unified Scripting LanguageToward a Unified Scripting Language

Toward a Unified Scripting Language : Toward a Unified Scripting Language : Lessons Learned from Developing CML Lessons Learned from Developing CML

and AMLand AML

Soft computing LaboratoryYonsei University

October 1, 2004

2


1. Introduction1. Introduction

• Two approaches to specifying scripting languages for character animation– Character Markup Language(CML)

Top-down approach by defining high-level attributes for character personality, emotion and behavior that are integrated to form the specification of synchronized animation script

– Avatar Markup Language(AML)Bottom-up approach in which the language provides a generic

mechanism for the selection and synchronized merging of animations

• Need for powerful yet generic scripting languages to bridge the gap between behavior generation and animation tools

3


• Visual Behavior Definition– A third factor that governs character behavior

The actions an agent needs to perform in a session to achieve given tasks

Personality and current mental stateThe role the agent is given

– The behaviors are defined as XML tags

2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language

CMLProcessor

High-level behavior tags

(map appropriate action point parameters)

Animation Script

4



• Classification of Motion– Derived from Blumberg and Russell’s architecture (three-

layer structure : geometry, motor, behavior system)

– The initial set of CML base motions is classified by the goal of the motion as follows:

Movement defines motion that require the rotation or movement of a character from one position to another. Positions are defined by exact coordinates, an object position, or a character position. (move-to, turn-to)

Pointing defines a pointing gesture toward a coordinate, object, or character. (point-to)

Grasping defines motions that require the character to hold, throw, or come in contact with an object or another character (grasp, throw, touch)

Gaze defines the movements related to the head and eyes. (Gaze are gaze, track, blink, look-to, and look-at. )

Gesture includes motions that represent known gestures like hand movement to convey an acknowledgment, a wave, etc. (gesture-at)

5



• CML Specification– Animated character behavior is expressed through the

interpretation of XML Schema structures

– The language containsLow-level tags (specific character gesture representations

defining movements, intensities, and explicit expression)High-level tags (commonly used combinations of these low-

level tags

– Synchronization between the audio and visual modalities is achieved through the use of SMIL ( <seq>, <par>)

6



• CML Representation Language– Head Gesture Taxonomy

Symbolic gestures, Iconic gestures, Deictic gesture

– Hand Gestures TaxonomyPosture, Motion, Orientation, ‘Gestlets’, Fingers

– Body Gestures TaxonomyNatural, Relax, Tense, Iconic, Incline

– Emotions (based on the OCC theory of emotion)Class, Valence, Subject, Target, Intensity, Time-stamp,

Origin

7



• CML Scripting Language– Face Animation Scripting

Head Movement tilt : movement in a slant with often subtle or superficial neck movement

turn : require more profound movement of the neck<turn-to>

<order {0 to n/before/after} /><priority 0 to n/><begin {ss:mmm:/before/after/object} /><end {ss:mmm:/before/after/object} />

<speed {0.n to n.n (unit) /defualt/slow/fast} />

<target {x,y,z/object/character} /><direction (rightside, leftside, frontside, backside) /><degree (n%) />

<repeat {0 to n/dur} /><interrupt {yes/no} />

<transAnimat {head groups} /><transPos {x,y,z/object/character} /><transSpeed default/slow/ intermediate/fast} />

</turn-to>

8



Head Gesture Deictic

Symbolic

Face Movement and GestureDefine the elements and behaviors for specific parts of the face including the Brow, Gaze, and Mouse

<disagree><tilt-to-right>…</tilt-to-right><tilt-to-left>…</tilt-to-left>

</disagree>

9



– Body Animation ScriptingA set of defined body elements which are low level tags

based on MPEG-4 BAPsA set of high-level tags representing body parts which are

grouped from a set of respective low-level tags

Base elements are..Movement (moving, bending, turning)

Gesture defines body postures that include motions representing common Iconic, Symbolic, or Deictic body gestures

Posture (expression) defines a set of high-level tags representing general body gesture

» Natural ,Relax ,Tense, Incline

10


CML Generation Animation Generation


• Generating Script

State and Context

Input

Emotion Signal

Behavior/Action

Speech Text

PDTD

EDTD

BDB

FDB

CML Processor

Action Composition

UtteranceComposition

Synchronisation

CM

L D

ecod

er

CMLScript

FAP

Audio/TTS

FAP

BAP

Audio/TTS

Face

Face&Body

11


3. Avatar Markup Language3. Avatar Markup Language

• Objective : to design and develop a full end-to-end MPEG-4 multimedia framework to support, amongst other features, 3D avatar-based multi-user chat rooms and autonomous synthetic characters

• Three component– A database of basic facial and body animation units(can

extended by third party)– A rendering system capable of merging multiple face and

body animation units and text to speech input in realtime– A high-level scripting language designed to allow animators

to specify which animations to use together with timing, priority, and decay information

12



• AML Specification

– Flexiblity : as many Expression Tracks as required, each containing as many Expressions as required

<AFML><Settings> …… </Settings>…<ExpressionsTrack name=“Track name”>

<Expression><StartTime>mm:ss:mmm</StartTime><ExName>”name”</ExName><Envelope> …… </Envelope>

</Expression></ExpressionTrack>…

</AFML>

<AFML syntax>

13



<BodyAnimationTrack name=“char, name of track”>…

<UserAnimation type=“bap|trk|wrl” filename=“char”><StartTime> mm:ss:mmm|autosync|autoafter</StartTime><Speed>normal|slow|fast</Speed><Intensity>float, 0 to fn </Intensity><Priority>integer, 0 to n</Priority>

</UserAnimation><Emotion>…</Emotion><StandardGesture>… </StandardGesture>

</BodyAnimationTrack>

<ABML’s Body AnimationTrack Syntax>

<BodyAnimationTrack name=“char, name of track”><FacingAction bodypart=“body|headonly”>

…<XCoor>float, target’s X coordinate in meters</XCoor><YCoor>float, target’s Y coordinate in meters</YCoor><ZCoor>float, target’s Z coordinate in meters</ZCoor>

</FacingAction><PointingAction handconfig=“onefinger|open”>…</ PointingAction><WalkingAction mode=“default|run”>…</WalkingAction><ResettingAction>…</ResettingAction>

</BodyAnimationTrack>

<ABML’s Body AnimationTrack Behavior>

14


4. CML and AML Applied4. CML and AML Applied

• CML– Sample CML – Happy Move and Point Script

<cml> <character name=“James” personality=“extravert” role=“psa” gender=“m” base-animation-file=“butler.liv”> <happy intensity=“0.3” decay=“0.5” target=“goal” priority=“1”> <move-to order=“0” priority=“0” speed=“default” object=“product1” /> <sync type=“par” order=“1” priority=“0”> <point-to object=“product1” /> <utterance> </sync> </happy> </character></cml>

15


4. CML and AML Applied4. CML and AML Applied• AML

– Sample AML – Walk and Point<AML face_id=“x” body_id=“y” root_path=“c:\” name=“point to phone”> <FA start_time=“00:00:000”> <TTS output_fap=speech.fap output_wav=speech.wav> <Text>”Let me show you another phone over here”</Text> </TTS> <AFML> <Settings> <Fps>25</Fps> <Duration>00:06:000</Duration> <FADBPath>.\Expressions\</FADBPath> <SpeechPath>.\Speech\</SpeechPath> </Settings> <ExpressionsTrack name=“smile”> <Expresion> <StartTime> 00:00:800 </StartTime> <ExName> smile.ex </ExName> <Envelope> <Point>… </Point> <Point>…</Point> <Point>…</Point> </Envelope> </Expression> </ExpressionTrack> <SpeechTrack> …</SpeechTrack> </AFML> </FA> <BA> <ABML>….</ABML> </BA></AML>

16


5. Discussion and Lessons Learned5. Discussion and Lessons Learned

• A Comparison

AML MPML MURML TVML APML CMLSTEPXTEP

VHML HumanML PAR RRL

objective animationpresatatio

n

verbal andnon-verbal

utterances

presatation

discourse animation animation animationhuman

representation

parameterised

actiondiscourse

XML V V V V V V V V V

Character Definition V V V V V V VAnimation V V V V V V V V V VDialogue Acts V V V VWorld V V V V VActions/Behavior V V V V V V V V V VVoice Controls V V

Inhibiting animation VMerging V V V V V V V V V ?Synchronisation V V V V V V V V V VAdditional Parameters V V V V V V V V VParameterised actions V V V V V V V V VFeedback to application V V V

Extensibility V V V V V V V V VMacro Elements V V V V V V V V V VMicro Elements V V V V V

Emotions V V V V V V V V VPersonality V V V V V V V

Human- like V V V V V V V V V V VNon- Human V V V V V

Face V V V V V V V VBody V V V V V V V V VSpeech V V V V V V V V V V

V

Character Parts/Modules

Specification Element

Format

Animation Control

Specification Granularity

Believability Attributes

Representation LanguagesBoth Scripting and Representation LanguagesScripting Languages

approach

17


5. Discussion and Lessons Learned5. Discussion and Lessons Learned

• Towards a Unified Language– Objectives of a Unified Language

Define a framework to decouple embodied agent animation tools and the underlying affect and planning engines

Establish a formal specification for unified /consistent interpretation

Provide for modular developmentCreate a markup language based on XML (providing

semantic and scripting annotations)– Language Requirements

High LevelUsabilityExtensibilityParameterizedSynchronization SupportConsistencyDomain

18


6. Conclusion6. Conclusion

• There is a trade-off to be made between higher levels of control (high granularity) and the complexity of the resulting language

• Striving toward a unified scripting and representation language may be the catalyst for much needed agreement

toward a unified scripting language : lessons learned from developing cml and aml

Documents