multimodal plan representation for adaptable bml scheduling
DESCRIPTION
Multimodal Plan Representation for Adaptable BML Scheduling. Dennis Reidsma, Herwin van Welbergen, Job Zwiers. SAIBA. BML. BML behavior and sync points. The BML scheduling process. Scheduling the multimodal behavior plan. The behavior plan. BML - PowerPoint PPT PresentationTRANSCRIPT
Multimodal Plan Representation for Adaptable BML Scheduling
Dennis Reidsma, Herwin van Welbergen, Job Zwiers
SAIBA
BML
BML behavior and sync points
The BML scheduling process
Scheduling the multimodal behavior plan
The behavior plan
BML• Specifies behaviors and the constraints between them
The multimodal behavior plan• Specified on the basis of, e.g., communicative intent• Incrementally constructed from BML blocks
The motor plan• Executes the behavior plan on the embodiment of the
virtual human, using sound, joint rotations, FAPs, … Making the motor plan flexible
• Achieve interpersonal coordination• Allow on the fly adjustments of ongoing behavior• But do not violate the specified constraints
BML constraints
Explicit constraints• Before/after• At
Implicit constraints• Sync points should not occur before the start of their
block• Behaviors should have >0 duration• The default sync points (e.g. start, ready, stroke_start,
stroke, stroke_end, relax, end) must remain in order
BML constraints
Realizer specific constraints• Due to technical limitationor• Theoretically motivated
Block level constraints• Merge: start block at current time• Append: start as soon as possible after all behaviors in
the current plan (but not earlier than current time)
Additional constraints in Elckerlyc
Elckerlyc specific constraints• Whitespace constraint: no unnecessary ‘whitespace’
between behaviors• Append-after(X) block constraint
Flexible plan representation
The (motor) plan• Describes the low level execution of motor behavior on
the embodiment of the virtual human: joint rotations, MPEG4 FAP movement, sound, …etc
To be flexible, the plan representation must• maintain information about the relation between
elements in the plan and the original BML expressions from which they originated
• be capable of expressing all the constraints• be capable of being modified…• …in such a way that the (remaining) constraints
automatically stay satisfied
• Central: the PegBoard• Each sync point is assigned to a TimePeg on this board• Sync points that are connected by an ‘at’ constraint share the
same TimePeg• TimePegs can be moved, changing the time of
the associated sync(s)• TimePegs provide local timing
(irt the start of the BML block)• Each TimePeg is connected to a
BMLBlockPeg• BMLBlockPegs provide global
timing
Elckerlyc’s plan representation
Elckerlyc’s plan representation
• OffsetPeg: links to another TimePeg and moves with it, retaining a time offset
• All ‘at’ constraints can be expressed in Pegs, OffsetPegs en Blockpegs
• After en before constraints require a specific pair of Pegs – Not implemented yet– (but: trivial to design and implement)
Example: solving BML to TimePegs
<bml id="bml1"> <speech id="speech1"> <text>As you can see on <sync
id="s1"> this painting, ... </text> </speech>
<gesture id="point1" start="walk1:relax" type="POINT"
target="painting1" stroke="speech1:s1+0.5"/>
<head id="nod1" action="ROTATION" rotation="NOD" stroke="speech1:s1"/>
<locomotion id="walk1" target="painting1"/>
</bml>
Managing block timing
Architecture
• Separated parsing, scheduling, execution
• Central: the PegBoard
• Engines handle unimodal motor plans
• Sync points of PlanUnits are connected by TimePegs
• Anticipators may make time adjustments through TimePegs
Architecture
• The scheduler communicates with a set of engines to set up the multimodal behavior plan
• It knows for each BML behavior what Engine handles it• Engines have a standardized interface with functions to:
– Add a BML behavior to the motor plan– Remove a BML behavior from the motor plan– Resolve unknown time constraints on a BML behavior given its
known time constraints– Check which BML behavior in the motor plan are currently invalid
• Note that communication with Engines is in terms of BML behaviors
Managing adjustments of the plan
Interruption of a behavior• Will automatically shorten the block’s end• Satisfies block append constraint
Time adjustment of a sync point• Through its associated TimePeg• Will also move syncs that are connected to it with ‘at’
constraints• Might invalidate other constraints
– Each Engine has functionality to check for this– Solution: drop behavior, drop constraint, ..– And communicate back to the behavior planner
Conclusion
Conclusion• BML scheduling can be viewed as a constraint problem• Elckerlyc uses this view to maintain a flexible behavior
plan representation– Allows micro (timing) adjustments– But maintains constraints
• Enables applications that require tight mutual coordination between a user and a virtual human
• These capabilities have been used in: Attentive Speaker, recent version of Virtual Trainer
Thanks for your attention
Further work
Further work• Capture before/after constraints• Multithreaded scheduling