agents and avatars - hwruth/year4ves/slides11/l15.pdf · agents and avatars ... mocap process...
TRANSCRIPT
![Page 1: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/1.jpg)
Agents and Avatars
Ruth Aylett
![Page 2: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/2.jpg)
Overview
Agents and Avatars Believability v naturalism Building the body H-anim Moving and sensing
![Page 3: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/3.jpg)
IVAS
Intelligent Virtual Agents (IVAs)– Also:
• Synthetic characters• Embodied conversational characters (ECAs)• Virtual humans• BUT do not have to be humanoid…
– Embodied and autonomous• Require a control architecture - or ‘agent mind’• Insides v outsides: combining AI with graphics
– ‘Inhabit’ a virtual environment
![Page 4: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/4.jpg)
Why IVAs?
Adding life to a VE– Animals, birds, insects– Crowds
• E.g. students in the virtual campus– Increase sense of presence
As a guide or teacher– Front-end to embedded knowledge in a VE
As a character in a story– Computer games
Interface agents– On web pages to make them more human– Personal representative– Sales rep
![Page 5: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/5.jpg)
Why IVAs? - 2
As virtual actors– Instead of extras– Immersive Education Media Stage
As part of a simulation– Hostage release training– Battlefield medical training
Scientific investigation– Buildings evacuation– Testing for disability friendliness– Ecology and animal behaviour
![Page 6: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/6.jpg)
Avatars
Hindu Sanskrit Term - “Representation of aDeity in visible form”– Snow Crash (Neil Stephenson 1992) - “A
graphical representation of yourself”
VE representation of the user– Embodiment need not be humanoid
Driven by the user– So NOT autonomous– A mapping rather than a control problem
![Page 7: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/7.jpg)
Video avatars
![Page 8: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/8.jpg)
“magic video mirror”with back-projection
video image mixedwith computergeneratedoverlay
Creating a video avatar (ALIVE, MIT)
•unencumbered interactionbetween human visitor andvirtual character based onposition, postures, and gestures
•main focus on “virtual presence”
http://www.ai.univie.ac.at/oefai/agents/
![Page 9: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/9.jpg)
Believability
Term introduced by Joe Bates of the OZgroup at CMU in the 1990s– Combined art and technology
Very hard to define– A willing suspension of disbelief?– Seem like ‘real’ characters?– The ‘illusion of life’?– Willingness to attribute an internal state
• Ascribe intentionality
![Page 10: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/10.jpg)
Believability and Naturalism
Are they the same thing? Graphics people seem to think so
– Is Mickey a real mouse?– Is he a believable character?
![Page 11: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/11.jpg)
The uncanny valley
The ‘uncanny valley’– Work by a Japanese
researcher, Mori– Acceptability v
naturalism
Goes -ve asbecomes ‘nearlyhuman’
SEE: http://en.wikipedia.org/wiki/Uncanny_Valley
![Page 12: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/12.jpg)
![Page 13: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/13.jpg)
The problem of expectations
Humans have hard-wired expectations– Used to interpret inter-personal behaviour– Fundamental social skill
Need to invoke this very carefully– Acceptability of movement
• Lip sync a key problem
– Interactive responsiveness• Memory of interaction a key issue• Games create a real problem with instant rewind
![Page 14: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/14.jpg)
Building a body
Use of 3D modelling package– 3ds Character Studio; Poser etc
Creation of skeleton– Required for animation later
Polygonal body– But overall count must be low for real-time
interaction, low 000s or less if possible
Texture to cover body– One body, many textures?
![Page 15: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/15.jpg)
![Page 16: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/16.jpg)
TextureImages
X 4
SilhouetteImages
X 4
Stage 1: Image Capture
![Page 17: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/17.jpg)
•Generic Avatar
•Polygonal Structure
•1500 Polygons
Change Data:Deforms the Mesh
Stage 2: Deform the Mesh
![Page 18: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/18.jpg)
Apply Texturesto Deformed
Mesh
Stage 3: Apply the Textures
![Page 19: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/19.jpg)
The touch up process
![Page 20: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/20.jpg)
What sort of motion?
Walking around– Jumping, running, swimming– Human and non-human
Picking things up– Agent-agent interaction
Gesture Talking heads
– Facial muscles, lip synch
Group Movement– Crowds, flocks, herds
![Page 21: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/21.jpg)
Moving the body
Animation– Drawing on cartoon animation
Motion capture– Drawing on gait analysis and then film
PBM– Drawing on robotics
• Analytically calculated• Learned
![Page 22: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/22.jpg)
Animation
Time consuming for good results– Requires artistic skills
The main character, Woody, in ToyStory had:– 700 degrees of freedom (200 for face and
fifty for the mouth)– 150 people (at Pixar) would generate 3
minutes of animation a week Can it be standardised?
![Page 23: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/23.jpg)
Self-animation
Poses new problems:– Parametrisable animation very desirable
• E.g a walk that could be reused with differentstride length or foot height
– Melding and combining of animations ‘onthe fly’• Not just a morphing problem• ‘starting position’?
– Extra actions inserted by character
![Page 24: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/24.jpg)
Standardising animation?
Animation depends on the structure ofskeleton that is being animated
Can we standardise a humanoidskeleton?– H-anim just such an attempt– Originally in the context of VRML– See www.h-anim.org
![Page 25: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/25.jpg)
H-anim
A set of standard components– Humanoid: root of a figure– Joint: attached using transform specifying
current state of articulation plus geometryassociated with attached body part
– Segment: specifies attributes of physical linksbetween joints
– Site: where can add semantics– Displacer:range of movement allowed for object
in which embedded
![Page 26: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/26.jpg)
H-anim
![Page 27: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/27.jpg)
And in X3d..
<HAnimJoint DEF='hanim_l_hip' center='0.0961 0.9124 -0.0001'name='l_hip'>
<HAnimSegment DEF='hanim_l_thigh' name='l_thigh'/> <HAnimJoint DEF='hanim_l_knee' center='0.1040 0.4867 0.0308' name='l_knee'> <HAnimSegment DEF='hanim_l_calf' name='l_calf'/> <HAnimJoint DEF='hanim_l_ankle' center='0.1101 0.0656 - 0.0736' name='l_ankle'> <HAnimSegment DEF='hanim_l_hindfoot’ name='l_hindfoot'/> </HAnimJoint> </HAnimJoint></HAnimJoint>
![Page 28: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/28.jpg)
Motion capture
Electro-magnetic or by camera Produces the most ‘natural’ results
– Extensively used in film for graphical extras– Can be used on avatars very successfully to transmit user
movement to their graphical representation
Even more problems for self-animation– Harder to parametrise– Worse combining/melding problems
![Page 29: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/29.jpg)
Mocap process
Calibration Capture 3D Position Reconstruction Fitting to the Skeleton Post Processing
![Page 30: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/30.jpg)
Calibration
Triangulate camera position and set origin
Wand– Calibrate and triangulate
cameras and capturevolume
L-frame– Define ground plane, up
vector and origin
![Page 31: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/31.jpg)
θ1
θ2
Calibration
![Page 32: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/32.jpg)
3D Position Reconstruction(Utopia)
![Page 33: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/33.jpg)
3D Position Reconstruction(Reality)
![Page 34: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/34.jpg)
Multiple Hypothesis Tracking
For small number of markers: Size Occlusions are a problem Multiple Hypothesis Tracking
Ringer, et al., 2002
C1
C2
![Page 35: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/35.jpg)
Fitting to the Skeleton
Utopian approach– 10 – 20% length
changes
Markers on bothsides
Joint Displacement Use Rotation Angles
Only
![Page 36: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/36.jpg)
Post Processing
Motion Editing– Cut, Copy, Paste
Motion Warping– Speed up or Slow Down– Rotate, Scale or Translate
Motion Signal Processing– Smoother Motions
![Page 37: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/37.jpg)
Physically-based modelling
Extending robot motor control Forward kinematics
– If you move joints so much– Where does the end effector go?
Inverse kinematics– If you want the end effector at x,y,z…– How much should which joints move?
Computationally demanding and often not verynaturalistic– But the most flexible option– Animation blending often added…
![Page 38: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/38.jpg)
Ragdoll models
Multiple rigid body simulation– Each tied to bone in skeleton– Constraints for joint movement
Extended into procedural animation ofwhole body– Natural Motion: Euphoria
![Page 39: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/39.jpg)
Fish example - spring-massmodel
![Page 40: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/40.jpg)
Learning to move
Use of AI learning algorithms– Move– Evaluate and score– Keep movements which work well
Karl Sims ‘Blockies’– Early 1990s
![Page 41: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/41.jpg)
Terzoplolous - Fish Learning
http://www.csri.utoronto.ca/~dt/
![Page 42: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/42.jpg)
Virtual sensors
Local v global interaction– Global - read from world data-structures– Local - ‘sense’ the environment
Global approach very common– In most computer games– Efficient, easy to test
![Page 43: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/43.jpg)
Advantages of local sensing
Scales well– Not affected by global size of environment
Agent has independence of environment– Up to a point
Makes for believability– Agent perceives what it should– Can’t see you round corners– Emergent complexity
![Page 44: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/44.jpg)
TeleTubby Sensors
Forward ray-tracing sensor isseven meters and sweeps 45o,five times/sec
One vector sensor directed vertically downwards. Intersectionwith the ground is continually beingdetected
All these sensors are attached tothe geometry of the agent
![Page 45: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/45.jpg)
Sensing by message passing
Requires an architecture that distributesevents as messages– Concept of locale
• What is local for local sensing?
– Semantically determined: e.g a room
Scenegraphs not set up for this style ofmessage passing– Think of routing in VRML for example
![Page 46: Agents and Avatars - HWruth/year4VEs/Slides11/L15.pdf · Agents and Avatars ... Mocap process Calibration Capture 3D Position Reconstruction](https://reader033.vdocuments.us/reader033/viewer/2022051405/5a95a75d7f8b9a18628cc370/html5/thumbnails/46.jpg)
Credits
Edward Tse, University of Calgary