real-time animation of realistic virtual humans. 1. the 3d virtual player is controlled by the real...
TRANSCRIPT
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
1. The 3D virtual player is controlled by the real 1. The 3D virtual player is controlled by the real
people who has a HMD and many sensors people who has a HMD and many sensors
2. Unlike in video games, animation is not predefined.2. Unlike in video games, animation is not predefined.
3. Unlike in rending films, animation is in real-time.3. Unlike in rending films, animation is in real-time.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
To achieve a real-time virtual humans application we To achieve a real-time virtual humans application we
need to consider:need to consider:
1. Modeling people.1. Modeling people.
2.2. How to present deformation of human body. How to present deformation of human body.
3.3. Motion control.Motion control.
Real-Time Animation of Realistic Real-Time Animation of Realistic Virtual HumansVirtual Humans
1.Body Creation and Skeleton Animation 1.Body Creation and Skeleton Animation -Zhou Bin-Zhou Bin
2.Facial Deformation 2.Facial Deformation -Hu Yi-Hu Yi
3. Body Deformation and Animation Framework3. Body Deformation and Animation Framework
--Yang YufeiYang Yufei
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Body Creation:Body Creation:
First layer: Skeleton hierarchy.First layer: Skeleton hierarchy.
Second layer: Metaballs attached on the skeleton.Second layer: Metaballs attached on the skeleton.
Third layer: Convert metaballs to mesh surface (skin mesh).Third layer: Convert metaballs to mesh surface (skin mesh).
Fourth layer: Texture.Fourth layer: Texture.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
First layer: Skeleton hierarchyFirst layer: Skeleton hierarchy
1. Skeleton articulated just similar to real human.1. Skeleton articulated just similar to real human.
2. We can define all the human postures using this skeleton.2. We can define all the human postures using this skeleton.
3. Relative limb sizes of the skeleton decides the figure of the 3. Relative limb sizes of the skeleton decides the figure of the
virtual people. virtual people.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Second layer: Metaballs attached on the skeletonSecond layer: Metaballs attached on the skeleton
Construct human body by column.Construct human body by column.
Can’t avoid gap on human body.Can’t avoid gap on human body.
Construct human body by column Construct human body by column
and sphere joint. Can avoid gap butand sphere joint. Can avoid gap but
can’t present human body’s smoothcan’t present human body’s smooth
and gradual shape.and gradual shape.
Construct human body by metaballs.Construct human body by metaballs.
Metaballs includes spheres and ellipMetaballs includes spheres and ellipsoids. Left figure is constructed by tsoids. Left figure is constructed by two ellipsoids. wo ellipsoids.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Second layer: Metaballs attached on the skeletonSecond layer: Metaballs attached on the skeleton
1.These two human models are assembled1.These two human models are assembled
by Metaballs (ellipsoid components). by Metaballs (ellipsoid components).
2. Metaballs are smooth and gradual, so 2. Metaballs are smooth and gradual, so
they are suitable for modeling human bodiesthey are suitable for modeling human bodies
3. Different color depends on the attributes of 3. Different color depends on the attributes of
that metaball: blendable or unblendable,that metaball: blendable or unblendable,
deformable or nondeformable. deformable or nondeformable.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Third layer: Convert metaballs to mesh surfaces (skin mesh).Third layer: Convert metaballs to mesh surfaces (skin mesh).
1. Convert metaballs to a number of meshes.1. Convert metaballs to a number of meshes.
We can do this job by doing ray tracing, and We can do this job by doing ray tracing, and
sampling the intersection points. sampling the intersection points.
2. These meshes can decide the lighting effect2. These meshes can decide the lighting effect
of human bodies.of human bodies.
3. Texture mapping corresponds to these 3. Texture mapping corresponds to these
mesh surfaces.mesh surfaces.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Fourth layer: Mapping textureFourth layer: Mapping texture
Texturing has low cost of computation but can Texturing has low cost of computation but can
observably improves the quality of the virtualobservably improves the quality of the virtual
object.object.
Real human hand textureReal human hand texture
Textured 3D modelTextured 3D model
3D Bernard Boxton3D Bernard Boxton 3D Bernard Boxton3D Bernard Boxton
without texturewithout texture
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Body Animation:Body Animation:
First layer: Skeleton motion.First layer: Skeleton motion. --Let avatars make postures and actions.--Let avatars make postures and actions.
Second layer: Mesh surface (skin mesh) deformationSecond layer: Mesh surface (skin mesh) deformation ..
--Present deformation of humans’ skin.--Present deformation of humans’ skin.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Skeleton:Skeleton:
1. The hierarchy of skeleton is defined by a set of joints.1. The hierarchy of skeleton is defined by a set of joints.
This set of joints corresponds to the main joints of real humansThis set of joints corresponds to the main joints of real humans
2. Each joint consists of a set of degrees of freedom (DOF). 2. Each joint consists of a set of degrees of freedom (DOF).
DOFs decide ranges of the joint can translate and rotate. DOFs decide ranges of the joint can translate and rotate.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Three methods of skeleton motion control:Three methods of skeleton motion control:
1. The skeleton motion is captured in real time and drives the 1. The skeleton motion is captured in real time and drives the avatar.avatar.
2. The skeleton motion is predefined. It will be activated from a2. The skeleton motion is predefined. It will be activated from a
database as a response to human’s input. E.g. Sony’s Eye Toy.database as a response to human’s input. E.g. Sony’s Eye Toy.
3. The skeleton motion is dynamically calculated. Doesn’t require3. The skeleton motion is dynamically calculated. Doesn’t require
a user’s continual intervention. E.g. complex games, and AIa user’s continual intervention. E.g. complex games, and AI
application. application.
Human Head Modeling Facial Animation
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
• To simulate humans requires real-time visualization and animation.
Human Head Modeling
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
• Scanning
scan the surface of the head and construct a head model
• Use a sculpturing model----Software Sculptor
Use a sculpturing model----Software Sculptor
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Facial Animation
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
• Facial deformation model
• Facial motion control
Facial Deformation Model
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
1. Consider human face as a polygonal mesh
2. Define regions on the mesh 3. Define a control lattice on the region of in
terest 4. Muscle actions are simulated by changing
the control points’ weight 5. A stiffness factor allows the amount of def
ormation for each point
Facial Deformation Model
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Facial Motion Control
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
MuscularAction
An AnimationSequence Real-time
Animation
Coding Composer
The Facial Action Coding System
1. Define basic motion parameters as minimum perceptible actions (MPAs)
open_mouth, close_upper_eyelids, or raise_corner_lip
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The Facial Action Coding System
1. Each MPA has a corresponding set of visible features
e.g. movement of eyebrows, jaw, or mouth
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The Facial Action Coding System
2. Real-time facial animation module uses three different input methods
• Video• Audio or Speech• Predefined Actions
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
I. Video Input
• Recognition and tracking of the facial features is based on color sample identification, edge detection, and other image processing operations. The feature capture and tracking rate is about 20 frames per second (fps)
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Facial FeatureExtraction And
Tracking
Video InputMagnenat-Thalmann
et al. Array of MPAs
II. Audio Input segment audio into phonemes with their durationText input to phonemes phonemes decomposes into MPAs
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
III. Predefined Action an action that has intensity, duration, and a start time e.g. emotion (surprise, anger)
head gestures (nodding, turning) decomposes into MPAs
predefined action of speechinputaudio input
video input
interpolation
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
IV. Synchronization
go to the composer
V. Composition
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
defined a sinusoid
weight function
considered 10 percent of
the action’s total duration
no jump effect
with the same MPA overlap
Body deformations• The ways we do the representing humans
– Polygonal representation– Visual accuracy representation– The combination of previous two
• Implement results
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Animation framework
Two case studies
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The first method
• The skin wrapped around the skeleton is represented with a fixed mesh divided at important joints where deformations occur.
Polygonal representation
• Pros– Simple and easy to
implement
• Cons– The virtual human
appears “rigid” and lacks realism
– Visually distracting artifacts may arise at joints
• The combined method
• Visual accuracy representation
Body deformation
Image from the internet
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The second method
• The application compute the skin from implicit primitives and use a physical model to deform the body’s envelope.
• Pros– Stress visual accuracy and
yields very satisfactory results in terms of realism
• Cons– So computationally
demanding – Unsuitable for real-time
applications
• Polygonal representation
• The combined method
Visual accuracy representation
Body deformation
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The third method
• It’s a combination of previous two methods, allowing a good trade-off between realism and rendering speed.
• Steps– Constructing a body mesh– Deforming by manipulating skin contours
• Polygonal representation
The combined method
• Visual accuracy representation
Body deformation
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The combined method--- step one
• Output the body data as cross-sectional contours.
• Convert contours data to triangle meshes.– Easy to render, performs better– Construct a triangle strip by connecting
the points from two adjacent cross-section
• Connecting two different body parts proves a bit more complicated.
The combined method Constructing the body mesh
• Manipulating skin contours
Body deformation
Image from: Daniel Thalmann, Jianhua Shen, Eric Chauvineau, 1996
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The combined method--- step two
• Transform a complicated 3D operation into a 2D operation by manipulate the cross-section contours.
• By setting the orientation and position of the plane in which every contour lies, we can achieve a smooth deformation of the skin.
The combined method
• Constructing the body mesh Manipulating skin contours
Body deformation
Image from: Daniel Thalmann, Jianhua Shen, Eric Chauvineau, 1996
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The combined method--- step two (cont.)
• Very joint lies in the plane of its contour when the skeleton is in the at-rest posture.
• Two segments of the arm whose directions are L1 and L2.• Nu, N0 and Nl are the normal vectors of the cross-section planes.• Oi and Ni are the center and normal respectively of the ith cross-section plane.• Since we know Ni by
interpolation, we can
compute each vertex
belonging to the ith contour.
The combined method
• Constructing the body mesh Manipulating skin contours
Body deformation
Image from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
The combined method --- step two (conclusion)
• The contours run parallel at the region above the elbow.• This is because our eyes naturally go to the areas
surrounding major joints like the elbows or knees.• Practically we determined the number of upper and lower
contours to deform in a heuristic fashion.• It saves the rendering time and has little degradation.
The combined method
• Constructing the body mesh Manipulating skin contours
Body deformation
Image from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Results• Using real-time-oriented 3D graphics
toolkit called Performer.
• In right image: the virtual human on the left is made up of 14,000 vertices and containing 13,500 textured triangles using deformation; on the right uses rigid meshes with 17,000 triangles.
Image and figure from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Animation framework
• The close link between modeling and animation.• The system separates into three units: modeling,
deformation and motion control.– Modeling provides geometrical models for the body, hands and
face.– Deformations are performed separately on different entities
based on the model used for each part.– Motion control generates and controls the movements for
different entities.
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Animation framework
Figure from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Case studies--- CyberTennis
Image from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998
Real-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual HumansReal-Time Animation of Realistic Virtual Humans
Case studies--- CyberDance
Image from:Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, and Gael Sannier, 1998