![Page 1: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/1.jpg)
Learning Human Pose and Motion Models for Animation
Aaron HertzmannUniversity of Toronto
![Page 2: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/2.jpg)
Animation is maturing …
… but it’s still hard to create
![Page 3: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/3.jpg)
Keyframe animation
![Page 4: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/4.jpg)
Keyframe animation
http://www.cadtutor.net/dd/bryce/anim/anim.html
q1 q2 q3
q(t) q(t)
![Page 5: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/5.jpg)
Characters are very complex
Woody:- 200 facial controls- 700 controls in his body
http://www.pbs.org/wgbh/nova/specialfx2/mcqueen.html
![Page 6: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/6.jpg)
Motion capture
[Images from NYU and UW]
![Page 7: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/7.jpg)
Motion capture
![Page 8: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/8.jpg)
Mocap is not a panacea
![Page 9: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/9.jpg)
Goal: model human motion
What motions are likely?
Applications:• Computer animation• Computer vision
![Page 10: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/10.jpg)
Related work: physical models
•Accurate, in principle•Too complex to work with
(but see [Liu, Hertzmann, Popović 2005])
•Computationally expensive
![Page 11: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/11.jpg)
Related work: motion graphs
Input: raw motion capture
“Motion graph”(slide from J. Lee)
![Page 12: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/12.jpg)
Approach: statistical models of motions
Learn a PDF over motions, and synthesize from this PDF [Brand and Hertzmann 1999]
What PDF do we use?
![Page 13: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/13.jpg)
Style-Based Inverse Kinematics
with: Keith Grochow, Steve Martin, Zoran Popović
![Page 14: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/14.jpg)
Motivation
![Page 15: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/15.jpg)
Body parameterization
Pose at time t: qt
Root pos./orientation (6 DOFs)
Joint angles (29 DOFs)
MotionX = [q1, …, qT]
![Page 16: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/16.jpg)
Forward kinematics
Pose to 3D positions:
qt
[xi,yi,zi]t
FK
![Page 17: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/17.jpg)
Problem Statement
Generate a character pose based on a chosen style subject to constraints
Constraints
Degrees of freedom (DOFs) q
![Page 18: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/18.jpg)
Real-time Pose Synthesis
Off-Line Learning
Approach
Motion Learning Style
Synthesis
Pose
Constraints
![Page 19: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/19.jpg)
y(q) = q orientation(q) velocity(q) [ q0 q1 q2 …… r0 r1 r2 v0 v1 v2 … ]
Features
![Page 20: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/20.jpg)
Goals for the PDF
• Learn PDF from any data
• Smooth and descriptive
• Minimal parameter tuning
• Real-time synthesis
![Page 21: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/21.jpg)
Mixtures-of-Gaussians
![Page 22: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/22.jpg)
GPLVM
y1
y2
y3
x1
x2
Latent Space Feature Space
Gaussian Process Latent Variable Model [Lawrence 2004]
GP
-1
x ~ N(0,I)y ~ GP(x; )
Learning: arg max p(X, | Y) = arg max p(Y | X, ) p(X)
![Page 23: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/23.jpg)
Scaled Outputs
Different DOFs have different “importances”
Solution:RBF kernel function k(x,x’)ki(x,x’) = k(x,x’)/wi
2
Equivalently: learn x Wywhere W = diag(w1, w2, … wD)
![Page 24: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/24.jpg)
Precision in Latent Space
2(x)
![Page 25: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/25.jpg)
);(ln2);(2
);();(L 2
2
2
IK θxθx
θ)f(xyWθyx, θ
D
SGPLVM Objective Function
y1
y2
y3
x1
x2 θ)f(x;y
xx
C
![Page 26: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/26.jpg)
Baseball Pitch
![Page 27: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/27.jpg)
Track Start
![Page 28: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/28.jpg)
Jump Shot
![Page 29: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/29.jpg)
Style interpolation
Given two styles 1 and 2, can we “interpolate” them?
));(exp()(1 1θyy IKLp
Approach: interpolate in log-domain
));(exp()(2 2θyy IKLp
![Page 30: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/30.jpg)
Style interpolation
));(exp()( 22 θyy IKLp ));(exp()(1 1θyy IKLp
(1-s) s
)(s)()s1( 21 ypyp
![Page 31: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/31.jpg)
Style interpolation in log space
));(exp( 1θyIKL ));(exp( 1θyIKL
(1-s)s
));(s);()s1((exp( 21 θyθy LL
![Page 32: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/32.jpg)
Interactive Posing
![Page 33: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/33.jpg)
Interactive Posing
![Page 34: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/34.jpg)
Interactive Posing
![Page 35: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/35.jpg)
Multiple motion style
![Page 36: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/36.jpg)
Realtime Motion Capture
![Page 37: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/37.jpg)
Style Interpolation
![Page 38: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/38.jpg)
Trajectory Keyframing
![Page 39: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/39.jpg)
Posing from an Image
![Page 40: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/40.jpg)
Modeling motion
GPLVM doesn’t model motions• Velocity features are a hack How do we model and learn dynamics?
![Page 41: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/41.jpg)
Gaussian Process Dynamical Models
with: David Fleet, Jack Wang
![Page 42: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/42.jpg)
Dynamical models
xt+1xt
![Page 43: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/43.jpg)
Hidden Markov Model (HMM)Linear Dynamical Systems (LDS)
[van Overschee et al ‘94; Doretto et al ‘01]
Switching LDS[Ghahramani and Hinton ’98; Pavlovic et al ‘00; Li et al ‘02]
Nonlinear Dynamical Systems[e.g., Ghahramani and Roweis ‘00]
Dynamical models
![Page 44: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/44.jpg)
Gaussian Process Dynamical Model (GPDM)
Marginalize out , and then optimize the latent positions to simultaneously minimize pose reconstruction error and (dynamic) prediction error on training data .
pose reconstruction
latent dynamics
Latent dynamical model:
Assume IID Gaussian noise, and
with Gaussian priors on and
![Page 45: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/45.jpg)
DynamicsThe latent dynamic process on
has a similar form:
where
is a kernel matrix defined by kernel function
with hyperparameters
![Page 46: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/46.jpg)
Subspace dynamical model:
Markov Property
Remark: Conditioned on , the dynamical model is 1st-order Markov, but the marginalization introduces longer temporal dependence.
![Page 47: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/47.jpg)
Learning
To estimate the latent coordinates & kernel parameters we minimize
with respect to and .
GPDM posterior:
reconstruction likelihood
priorsdynamics likelihood
training motions
hyperparameterslatent trajectories
![Page 48: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/48.jpg)
Motion Capture Data
~2.5 gait cycles (157 frames) Learned latent coordinates (1st-order prediction, RBF kernel)
56 joint angles + 3 global translational velocity + 3 global orientation from CMU motion capture database
![Page 49: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/49.jpg)
3D GPLVM Latent Coordinates
large “jumps’ in latent space
![Page 50: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/50.jpg)
Reconstruction Variance
Volume visualization of .
(1st-order prediction, RBF kernel)
![Page 51: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/51.jpg)
Motion Simulation
Animation of mean motion (200 step sequence)
initial state
Random trajectories from MCMC (~1 gait cycle, 60 steps)
![Page 52: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/52.jpg)
Simulation: 1st-Order Mean Prediction
Red: 200 steps of mean prediction
Green: 60-step MCMC mean
Animation
![Page 53: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/53.jpg)
Missing Data
50 of 147 frames dropped (almost a full gait cycle)
spline interpolation
![Page 54: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/54.jpg)
Missing Data: RBF Dynamics
![Page 55: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/55.jpg)
Determining hyperparameters
GPDM Neil’s parameters MCEM
Data: six distinct walkers
![Page 56: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/56.jpg)
Where do we go from here?
Let’s look at some limitations of the model
60 Hz 120 Hz
![Page 57: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/57.jpg)
What do we want?
Phase
Variation
x1
x2
A walk cycle
![Page 58: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/58.jpg)
Branching motions
Walk Run
![Page 59: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/59.jpg)
Stylistic variation
![Page 60: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/60.jpg)
Current work: manifold GPs
Latent space (x) Data space (y)
![Page 61: Learning Human Pose and Motion Models for Animation](https://reader035.vdocuments.us/reader035/viewer/2022081515/568145ef550346895db2f664/html5/thumbnails/61.jpg)
Summary
GPLVM and GPDM provide priors from small data sets
Dependence on initialization, hyperpriors, latent dimensionality
Open problems modeling data topology and stylistic variation