real-time classification of dance gestures from skeleton animation michalis raptis, ucla csd darko...
TRANSCRIPT
Real-Time Classification of Dance Gestures from Skeleton
Animation
Michalis Raptis, UCLA CSDDarko Kirovski, Hugues Hoppe,
Wei-ge Chen, Microsoft ResearchJoel Pritchett, Tom Fuller, Chuck Noble,
XBOX Microsoft Game Studios
Absolute motion Relative motion
1
1 11
1 1
1
1
1
Ora
cles
Oracle vs. Random Dancer
Head
NeckLeft
shoulder
Right shoulder
Right elbow
Left elbow
Spine
Tail
Left hipRight hip
Right knee
Left knee
Left foot
Right foot
(0,0,0)
Left hand
Right hand
TORSO
1st degree joint
2nd degree joint
TORSO
ur t
torso PCA
LS
t
u
r
q
jLEp
R
LHP
LErP q
j
b
R
S
0 20 40 60 80 100 1200
2
4Timing analysis - Elbows
left, y
right, y
0 20 40 60 80 100 1200
0.5
1Timing analysis - Knees
left, y
right, y
0 20 40 60 80 100 1200
2
4Timing analysis - Hands
left, y
right, y
0 20 40 60 80 100 1200
2
4Timing analysis - Feet
left, y
right, y
10 20 30 40 50 60 70 80 90 100 110 120-pi/2
0
pi/2Torso angles
face
ski turn
ski downhill
0 20 40 60 80 100 1201
2
3Timing analysis - Elbows
left, q
right, q
0 20 40 60 80 100 1201
2
3Timing analysis - Knees
left, q
right, q
0 20 40 60 80 100 1200
1
2Timing analysis - Hands
left, q
right, q
0 20 40 60 80 100 1200
1
2Timing analysis - Feet
left, q
right, q
0 50 100 150-0.4
-0.2
0
0.2
0.4
0 50 100 150-0.4
-0.2
0
0.2
0.4
0 50 100 150-0.5
0
0.5
1
0 50 100 1501
1.5
2
2.5
0 50 100 1501
1.5
2
2.5
3
0 50 100 1501
1.5
2
2.5
0 50 100 1501.5
2
2.5
3
0 50 100 1501
1.5
2
2.5
0 50 100 1501
1.5
2
2.5
3
0 50 100 1501
1.5
2
2.5
3
0 50 100 1501
1.5
2
2.5
3
0 50 100 1500
0.5
1
1.5
2
0 50 100 1500
1
2
3
0 50 100 1500
0.5
1
1.5
0 50 100 1500
1
2
3
0 50 100 1500
0.5
1
1.5
0 50 100 1500
1
2
3
0 50 100 1500
0.5
1
1.5
0 50 100 1501
1.5
2
2.5
3
OraclePlayer meanTo
rso
angl
esEl
bow
sW
rists
Knee
sFe
et
Left Right
Oracle vs. Player Mean
-1 0 10
0.01
0.02
-1 0 10
0.01
0.02
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.02
0.04
-1 0 10
0.01
0.02
-1 0 10
0.01
0.02
-1 0 10
0.01
0.02
-1 0 10
0.01
0.02
-1 0 10
0.02
0.04
-1 0 10
0.01
0.02
-1 0 10
0.02
0.04
-1 0 10
0.01
0.02
True negativeTrue positiveTo
rso
angl
esEl
bow
sW
rists
Knee
sFe
etTowards MLE
Depth sensing
Player performance
Depth image
Skeletal tracking
WireframeFeature
representation
Normalized correlations
Pairwise matching
Matched class
Distance metric
Features
Distance report
What gesture?
How well?
K inect
Gesture model
Prototype mean
Correlation stats
Energy stats
Logistic regression
coefs
Score computationC
lass
ran
kin
g
Ora
cle
Video game asks
Gesture Classification
System
Confusion Matrix
• HMM-based, others 50-60%
Summary
• Features – generic, nearly lossless• Dance aligned to beat – no major time-warp• Detection
– Normalized circular cross correlation via FFT– Maximum Likelihood estimator– Logistic regression to resolve class similarity
• Fast, scalable, extraordinary ROC