real-time classification of dance gestures from skeleton animation michalis raptis, ucla csd darko...

Real-Time Classification of Dance Gestures from Skeleton

Animation

Michalis Raptis, UCLA CSDDarko Kirovski, Hugues Hoppe,

Wei-ge Chen, Microsoft ResearchJoel Pritchett, Tom Fuller, Chuck Noble,

XBOX Microsoft Game Studios

Absolute motion Relative motion

1

1 11

1 1

1

1

1

Ora

cles

Oracle vs. Random Dancer

Head

NeckLeft

shoulder

Right shoulder

Right elbow

Left elbow

Spine

Tail

Left hipRight hip

Right knee

Left knee

Left foot

Right foot

(0,0,0)

Left hand

Right hand

TORSO

1st degree joint

2nd degree joint

TORSO

ur t

torso PCA

LS

t

u

r

q

jLEp

R

LHP

LErP q

j

b

R

S

0 20 40 60 80 100 1200

2

4Timing analysis - Elbows

left, y

right, y

0 20 40 60 80 100 1200

0.5

1Timing analysis - Knees

left, y

right, y

0 20 40 60 80 100 1200

2

4Timing analysis - Hands

left, y

right, y

0 20 40 60 80 100 1200

2

4Timing analysis - Feet

left, y

right, y

10 20 30 40 50 60 70 80 90 100 110 120-pi/2

0

pi/2Torso angles

face

ski turn

ski downhill

0 20 40 60 80 100 1201

2

3Timing analysis - Elbows

left, q

right, q

0 20 40 60 80 100 1201

2

3Timing analysis - Knees

left, q

right, q

0 20 40 60 80 100 1200

1

2Timing analysis - Hands

left, q

right, q

0 20 40 60 80 100 1200

1

2Timing analysis - Feet

left, q

right, q

0 50 100 150-0.4

-0.2

0

0.2

0.4

0 50 100 150-0.4

-0.2

0

0.2

0.4

0 50 100 150-0.5

0

0.5

1

0 50 100 1501

1.5

2

2.5

0 50 100 1501

1.5

2

2.5

3

0 50 100 1501

1.5

2

2.5

0 50 100 1501.5

2

2.5

3

0 50 100 1501

1.5

2

2.5

0 50 100 1501

1.5

2

2.5

3

0 50 100 1501

1.5

2

2.5

3

0 50 100 1501

1.5

2

2.5

3

0 50 100 1500

0.5

1

1.5

2

0 50 100 1500

1

2

3

0 50 100 1500

0.5

1

1.5

0 50 100 1500

1

2

3

0 50 100 1500

0.5

1

1.5

0 50 100 1500

1

2

3

0 50 100 1500

0.5

1

1.5

0 50 100 1501

1.5

2

2.5

3

OraclePlayer meanTo

rso

angl

esEl

bow

sW

rists

Knee

sFe

et

Left Right

Oracle vs. Player Mean

-1 0 10

0.01

0.02

-1 0 10

0.01

0.02

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.02

0.04

-1 0 10

0.01

0.02

-1 0 10

0.01

0.02

-1 0 10

0.01

0.02

-1 0 10

0.01

0.02

-1 0 10

0.02

0.04

-1 0 10

0.01

0.02

-1 0 10

0.02

0.04

-1 0 10

0.01

0.02

True negativeTrue positiveTo

rso

angl

esEl

bow

sW

rists

Knee

sFe

etTowards MLE

Depth sensing

Player performance

Depth image

Skeletal tracking

WireframeFeature

representation

Normalized correlations

Pairwise matching

Matched class

Distance metric

Features

Distance report

What gesture?

How well?

K inect

Gesture model

Prototype mean

Correlation stats

Energy stats

Logistic regression

coefs

Score computationC

lass

ran

kin

g

Ora

cle

Video game asks

Gesture Classification

System

Confusion Matrix

• HMM-based, others 50-60%

Summary

• Features – generic, nearly lossless• Dance aligned to beat – no major time-warp• Detection

– Normalized circular cross correlation via FFT– Maximum Likelihood estimator– Logistic regression to resolve class similarity

• Fast, scalable, extraordinary ROC