review of gesture recognition...

23
Review of Gesture Recognition Techniques By François Rioux, Feb. 23, 2004

Upload: others

Post on 14-Jul-2020

12 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Review of Gesture Recognition Techniques

By François Rioux, Feb. 23, 2004

Page 2: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Presentation Outline

Overview of the problemHuman gesture representationFeatures selectionRecognition techniques

HMM and « improved » HMMs

Other techniques

Page 3: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Gesture Recognition = Complex Task

Motion modelingMotion analysisPattern recognitionMachine learningPsycholinguistic studies…

Page 4: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Human Gesture Representation

Psycholinguistics research by Stokoe:Hand shape

PositionOrientation

Movement

Page 5: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Human Gesture Representation(cont)

Application scenarios of gesturesConversational

Controllingeg: vision-based interfaces

Manipulationeg: Interact with virtual objects

Communicationeg: Sign language → Highly structured

Page 6: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

More on Communicative Gestures

Highly structured3 phases

PreparationStroke (dynamic part)

Retraction

The dynamic part contains the information

Page 7: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Features Extraction

Good feature extraction is CRUCIALStatic hand posture

FingertipsFinger direction

Hand contour…

Page 8: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Features Extraction (cont)

Dynamic of gesturesSpatial features

eg: 2D location of hands. More general: 3D

Temporal featuresNeed of statistical method independent of time

Choosing only most relevant featuresMEF: Most Expressive Features

Karhunen-Loeve projection

MDF: Most Discriminative Features

Page 9: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Temporal Gesture Modeling and Recognition

Similar to speech recognitionHMMs can be used for low level recognitionHowever more complicated than speech…

Gesture semantics for high level recognition

Page 10: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

(Hidden Markov Model Overview)

Page 11: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Modeling the Dynamics

Low-level dynamics of human motionUseful for human motion recognition

Quantitative representation of simple motionNot sufficient for complex motion

Kalman filter:Estimate, interpolate and predict motion Not sufficient, Gaussian assumption

Page 12: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Modeling the Dynamics (cont)

Condensation Algorithm (Black & Jepson, 1998)Combine Dynamic Time Warping (DTW) and HMM

Capture detailed information of the motionHas a probabilistic framework

Goal: match trajectory models to input dataExtension to the standard condensation algorithm

Page 13: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Modeling the Dynamics (cont)

Condensation Algorithm (Black & Jepson, 1998)

Tests on whiteboardSlower than real timeManual evaluation of transition probabilitiesTraining data pre-segmented

Page 14: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Modeling the Dynamics (cont)

Modeling and Prediction of Human BehaviourPentland & Liu (1999)Multiple dynamic models sequenced by Markov chain

Kalman filter + HMM (observations are innovations of Kalman filter)

Application: prediction of driver’s behaviour

Page 15: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Modeling the Semantics

When modeling the dynamics is not enoughMore complex gestures

Finite State Machine (FSM)Extended variable-valued logic

Rule-based induction algorithm

PNF network {past, now, fut}Constraints on states based on previous

Bayesian network

Page 16: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

HMM Framework

Capacity to model low level and semanticsNam & Wohn (1996)

Problems:P(O|λ)

max P(S|O,λ)

Training => λ

Variation:Multi dimensional HMM

Page 17: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

HMM Framework (cont)

Real-Time Context-based Gesture Recognition Using HMM and Automaton

Iwai, Shimizu & Yachida (1999)

KL-transform to compress input data

Context-based HMM

Page 18: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

HMM Framework (cont)

HMM: only piecewise stationary processesGestures: all parts are transient => HMM Not always suitable for gesture rec.

Need improvements to HMM

Page 19: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

HMM Framework (cont)

Partly Hidden Markov Model Kobayashi & Haruyama (1997)Second order model

States

Observations

HMM

Page 20: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

HMM Framework (cont)

Markov condition violated => HMM fails

Coupled Hidden Markov Models (CHMM)Brand, Oliver & Pentland (1997)Coupling HMMs to model interactions between them

Page 21: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Other Techniques

Multi-class multi dimensional discriminant analysisCui & Weng (1996)Self-organizing framework

Action Recognition using Probabilistic ParsingBobick & Ivanov (1998)HMM for low level processing of featuresProbabilistic parsing using a particular grammar and ML

Time delay neural networksYang & Ahuja (1999)

Page 22: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

Conclusion

Important to model a gesture correctlyFeatures extraction can be hardMany features => CPU intensiveNeed to keep only most relevant featuresHMMs are good, but…Other method???

Page 23: Review of Gesture Recognition Techniquessrl.mcgill.ca/projects/gesture/ReviewGestureRecognitionTechniques.… · Gesture semantics for high level recognition (Hidden Markov Model

References

Wu & Yang, Vision-based Gesture Recognition: A Review, Lecture Notes in Artificial Intelligence, 1999