A Discriminative Key Pose Sequence Model for Recognizing Human

Interactions

Arash Vahdat, Bo Gao, Mani Ranjbar, and Greg Mori

ICCV2011

Goal

Outline

• Introduction• Related methods• Modeling Human Interactions• Single Subject Key Pose Sequence Model• Interaction Key Pose Sequence Model• Learning the parameters• Experiments

Introduction

• This paper focuses on recognizing interactions between individuals.

• The sequences of key poses between peoples will be combined into activity level.

• The activities to be recognized include hugging ,shaking hands,pointing, punching, kicking,pushing each others.

Introduction

• Not every movement or pose by the target is relevant to the activity to be recognized.

• We use an examplar-based model by giving the pose a score to match the key poses.

• When people do some activities, they will do the key poses in a chronological order.

Related methods

• Ryoo and Aggarwal [13]develop a matching kernel that considers spatial and temporal relations between space-time interest points.

• Yao et al. [23] use a Hough transform voting scheme from an interest point representation.

[13] M. Ryoo and J. Aggarwal. Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities. In ICCV, 2009.[23] A. Yao, J. Gall, and L. Van Gool. A hough transform-based voting framework for action recognition. In CVPR, 2010. 2, 6

Recognition result in [13]

Modeling Human Interactions

• There are four things to know:• 1. Who is involved in the interaction?– Subject or object

• 2. When do the key poses occur? – The interval of the key poses

• 3. How are the key poses executed?– With hand or leg , powerful or weak

• 4. Where are the people when the key poses occur?

Modeling Human Interactions

• we will assume F maximizes a model G that includes the latent variables H:

• The variables H are the answer of the four questions above.

Single Subject Key Pose Sequence Model

Single Subject Key Pose Sequence Model

• We represent each key pose by h:

• Denote K key poses of a sequense by H

Single Subject Key Pose Sequence Model

• Exemplar Matching Link:

• Compute the pose connection strength to the exemplar poses.

Single Subject Key Pose Sequence Model

• Activity-Key Pose Link:

– Compute the sequence similarity in the activity to give a score.

• Direct Root Model:

Interaction Key Pose Sequence Model

• Update the model from individual to two people interaction.

• We should recognize who is subject or object.• The scoring function will be:

Learning the parameters

• Define the scoring function E(x,y):

• Use multiclass linear SVM classifier to find the best parameters.

Experiments

• The UT-Interaction dataset contains videos of 6 classes of human-human interactions.

• Set 1 is with a stationary background,and Set 2 is with slight background movement and camera jitter.

Experiments

Experiments

Experiments

Experiments

Conclusion

• This paper focuses on the key poses method to recognize the interaction.

• The precision is over 90% and outperform other methods in UT-interaction dataset.