clustering human behaviors with dynamic time warping and hidden markov models for a video...

Introduction Human Behavior Pattern Clustering Experimental Results Conclusion

Clustering Human Behaviors with Dynamic Time Warpingand Hidden Markov Models for a Video Surveillance System

Kan Ouivirach and Matthew N. Dailey

Computer Science and Information ManagementAsian Institute of Technology

ECTI-CONMay 19-21, 2010

1 / 29

Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System

Outline

1 Introduction

2 Human Behavior Pattern Clustering

3 Experimental Results

4 Conclusion

2 / 29

Introduction

Human behavior understanding is important for intelligent systems.

Difficult due to the wide range of activities possible in any givencontext

Figure: Reprinted from http://www.sourcesecurity.com/

A classic work by Yamato et al. who model tennis actions usinghidden Markov models (HMMs)

3 / 29

Introduction

3 / 29

Introduction

3 / 29

Introduction (cont.)

To help security personnel work reliably and efficiently,

filter out typical events;automatically present anomalous events to human operator.

Figure: Reprinted from http://sikafutu.com/

4 / 29

One limitation of most of the work

The number of “normal” behavior patterns need to be knownbeforehand.Nair and Clark (2002) use HMMs to model a common, predefinedactivity in a scene.

Unsupervised analysis and clustering of behaviors for a variety ofpurposes has started to draw attentions.

Li et al. (2006) cluster human gestures by constructing an affinitymatrix using dynamic time warping (DTW), and apply thenormalized-cut approach to cluster.

5 / 29

Some recent related works using HMMs to cluster behavior patterns

Swears et al. (2008) propose hierarchical HMM-based clustering tofind motion trajectories and velocities in a highway interchangescene.

6 / 29

Some recent related works using HMMs to cluster behavior patterns

Swears et al. (2008) propose hierarchical HMM-based clustering tofind motion trajectories and velocities in a highway interchangescene.

6 / 29

New method for clustering human behaviors in the context of videosurveillance

Combination of clustering and HMMs to group human behaviors

Summary flow

1 After extracting sequences, we use DTW to measure the pairwisesimilarity between sequences.

2 Construct an agglomerative hierarchical clustering dendrogrambased on the DTW similarity measure.

3 Recursively, find the optimal set of behavior clusters using HMMs.

Oates et al. (2001) first proposed the idea of using the DTW withHMMs to cluster time series.

7 / 29

Summary flow

7 / 29

Summary flow

7 / 29

Summary flow

7 / 29

Summary flow

7 / 29

Summary flow

7 / 29

Summary flow

7 / 29

Compared to the related works

Potential to improve upon the state of the art in intelligent videosurveillance applications by

Bootstrapping human behavior classification and anomaly detectionmodulesSupporting incremental HMM learning (performing statistical teststo select which cluster should be incrementally updated)

8 / 29

Outline

1 Introduction

4 Conclusion

9 / 29

Overview

We divide the proposed method into 2 phases.

1 Blob extraction

2 Behavior clustering

10 / 29

Blob Extraction

Foreground

Extraction

Background

Modeling

Single�Blob

Tracking

List�of�blobs

Vector

Quantization

Blob�features

CCTV�camera

Discrete�symbolsequences

Backgroundmodel

Sequence

Aggregation

Observationsymbols

Figure: Block Diagram of Blob Extraction

11 / 29

Blob Extraction (cont.)

We represent a blob at time t by the feature vector

~ft =[xt yt st rt dxt dyt vt

(xt , yt) is the centroid of the blob.

st is the size of the blob in pixels.

rt is the aspect ratio of the blob’s bounding box.

(dxt , dyt) is the unit-normalized motion vector for the blobcompared to the previous frame.

vt is the blob’s speed compared to the previous frame.

12 / 29

Behavior Clustering

Similarity

Measurement

Discrete�symbol

sequences

Agglomerative

Hierarchical�Clustering

Distancematrix

Dendrogram

HMM-based

Hierarchical�Clustering

Set�of�HMMs

Figure: Block Diagram of Behavior Clustering

13 / 29

Behavior Clustering (cont.)

Is�the�HMMa�sufficient�modelof�the�sequences�

in�c?

Is�C�empty?

c��cluster�at�rootof�dendrogram

Train�a�HMMon�the�sequences

in�c

C��{��}

Add�the�trained�HMMto�model�list�M

Replace�c�in�Cwith�the�children

of�c�from�DTW�dendrogram

c��any�clusterin�C

Remove�c�from�C

Figure: Processing flow of the use of HMM clustering method

14 / 29

How the processing flow works

15 / 29

The HMM is sufficient?

15 / 29

Not sufficient

15 / 29

Child Child

15 / 29

Child Child

The HMM is sufficient?

15 / 29

Blob Clustering (cont.)

The HMM is not sufficient to model the sequences

When there are more than N sequences in a cluster whoseper-observation log-likelihood is less than a threshold.

16 / 29

The HMM is not sufficient to model the sequences

When there are more than N sequences in a cluster whoseper-observation log-likelihood is less than a threshold.

16 / 29

To determine the optimal rejection threshold

Use an approach similar to that of Oates et al. (2001).Generate random sequences from the HMM.Calculate µc and σc of the per-observation log-likelihood over theset of generated sequences.Let a threshold be pc = µc − zσc , where z is experimentally tuned.

17 / 29

Outline

1 Introduction

4 Conclusion

18 / 29

Overview

Recorded videos at a resolution of 320× 240 and 25 fps over 1 week.

Used a motion detection to save disk space.

Obtained videos corresponding to over 500 motion events, butselected the 298 videos containing only a single motion.

19 / 29

Overview (cont.)

Found that at least 4 common behaviors:

Walking into the building (Walk-in)Walking out of the building (Walk-out)Parking a bicycle (Cycle-in)Riding a bicycle out (Cycle-out)

Other less common activities:

Walking while telephoning, etc. (Other)

20 / 29

Overview (cont.)

Figure: Example of common human activities in our testbed scene. (a)Walking in. (b) Walking out. (c) Cycling in. (d) Cycling out.

21 / 29

Overview (cont.)

Our main hypothesis

Using DTW as a pre-process prior to HMM-based clustering shouldimprove the quality of the clusters in term of separating anomalousfrom typical behaviors.

Compared to

Using only HMMsSupervised classification with HMMs

22 / 29

Overview (cont.)

Our main hypothesis

Compared to

22 / 29

Overview (cont.)

Our main hypothesis

Compared to

22 / 29

Overview (cont.)

Our main hypothesis

Compared to

22 / 29

Overview (cont.)

Performed 3 experiments.

Evaluated the results according to how well the induced categoriesseparate the anomalous sequences (hand-labeled with the category“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,Cycle-out).

1 Using our proposed method2 Using only HMMs3 Using HMMs with supervised learning

In all 3 experiments, we chose linear HMMs with 4 states based onour previous empirical experience.

23 / 29

Overview (cont.)

23 / 29

Overview (cont.)

23 / 29

Overview (cont.)

23 / 29

Overview (cont.)

23 / 29

Overview (cont.)

23 / 29

Overview (cont.)

Our configuration

the number of deviant patterns allowed in a cluster N = 10

z = 2.0 for a threshold pc = µc − zσc

24 / 29

Results for Experiment I

Clustering results for Experiment I (DTW+HMMs).

Cluster # Walk-in Walk-out Cycle-in Cycle-out Other

1 96 0 18 0 0

2 0 54 0 5 0

3 0 3 0 8 0

4 0 2 0 0 0

5 0 1 0 2 0

14 0 0 0 0 4

15 0 0 0 0 4

16 0 0 0 0 2

17 0 0 0 0 2

One-seqclusters 4 17 34 21 4

25 / 29

Results for Experiment II

Begin by training a single HMM on all sequences.

Assign every sequence with a per-observation log-likelihood above athreshold pc to a cluster.

Repeat the process by training a new HMM on the remainingsequences.

Clustering results for Experiment II (HMMs only).

Cluster # Walk-in Walk-out Cycle-in Cycle-out Other

1 15 77 49 43 16

2 80 0 11 2 0

3 5 0 0 0 0

26 / 29

Results for Experiment III

Trained 4 HMMs on each of the four typical beahviors.

Maximize the F1 value to determine the best per-observationlog-likelihood threshold for each HMM.

For the best separation between the positive and negative testpatterns

Results for Experiment III (Supervised classification with HMMs).

Anomaly detection rate (%) False alarm rate (%)

50 24.6

27 / 29

Outline

1 Introduction

4 Conclusion

28 / 29

Conclusion

We have proposed and evaluated a new method for clusteringhuman behaviors.

Our method

provides an initial partitioning of a set of behavior sequences, thenautomatically identifies where to cut off the hierarchical clusteringdendrogram.could be used to bootstrap an anomaly detection module forintelligent video surveillance applications.shows a perfect separation between typical and anomalous behaviorson real-world surveillance data without any information about thelabels.

29 / 29

Conclusion

Our method

29 / 29

Conclusion

Our method

29 / 29

Conclusion

Our method

29 / 29

Conclusion

Our method

29 / 29

clustering human behaviors with dynamic time warping and hidden markov models for a video...

Education