Crowd Scene Understanding with Coherent Recurrent Neural Networks ∗

Hang Su∗, Yinpeng Dong∗, Jun Zhu∗, Haibin Ling†, and Bo Zhang∗∗ Tsinghua National Lab for Information Science and Technology

∗ State Key Lab of Intelligent Technology and Systems∗ Department of Computer Science and Technology, Tsinghua University, Beijing, China

† Department of Computer and Information Sciences, Temple University, USA∗{suhangss, dongyp13, dcszj, dcszb}@mail.tsinghua.edu.cn, † hbling@temple.edu

Abstract

Exploring crowd dynamics is essential in under-standing crowd scenes, which still remains as achallenging task due to the nonlinear characteris-tics and coherent spatio-temporal motion pattern-s in crowd behaviors. To address these issues,we present a Coherent Long Short Term Memory(cLSTM) network to capture the nonlinear crowddynamics by learning an informative representa-tion of crowd motions, which facilitates the criti-cal tasks in crowd scene analysis. By describingthe crowd motion patterns with a cloud of keypointtracklets, we explore the nonlinear crowd dynam-ics embedded in the tracklets with a stacked LST-M model, which is further improved to capture thecollective properties by introducing a coherent reg-ularization term; and finally, we adopt an unsuper-vised encoder-decoder framework to learn a hiddenfeature for each input tracklet that embeds its inher-ent dynamics. With the learnt features properly har-nessed, crowd scene understanding is conducted ef-fectively in predicting the future paths of agents, es-timating group states, and classifying crowd events.Extensive experiments on hundreds of public crowdvideos demonstrate that our method is state-of-the-art performance by exploring the coherent spatio-temporal structures in crowd behaviors.

1 IntroductionUnderstanding collective behaviors in crowd scenes has awide range of applications in video surveillance and crowdmanagement [Sulman et al., 2008], especially in present erawith recurrent and tragic accidents in populous and diversehuman activities. However, a crowd is more than sum of in-dividuals, thus making the vision-related tasks disproportion-ately difficult along with the crowd scales. The past decadehas witnessed a significant progress in crowd scene analy-sis in learning global motion patterns [Mehran et al., 2010;

∗This research is partly supported by National Natural ScienceFoundation of China (Nos. 61571261, 61322308, 61332007), andChina Postdoctoral Science Foundation (No. 2015M580099)

Wu et al., 2010], modeling local spatio-temporal variation-s [Kratz and Nishino, 2012; Su et al., 2013], analyzing in-teractions among individuals [Mehran et al., 2009], profil-ing group behaviors [Zhou et al., 2011; 2012b], and de-tecting abnormal crowd behaviors [Solmaz et al., 2012;Mahadevan et al., 2010; Li et al., 2014]. Recently, Li etal. [2015] gave a comprehensive review on the state-of-the-art techniques on crowd scene understanding.

Although various methods have been developed, there isstill no publicly accepted framework in understanding thecrowd scenes, especially when extreme clutters or severeocclusions occur. One of the essential challenges is thatcrowd spatio-temporal behavior patterns behave abundantlynonlinear dynamics, such as limit cycles, quasi-period andeven chaos. This non-linear interaction between individual-s always result in various complex, spatio-temporal motionpatterns, e.g., the oscillations of the pedestrian flow at bot-tlenecks [Helbing and Johansson, 2009]. The popular lin-ear dynamic systems in crowd modeling [Lin et al., 2009;Shao et al., 2014] may fail to capture the nonlinear character-istics. Although the nonlinear characteristics of crowd mo-tions investigated in crowd simulation [Massink et al., 2011],few attempts are made in the vision-based crowd motion anal-ysis.

Another challenge in crowd behavior analysis is the col-lective effect (or coherent motion) [Zhou et al., 2012a; 2014],e.g., pedestrians in crowds tend to form coherent groupsby aligning with other neighbors. Different from the indi-vidual motion phenomena, there widely exist various self-organized spatio-temporal patterns even without externallyplanned or organized, which has been well explained withsocial force assumption [Helbing and Johansson, 2009]. Inthis case, methods that do not leverage the coherent char-acteristics may hinder the capabilities in capturing the in-herent crowd dynamics. For instance, crowd features learn-t from a multi-task deep architecture [Shao et al., 2015],although more effective than the handcrafted features, suf-fer from the lack of considering the essential non-lineartemporal correlations and coherent motions in crowd be-havior analysis. More recently, coherent motions in crowdscenes are detected with a thermal energy field such that pre-defined activities are effectively recognized [Lin et al., 2016;Wang et al., 2014]. However, it still fails to explore the non-linear crowd dynamics which hinders the performance for

complex crowd behaviors.

1.1 Our ProposalTo address the aforementioned challenges, we propose to ex-plore the crowd dynamics with a coherent Long Short Ter-m Memory (LSTM) architecture, which investigates the non-linearities of crowd behaviors with a stacked LSTM and en-forces the consistence of spatio-temporal structures with a co-herent regularization.

Recently, deep learning [Schmidhuber, 2015] has achievedstate-of-the-art performance in several vision tasks related tovisual understanding [Simonyan and Zisserman, 2014] andscene analysis [Ramanathan et al., 2015] partly due to it-s good capabilities in modeling nonlinearity. It inspires usto explore the nonlinear dynamics of crowd behaviors witha deep architecture model. Specifically, we build our modelbased on the Long Short Term Memory (LSTM) [Hochreit-er and Schmidhuber, 1997] network, which is an improvedRecurrent Neural Network (RNN) for temporal data model-ing. LSTM has been proven successful on tasks in generat-ing image caption [Vinyals et al., 2015] and video descrip-tion [Donahue et al., 2015], since it provides a good proba-bility to explore the long term dynamics by overcoming theproblem of gradient vanishing and exploding for convention-al RNNs [Lipton, 2015]. In order to cope with nonlineardynamics in the long term crowd behaviors, we use a multi-layer LSTM network to learn an informative representation ofcrowd tracklets1, which are more conservative and less likelyto drift than long trajectories.

In order to capture the coherent spatio-temporal structures,we further improve the multi-layer LSTM network by intro-ducing a coherent regularization term to model the local s-patial and temporal dependency between neighboring pedes-trians within a coherent group. The memory unit in LSTMtherefore not only stores the dynamic information embeddedin the tracklet of its own but also the dynamics of its neigh-boring agents. The resulting model is denoted as coherentLSTM (cLSTM).

Finally, we adopt an unsupervised LSTM auto-encoderframework [Srivastava et al., 2015] to learn a representationto explore the crowd dynamics, such that the tedious efforts incollecting labeled data are significant reduced. Specifically, astacked cLSTM first encodes the input tracklets into a hid-den feature, which is subsequently decoded to reproduce theinput tracklets. By exploiting the hidden feature and the co-herent regularization, we can extrapolate the past dynamics tothe future and forecast the motion beyond what has been ob-served by recursively unrolling the feature to the future. Morecritical tasks in crowd scene analysis are also conducted us-ing the learnt representation, including group state estimationand crowd event classification.

In summary, our work differs significantly from the exist-ing studies [Shao et al., 2014; 2015] in that the crowd dynam-ics are captured via an LSTM model, which is “deep in time”

1A tracklet is a fragment of a trajectory obtained by a tracker,e.g., a KLT keypoint tracker [Baker and Matthews, 2004], whichstarts when a novel key-point is detected and terminates when ambi-guities arise.

and can identify informative structures in time domain. To thebest of our knowledge, this study is a first attempt that investi-gates the non-linear characteristics of crowd motion patternswith LSTM. Our main contributions are:• We propose to investigate the crowd dynamics with a

stacked LSTM model, such that the complex and non-linear crowd motion patterns are well captured;• To consider the collective properties in crowd motion

patterns, we propose to improve LSTM by introducinga coherent regularization which encourages a consistentspatio-temporal hidden feature;• Finally, we adopt the hidden features learnt from the co-

herent LSTM to critical tasks in crowd scene analysis,including future path prediction, group state estimation,and crowd behavior classification. Experiments demon-strate state-of-the-art performance of our method.

2 Model Crowd Motions with cLSTMWe consider to describe the crowd motion pattern with a setof tracklets {xt} due to its explainable semantics in crowdbehaviors [Li et al., 2015], as illustrated in Fig. 1. In this sec-tion, we aim to extract informative features from these track-lets to explore the crowd dynamics, which facilitates the sub-sequent tasks in crowd scene analysis. To this end, we firstintroduce the basic coherent LSTM unit which updates itsmemory with the tracklet of its own together with its neigh-boring agents; afterwards, we step into details of the coherentregularization term and describe the LSTM models by stack-ing the coherent LSTM unit.

Figure 1: Samples of tracklets for crowd motion patterns.

2.1 Coherent Long Short Term Memory UnitLSTM has a powerful capability in modeling the nonlin-ear dynamics for sequential data [Lipton, 2015; Greff et al.,2015]. As illustrated in Fig. 2, each LSTM unit has a cellas a memory, which maintains its state ct at time t by reg-ulating the information flow into/out of the LSTM unit withnon-linear gates [Greff et al., 2015].

Specifically, the state of a cell is controlled through sig-moidal gates including an input gate it which takes activationfrom the current data point xt and the hidden layer at the pre-vious time step ht−1 as

it = σ(Wxixt +Whiht−1 +Wcict−1 + bi), (1)

and a forget gate ft which enables the cell to reset its state as

ft = σ(Wxfxt +Whfht−1 +Wcfct−1 + bf ), (2)

where Ws are the weight matrices with a proper size; bs arebias vectors. Note that all the Wc• matrices corresponding to

σ

σ

σ

ϕ

Forget Gate

Input

Gate

Output Gate

Cell

tx

1th −

th

Coherent

Regularization

Figure 2: A diagram of a coherent LSTM unit.

the cell states are diagonal, whereas the rest are full matrices.The total input at the input terminal is passed through the tanhnon-linearity and multiplied by the activation of the input gateit, which is then added to the cell state ct−1 multiplied by theforget gate’s activation ft as

ct= ft � ct−1 + it � tanh(Wxcxt +Whcht−1 + bc), (3)

where � is the element-wise multiplication. The final outputfrom the LSTM unit ht is computed by multiplying the up-dated cell state that passed through a tanh non-linearity withan output gate’s activation as

ht = ot � tanh(ct), (4)

where ot is the output gate as

ot=σ(Wxoxt +Whoht−1 +Wcoct + bo). (5)

The output gate ot controls how much of the memory cellshould be transferred to the hidden feature. Compared withthe conventional RNN, the additional cells in LSTMs sumthe activities over time. Such strategy avoids a quick gradientvanishing and enables the LSTMs to learn extremely complexand long-term temporal dynamics in crowd behavior analysis.

Different from the individual behaviors, there widely existcoherent motion phenomena [Zhou et al., 2012a] in crowds,since the individuals are always willing to engage with “seed”groups and form spatially coherent structures. Therefore, wepropose to improve the conventional LSTM by taking theneighboring tracklets into account. The intuition behind themodel is if the dynamics of two tracklets are coherent in thespatial and temporal domain, i.e., when the neighboring re-lationship of individuals remains invariant over time or thecorrelation of their velocities remains high, they tend to havesimilar hidden states. To this end, we propose to update thememory unit by incorporating its own state together with itsneighboring agents with a coherent regularization as

ct = ft � ct−1 +∑j∈N

λj(t)fjt � cjt−1

+ it � tanh(Wxcxt +Whcht−1 + bc), (6)

where N denotes the set of neighboring tracklets within acoherent group; f jt and cjt−1 are corresponding to the forgetgate and cell state for LSTM in the coherent group; and λj(t)weights the dependency between the tracklets, as detailed be-low.

2.2 Coherent Motion ModelingIn this section, we first investigate the dependency betweenagents in coherent groups, which are discovered using the co-herent filtering [Zhou et al., 2012a], as illustrated in Fig. 3.The coherent keypoints with similar motion patterns and ten-dencies are marked with the same color.

Figure 3: Group detection using coherent filtering [Zhou et al.,2012a], in which different groups are indicated with different col-ors (best viewed in color version).

The dependency relationship between two tracklets with-in the same group is measured with their pairwise velocitycorrelations as

τj(t) =vi(t) · vj(t)

‖vi(t)‖‖vj(t)‖, (7)

where vi(t) and vj(t) are the velocities of the ith and jthtracklets, respectively. The dependency coefficient betweenthe ith and jth tracklets in Eq. (6) is defined as

λj(t) =1

Ziexp

(τj(t)− 1

2σ2

)∈ (0, 1], (8)

where Zi is the normalization constant corresponding to theith tracklet. λj(t) tends to be 1 when the tracklets i and j aresimilar, and decreases when the tracklets become different. Inthis case, our model with coherent regularization encouragesthe tracklets to learn similar feature distributions by sharinginformation across tracklets within a coherent group.

In order to model the long term crowd dynamics, additionaldepths are also added to LSTMs by stacking them on top ofeach other, i.e., using the output of the LSTM in the (l−1)thlayer as the input to the LSTM in the lth layer, as illustratedin Fig. 4.

3 Crowd Scene Analysis based on cLSTMIn this section, we describe an unsupervised encoder-decoderframework using the coherent LSTM to generate a hiddenfeature, which embeds the inherent characteristics for eachtracklet. It shares similar ideas to that of auto-encoders [Vin-cent et al., 2010] such that the parameters are optimized byminimizing the difference between the reproduced and targetsequences. Critical tasks in crowd scene analysis is conduct-ed based on the hidden features, e.g., future path forecasting,group state estimation and crowd behavior classification.

To learn an informative representation, we take the“encoder-decoder”approach inspired by [Donahue et al.,

LSTM

LSTM LSTM

LSTM

Coherent

Regularization

Tracklet: Tracklet:iT

jT

Figure 4: We stack a series of coherent LSTM units to capture thetime-varying nonlinear crowd dynamics by mapping tracklets in acoherent group to similar hidden features, which is conducted byincorporating a coherent regularization.

2015], which consists of an encoder coherent LSTM and adecoder coherent LSTM, as is shown in Fig. 5. The encodercoherent LSTM runs though a tracklet to come up with a hid-den feature, which is decoded to produce a target sequencewith a decoder coherent LSTM. Note that our model allowsto reproduce the input tracklets, and also provides a methodto forecast the unseen future paths by exploiting the hiddenfeatures.

Coherent Regularization

1x 2x 3x

2x 1x3x

4x 5x 6xeW eW

rdW rdW

pdW pdWEncoder

Reconstruction

Decoder

Prediction

DecoderLearnt Hidden

Features

Th

Figure 5: The encoder-decoder framework with coherent LSTM.The encoder generates a hidden feature of each tracklet, and the de-coder reproduces the input tracklets and predicts the future paths.

At the encoding stage, we obtain representation vectors fortracklets with the encoder coherent LSTM following Eq. (4)as

hT = cLSTMe(xT ,hT−1), (9)

where cLSTMe is an encoding operation to map the input

Coherent

regularization

Coherent

regularization

Motion Prediction

LSTM

LSTM

LSTM

Figure 6: Prediction of future path with coherent regularization, inwhich the blue paths are reliable tracklets obtained from KLT track-ers and the green ones are the forecasted paths from the cLSTM.

tracklet to a hidden feature; xT and hT−1 are the input track-lets at time step T and hidden feature vectors at the previoustime step T−1, respectively.

As for the reconstruction decoder, the coherent LSTM gen-erates a set of estimated tracklet x for each tracklet, whichreproduces the input tracklets but in a reverse order to avoidthe long range correlations as

xt = cLSTMdr(ht, xt+1), where t ∈ [1, T ], (10)

where cLSTMdr decodes the representation recursively toreproduce the input tracklets; ht is the feature that is de-rived from the hidden features hT in Eq. 9. The parame-ters {Ws,bs} are optimized by minimizing the reconstruc-tion error between the input tracklets and the reproduced oneswhen training the model.

The prediction decoder is similar to that of the reconstruc-tion decoder, except that the decoder LSTM extrapolates fu-ture paths. Specifically, the prediction is implemented by un-rolling the hidden feature by

xt = cLSTMdp(ht, xt−1), where t > T, (11)

where cLSTMdp is a decoding operation to predict the futurepath of an agent derived from the hidden feature by takingthe coherent motion into account. During the training stage,tracklets in the training dataset {xt}Tt=1 are divided into twosegments as {xt}T0

t=1 and {xt}Tt=T0+1. The former segmentsare input to the encoder to learn a hidden feature, which isused to predict the latter segment by minimizing the differ-ence between the original and estimated tracklets. The pa-rameter T0 is modified to adjust the length ratio between thesegments, which enhances inherent evolutionary dynamicsare well captured.

The success of path prediction lies in the facts that the hid-den states generated from the encoder captures the dynamicsof tracklets to forecast the future within a coherent group,which is further enhanced by incorporating with the coherentregularization, as the schematic shown in Fig. 6.

In summary, we capture the inherent crowd dynamics bymapping the tracklets to hidden features with the cLSTMmodel. With the features properly harnessed, we reconstruc-t the input tracklets as well as predict the future paths. Areconstruction-oriented encoder would suffer from the ten-dency to memorize the inputs, and the future predictor wouldsuffer from the tendency to ignore the initial frames becauseof the more significant impacts from the last few frames.Therefore, the proposed method encourages a more inher-ent feature by mutually enhancing each other, such that the

features maintain the dynamics embedded in the whole se-quences by not just memorizing the information.

3.1 Crowd Scene ProfilingWith crowd dynamics properly captured by our coherent L-STM model, the critical tasks in crowd scene analysis can beconducted based on the learnt features, e.g., understandinggroup states and recognizing crowd behaviors.

The states of groups with coherent spatio-temporal struc-tures are generally recognized as Gas, Solid, Pure Fluid andImpure Fluid [Shao et al., 2014], which is related to multiplesocial psychological and physical factors, e.g., crowd densi-ty, goals, interactions between group members, etc. In thissection, we implement group state estimation with the learntrepresentation since it embeds dynamics of coherent groups.Specifically, we feed the feature learnt from the unsupervisedcLSTM to a softmax classifier, which infers the group statesbased on the dynamics of tracklet within a coherent group as

L =1

NT

N∑i=1

T∑j=1

C∑c=1

1{yi(t) = c} log eηTc hi(t)

Nc∑c=1

eηTc hi(t)

, (12)

where 1(·) is an indicator function with value 1 if the predi-cate is true otherwise 0; ηc is a parameter to weight the hid-den feature corresponding to class c; hi(t) is the hidden fea-tures corresponding to the ith tracklet in the group at timestep t; N denotes the total number of tracklets in a coherent

group; and T is length of tracklets. The term logNc∑c=1

eηTc hi(t)

normalizes the distribution to guarantee the probability con-dition. A reliable inference is conducted by maximizing thesoftmax regression.

Besides estimating the state of each individual group, wealso implement the holistic crowd video classification bytraining another softmax classifier over the sequential hid-den features of all the tracklets in a crowd video, producing adistribution over the holistic crowd behavior categories. Thesuccess of complex crowd behavior recognition lies on thefacts that composing deep layers of cLSTMs can result in apowerful capability in exploring the nonlinearities and coher-ent spatio-temporal structures in crowd motions.

4 Experimental ResultsIn this section, we demonstrate the effectiveness of the fea-tures learnt from our algorithm on three critical applicationsin crowd scene analysis: pedestrian future path prediction,group state estimation, and crowd behavior classification. E-valuations are conducted on the CUHK Crowd Dataset [Shaoet al., 2014], which includes crowd videos with differen-t densities and perspective scales in many environments, e.g.,street, airports, etc. The ground truth of keypoint tracklets,group state, and crowd video classification are also availablefor the dataset. It consists of more than 400 sequences, and200,000+ tracklets in total.

In each experiment, we construct a coherent LSTM with128 hidden units, such that the input tracklets are mappedto 128-dimensional hidden features. Similar as the work

in [Shao et al., 2014], we randomly select half of the track-lets in the sequences for training and the remaining for test-ing. When optimizing the parameters in predicting the fu-ture paths, we divide each tracklet into two segments, and usethe hidden features learnt from the first segments (e.g., 2/3 ofeach tracklet) to predict the latter segments (e.g., the rest 1/3tracklet).

4.1 Future Path Forecasting of PedestriansWe first test the performance of our framework on path fore-casting by comparing our approach with a baseline methodof Kalman filter, which implements path prediction by incor-porating a linear dynamic model with the uncertainties of thecurrent state, which cannot capture the nonlinear characteris-tics of the complex crowd motions and leverage informationof the coherent groups. We also conduct path prediction witha variant of the proposed cLSTM model by neglecting the co-herent regularization between neighboring agents, which isdenoted as Un-coherent LSTM. In each experiment, we takea fragment of tracklets as the input (e.g., 2/3 of each trackletin this paper), reconstruct them and then generate the rest ofthe predicted tracklets (e.g., 1/3 of each tracklet) to evaluatethe performance.

Figure 7: Prediction of future paths with coherent regularization, inwhich the red paths are reliable tracklets obtained from KLT trackersand the green ones are the forecasted paths from the cLSTM.

Sample results are demonstrated in Fig. 7, in which the redtracklets are the paths obtained with the KLT tracker [Bak-er and Matthews, 2004] that are followed by green curves oftracklets generated with our cLSTM prediction model. Theresults demonstrate that our algorithm is capable to capturethe inherent dynamics of each tracklet, which reflects the ten-dency of neighboring agents by taking the coherent regular-ization into account. Since the near future paths are well pre-dicted, it provides a good probability to prevent the seriousincidents before they would emerge.

In Table 1, we report the quantitative performance of pathforecasting in terms of prediction error, which measures theaverage distance between the ground-truth tracklets and theestimated paths in terms of pixel as unit. It shows that ourapproach significantly outperforms the alternative methods.Compared with the baseline method of Kalman filter, ourcLSTM captures the inherent nonlinearity of the crowd be-havior such that the complex crowd motions are well predict-

ed with a high precision; moreover, our method also benefitsfrom the coherent regularization due to the collective proper-ties in moving crowds.

Table 1: Error of Path Prediction

Kalman Filter Un-coherent LSTM Coherent LSTM9.32 ± 1.99 6.64 ± 1.76 4.37±0.93

4.2 Group State EstimationGroup states well reflect the characteristics of a crowd, whichare useful in various applications. In the CUHK CrowdDataset [Shao et al., 2014], groups are classified into fourstates as Gas2, Solid3, Pure Fluid4 and Impure Fluid5. Sam-ples of groups with different states are illustrated in Fig. 8.

(a) Gas (b) Solid (c) Pure Fluid (d) Impure Fluid

Figure 8: Samples of groups undergoing different states.

In this section, we train a softmax classifier using the hid-den features learnt by our cLSTM, and then implement thegroup state estimation. As a comparison, we also conductgroup state estimation by feeding the descriptors learnt fromvariants of the proposed cLSTM model, including the predic-tion LSTM by only targeting at predicting future paths, re-construction LSTM by neglecting the prediction components,and un-coherent LSTM by neglecting the coherent regulariza-tion. Besides, we also report the results based on a collectivetransition [Shao et al., 2014], which is the state-of-the-art al-gorithm in group state estimation.

The quantitative evaluation in terms of confusion matrix isreported in Fig. 9. Obviously, our proposed algorithm withcoherent LSTM (Fig. 9 (e)) outperforms the alternative meth-ods. As a baseline method, group state estimation based oncollective transition [Shao et al., 2014] (Fig. 9(a)) exploresthe crowd dynamics via a linear transition matrix, which isnot valid for groups with complex motion patterns, e.g., whengroups undergo an impure fluid state. The performance alsodegrades when we neglect the reconstruction or predicationcomponents for prediction LSTM (Fig. 9 (b)) or reconstruc-tion LSTM (Fig. 9 (c)), since the prediction LSTM tends tomaintain information of the last few frames rather than the w-hole tracklets, and the reconstruction LSTM suffers from the

2Gas: Particles move in different directions without forming col-lective behaviors

3Solid: Particles move in the same direction with relative posi-tions unchanged

4Pure Fluid: Particles move towards the same direction withever-changing relative positions

5Impure Fluid: Particles move in a pure fluid style with invasionof particles from other groups

(a) Collective Transition

(e) Coherent LSTM(d) Un-coherent LSTM

(b) Prediction LSTM (c) Reconstruction LSTM

Figure 9: Confusion matrices of estimating group states using differ-ent methods: (a) collective transition [Shao et al., 2014]; (b) predic-tion LSTM; (c) reconstruction LSTM; (d) un-coherent LSTM; and(e) coherent LSTM. See text for the description of each method.

tendency to memorize the tracklets rather than explore the in-herent dynamics. When we implement group state estimationwithout taking the collective properties into account using theun-coherent LSTM (Fig. 9(d)), the performance is significant-ly inferior to our cLSTM method especially for groups withorganized structures, e.g., the group undergoing a solid or flu-id state.

4.3 Crowd Video ClassificationFinally, we demonstrate the effectiveness of our method inclassifying crowd video dependent on the holistic crowd be-haviors in a scene. In CUHK Crowd Dataset [Shao et al.,2014], all video clips are annotated into 8 classes which arecommonly seen in crowd videos as 1) Highly mixed pedes-trian walking; 2) Crowd walking following a mainstream andwell organized; 3) Crowd walking following a mainstream butpoorly organized; 4) Crowd merge; 5) Crowd split; 6) Crowdcrossing in opposite directions; 7) Intervened escalator traf-fic; and 8) Smooth escalator traffic.

Similar as the implementations of group state estimation,we conduct crowd video classification by feeding the repre-sentations, which are learnt by our coherent LSTM, to a soft-max classifier. As comparisons, crowd video classificationis also implemented with variants of the proposed cLSTMmethod, including prediction LSTM, reconstruction LSTM,and un-coherent LSTM (See Section 4.2 for details). Besides,we also compare with collective transition [Shao et al., 2014].In Fig. 10, we report a confusion matrix in crowd video classi-fication based on our cLSTM, which demonstrates that videosare classified into specific categories with high qualities.

In Fig. 11, we demonstrate the accuracy of crowd videoclassification on each class by various methods as mentionedabove. We can see that the coherent LSTM outperforms thealternative methods in general, since it captures the nonlineardynamics embedded in the complex crowd motion by tak-ing into the coherent motion at the same time, especially forcrowds with high nonlinearity, e.g., highly mixed pedestri-an walking. Our method significantly outperforms the col-lective transition method [Shao et al., 2014], which fails toexplore the nonlinear characteristics of crowd motion and

Figure 10: Confusion matrix of crowd video classification based oncoherent LSTM. See text for the name of each class.

meanwhile neglects the coherent spatio-temporal structuresin crowd videos. Note that the videos of Crowd walking fol-lowing a mainstream and well organized are well recognizedby collective transition, since crowd motions of these videossatisfy the linear dynamics assumption of collective transitionin most cases.

Our cLSTM also benefits from exacting hidden features byincorporating the reconstruction with the prediction tasks to-gether. Obviously, neither prediction LSTM or reconstructionLSTM fails to capture the inherent dynamics of overall crowdmotion, which degenerates the performance in video classi-fication. Besides, the effect of coherent regularization is al-so essential in the scenarios when the collectiveness within acrowd are significant, e.g., crowd merge or split.

Figure 11: Per-class accuracy comparison of crowd video classifica-tion using different methods. See text for the name of each class.

5 ConclusionsWe present a novel recurrent neural network with coherentlong short term memory (cLSTM) units to understand crowdscenes. To address the nonlinear dynamics in complex crowdscenes, we propose to map a set of tracklets that describethe crowd motion patterns to hidden features with LSTM,which can keep track of an input tracklet. To consider thecollective properties of moving crowds, we introduce a co-herent regularization such that the memory units are updat-ed by taking into account dynamics of tracklets with coher-ent motions, which encourages the learnt hidden features tohave consistent spatial and temporal structures. With the in-herent dynamics properly harnessed, our algorithm also pro-vides a method to predict the possible future paths, which is

of great importance in crowd management to prevent seriousaccidents before they emerge. Extensive experiments on real-world dataset demonstrate that our method outperforms itsvariants and alternative method in group state estimation andcrowd video classification.

References[Baker and Matthews, 2004] Simon Baker and Iain

Matthews. Lucas-kanade 20 years on: A unifyingframework. International Journal of Computer Vision,56(3):221–255, 2004.

[Donahue et al., 2015] Jeffrey Donahue, Anne Hendricks,Kate Saenko, and Trevor Darrell. Long-term recurrentconvolutional networks for visual recognition and descrip-tion. In Proceedings of the IEEE Conference on Comput-er Vision and Pattern Recognition (CVPR), pages 2625–2634, 2015.

[Greff et al., 2015] Klaus Greff, Rupesh Kumar Srivastava,Jan Koutnık, Bas R Steunebrink, and Jurgen Schmidhu-ber. Lstm: A search space odyssey. arXiv preprint arX-iv:1503.04069, 2015.

[Helbing and Johansson, 2009] Dirk Helbing and Anders Jo-hansson. Pedestrian, crowd and evacuation dynamics.Springer, 2009.

[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter andJurgen Schmidhuber. Long short-term memory. NeuralComputation, 9(8):1735–1780, 1997.

[Kratz and Nishino, 2012] Kratz and K. Nishino. Trackingpedestrians using local spatio-temporal motion patternsin extremely crowded scenes. Pattern Analysis and Ma-chine Intelligence, IEEE Transactions on, pages 987 –1002, May 2012.

[Li et al., 2014] Weixin Li, Vijay Mahadevan, and NunoVasconcelos. Anomaly detection and localization incrowded scenes. Pattern Analysis and Machine Intelli-gence, IEEE Transactions on, 36(1):18–32, 2014.

[Li et al., 2015] Teng Li, Huan Chang, Meng Wang, Bing-bing Ni, Richang Hong, and Shuicheng Yan. Crowdedscene analysis: A survey. Circuits and Systems for VideoTechnology (TCSVT), IEEE Transactions on, 25(3):367–386, 2015.

[Lin et al., 2009] Dahua Lin, E. Grimson, and J. Fisher.Learning visual flows: A lie algebraic approach. In Com-puter Vision and Pattern Recognition, IEEE Conference on(CVPR), pages 747 –754, June 2009.

[Lin et al., 2016] Weiyao Lin, Yang Mi, Weiyue Wang,Jianxin Wu, Jingdong Wang, and Tao Mei. A diffusionand clustering-based approach for finding coherent mo-tions and understanding crowd scenes. IEEE Transactionson Image Processing, 25(4):1674–1687, 2016.

[Lipton, 2015] Zachary C Lipton. A critical review of recur-rent neural networks for sequence learning. arXiv preprintarXiv:1506.00019, 2015.

[Mahadevan et al., 2010] Vijay Mahadevan, Weixin Li, ViralBhalodia, and Nuno Vasconcelos. Anomaly detection in

crowded scenes. In Computer Vision and Pattern Recog-nition (CVPR), 2010 IEEE Conference on, pages 1975–1981, 2010.

[Massink et al., 2011] Mieke Massink, Diego Latella, An-drea Bracciali, and Jane Hillston. Modelling non-linearcrowd dynamics in bio-pepa. In Fundamental Approachesto Software Engineering, pages 96–110. 2011.

[Mehran et al., 2009] R. Mehran, A. Oyama, and M. Shah.Abnormal crowd behavior detection using social forcemodel. In Computer Vision and Pattern Recognition, IEEEConference on (CVPR), pages 935 –942, June 2009.

[Mehran et al., 2010] Ramin Mehran, Brian E. Moore, andMubarak Shah. A streakline representation of flow incrowded scenes. In European Conference on ComputerVision (ECCV), pages 439–452, 2010.

[Ramanathan et al., 2015] Vignesh Ramanathan, KevinTang, Greg Mori, and Li Fei-Fei. Learning temporalembeddings for complex video analysis. arXiv preprintarXiv:1505.00315, 2015.

[Schmidhuber, 2015] Jurgen Schmidhuber. Deep learning inneural networks: An overview. Neural Networks, 61:85–117, 2015.

[Shao et al., 2014] Jing Shao, Chen Change Loy, and Xiao-gang Wang. Scene-independent group profiling in crowd.In Computer Vision and Pattern Recognition (CVPR),2014 IEEE Conference on, pages 2227–2234, 2014.

[Shao et al., 2015] Jing Shao, Kai Kang, Chen Change Loy,and Xiaogang Wang. Deeply learned attributes for crowd-ed scene understanding. In Computer Vision and Pat-tern Recognition (CVPR), IEEE Conference on, pages 1–9,2015.

[Simonyan and Zisserman, 2014] Karen Simonyan and An-drew Zisserman. Two-stream convolutional networks foraction recognition in videos. In Advances in Neural Infor-mation Processing Systems (NIPS), pages 568–576, 2014.

[Solmaz et al., 2012] Berkan Solmaz, Brian Moore, andMubarak Shah. Identifying behaviors in crowd scenesusing stability analysis for dynamical systems. PatternAnalysis and Machine Intelligence (TPAMI), IEEE Trans-actions on, pages 2064 –2070, Oct. 2012.

[Srivastava et al., 2015] Nitish Srivastava, Elman Mansi-mov, and Ruslan Salakhutdinov. Unsupervised learningof video representations using lstms. arXiv preprint arX-iv:1502.04681, 2015.

[Su et al., 2013] Hang Su, Hua Yang, Shibao Zheng, YawenFan, and Sha Wei. The large-scale crowd behavior per-ception based on spatio-temporal viscous fluid field. In-formation Forensics and Security, IEEE Transactions on,8(10):1575–1589, 2013.

[Sulman et al., 2008] Noah Sulman, Thomas Sanocki, Dmit-ry Goldgof, and Rangachar Kasturi. How effective is hu-man video surveillance performance? In Pattern Recogni-tion, 19th International Conference on (ICPR), pages 1–8,2008.

[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is-abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man-zagol. Stacked denoising autoencoders: Learning usefulrepresentations in a deep network with a local denoisingcriterion. Journal of Machine Learning Research (JLMR),11:3371–3408, 2010.

[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev,Samy Bengio, and Dumitru Erhan. Show and tell: A neu-ral image caption generator. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition(CVPR), pages 3156–3164, 2015.

[Wang et al., 2014] Weiyue Wang, Weiyao Lin, YuanzheChen, Jianxin Wu, Jingdong Wang, and Bin Sheng. Find-ing coherent motions and semantic regions in crowdscenes: A diffusion and clustering approach. In EuropeanConference on Computer Vision (ECCV), pages 756–771.2014.

[Wu et al., 2010] Shandong Wu, Brian E. Moore, andMubarak Shah. Chaotic invariants of lagrangian particletrajectories for anomaly detection in crowded scenes. InComputer Vision and Pattern Recognition (CVPR), IEEEConference on, pages 2054 –2060, June 2010.

[Zhou et al., 2011] Bolei Zhou, Xiaogang Wang, and XiaoouTang. Random field topic model for semantic region anal-ysis in crowded scenes from tracklets. In Computer Vi-sion and Pattern Recognition (CVPR), IEEE Conferenceon, June 2011.

[Zhou et al., 2012a] Bolei Zhou, Xiaoou Tang, and Xiao-gang Wang. Coherent filtering: detecting coherent motionsfrom crowd clutters. In European Conference on Comput-er Vision (ECCV), pages 857–871, 2012.

[Zhou et al., 2012b] Bolei Zhou, Xiaogang Wang, and Xi-aoou Tang. Understanding collective crowd behaviors:Learning a mixture model of dynamic pedestrian-agents.In Computer Vision and Pattern Recognition (CVPR),IEEE Conference on, pages 2871 –2878, June 2012.

[Zhou et al., 2014] Bolei Zhou, Xiaoou Tang, HepengZhang, and Xiaogang Wang. Measuring crowd collective-ness. Pattern Analysis and Machine Intelligence, IEEETransactions on (TPAMI), 36(8):1586–1599, 2014.