Learning

Tempo

ralRegularity

inVideo

Seq

uence

Dataand

Cod

es:

http://w

ww.ee.ucr.edu/~mhasan/regularity.htm

l

Mah

mud

ulHasan

1Jong

hyun

Cho

i2JanNeu

man

n2Am

itK.Roy-Cho

wdh

ury1

LarryS.Davis3

Thisresearchispartly

sup

ported

byNSFgrantIIS-13

1693

4and

ONRMURIGrantNo.N00

014-10

-1-093

4.

Anom

alyDe

tection:Com

parison

with

State-of-the

-artM

etho

ds

VisualizingTempo

ralR

egularity

Motivation

qWatchinglonghoursof

uncontrolledvideosisextremely

tedious

Nodatasetbiascompensated

Regularitiesb

ytheGe

neralM

odel

MoreRe

sultsareinth

epa

per

Applicationsand

Experim

ents

Asampleirregularfram

eSynthe

sized

Regularfram

eRe

gularitys

core

Asampleirregularfram

e Syn

thesize

dRe

gularframeRe

gularitys

core

UCSDPe

d2Dataset

UT-Au

stinSu

bway-ExitD

ataset

Pred

ictin

gNearP

asta

ndFuture

CUHK

Avenu

eDa

taset

UT-Au

stinSu

bway-ExitD

ataset

RegularityScores

UT-Au

stinSu

bway-ExitD

ataset

CUHK

Avenu

eDa

taset

UT-Au

stinSu

bway-EnterDataset

FeatureBa

sedFullyCon

nected

Autoe

ncod

er(FCo

nn)

End-to-End

FullyCon

volutio

nalA

utoe

ncod

er(FCo

nv)

UCRiverside1

ComcastLabs,DC2

UniversityofMaryland,CollegePark

3

qWewanttosegment‘m

eaningful’m

omentsinsuch

videoswithoutsupervision

Challenges

qLearningaclassificationmodelofthesemeaningful

(irregular)m

omentsisnottrivialbecause–

qIlldefined(anythingcanbem

eaningful)

qInfrequent(smalltrainingdata)

qLabelingthemisexpensive.

Approa

chq

Usetwohighcapacitygenerativem

odels-deep

neuralnetw

orkbasedauto-encoders(DNN-AE):

qFullyconnectedDNN-AEonhand-craftedfeature

qFullyconvolutionalDNN-AEonframes

qRe

gularinput->Reconstructioncostissmall.

qIrregular

input->Reconstructioncostislarge.

Exe

mpl

ar o

utpu

t of o

ur m

odel

whe

n th

ere

are

irreg

ular

mot

ions

, the

regu

larit

y sc

ore

drop

s sig

nific

antly

.

OurObjectiv

eq

Learningagenerativem

odelforregularitywith

qLimitedsupervisionrequired

qEaseoflearning

qMultipledatasetsusedtotrain

Overviewofthe

App

roach

Mod

elArchitecture

Input:HOG+HOF

collectedaroundthe

trajectoryofinterest

point

Inpu

tDataLayera

ndDataAu

gmen

tatio

nq

Inputcuboidsizes–5,10,and20(Weuse10)

qLargecuboid->betterdiscrim

inationandincreasedrunningtim

e.

qSlidingwindowsize:10,20,and30withsamplerateofstride

1,2,and3respectively.

qSlidingwindowsarem

oved2framesatatim

e.

TrainingaGen

eralM

odel

RegularityScore:s(t)

Optim

izatio

nq

LRScheme:AdaGrad

qInit.LR:0.001(FConv)and

0.01(FConn)

qMini-batchsize:1024

(FConv)and32(FConn)

qWeightinitialization:Xavier

e(x,y,t)=

kI(x,y,t)�

f

W(I(x,y,t))k 2

e(t)=

⌃(x

,y)e(x,y,t)

s(t)

=1�

e(t)�

min

te(t)

max

te(t)