【itsc2015】fine-grained walking activity recognition via driving recorder dataset
TRANSCRIPT
Fine-grained Walking Activity Recognition via Driving Recorder Dataset
Hirokatsu KATAOKA, Yoshimitsu AOKI†, Yutaka SATOH Shoko, OIKAWA‡, Yasuhiro MATSUI‡
National Institute of Advanced Industrial Science and Technology (AIST) † Keio University
‡ National Traffic Safety and Environment Laboratory (NTSEL)
http://www.hirokatsukataoka.net/
Background • ADAS; Advanced Driver Assistance Systems – A large amount of technologies have been proposed – The pedestrian deaths are on the rise – Detection systems, environment, autonomous driving car
@Pedestrian and vehicle detec0on @Lane detec0on (Environment understanding)
@Autonomous driving in Google
ADAS technologies are highly required!
Pedestrian detection • Vision-based detection is one of the important techniques – Pedestrian detection survey [Benenson+, ECCVW2014] • They implemented and compared 40+ detection approaches
– Deep Learning is applied to detect pedestrians [Sermanet+, CVPR2013] • Convolutional neural networks (CNN) • Automatic feature training and classifier
Better
Detection rate has been improving
New step toward “pedestrian analysis” • High-performance pedestrian localization – Task-assistant CNN (TA-CNN) [Tian+, CVPR2015] • The framework is consist of CNN feat. & attribute (e.g. background, location)
• Limitations of pedestrian safety systems – Pedestrian detection at present – Detection range: width of the vehicle
Going to the next “pedestrian analysis” researches!
Motivation • Fine-grained pedestrian activity recognition in addition to pedestrian detection – More detailed activity analysis – Pedestrian activity intention understanding
Probability map of danger
1.0 second is crucial time in ADAS
Why fine-grained?
Walking along a sidewalk
Turning
Crossing a roadway
Process flow • Fine-grained walking activity recognition
1. Pedestrian localization 2. Activity analysis
Improved dense trajectories (iDT)
Pedestrian detection
x x x x x x x x x x x x x x x
x x x
Trajectory (in t + L frames)
Feature extraction (HOG, HOF, MBH, Traj.)
Bag-of-words (BoW)
iDT
Detection system • Per-frame CNN feature and NMS – Region of interesting (ROI) – VGGNet feature in the detection problem – Non-maximum suppression for combining detection windows
・・・~
~・・・
NMS
Activity Recognition • Improved Dense Trajectories (iDT) [Wang+, ICCV2013] – Pyramidal image sequences and flow tracking – Feature descriptors on trajectories – Feature representation with bag-of-words (BoW)
Walking Crossing Turning
Experiments • Fine-grained walking activity recognition – Understanding small changes while people walking • Walking along a side walk & Crossing a road way • Walking straight & turning • Walking & riding a bicycle
(a) crossing (b) walking (c) turning (d) bicycle
Datasets and implementations • NTSEL dataset & Near-miss dataset
• Implementation – Localization: VGGNet layer-pooling-5 – Feature: IDT (HOG, HOF, MBH, Traj.) – Classifier: Support vector machine (SVM)
(a) crossing (b) walking (c) turning (d) bicycle
NTSEL dataset Near-miss DR dataset
http://www.jsae.or.jp/hiyari/0907/
Results • On the NTSEL and Near-miss DR dataset
Descriptor % on NTSEL % on Near-miss DT (Traj.) 76.5 77.9 DT (HOF) 93.7 75.9 DT (HOG) 85.6 76.4 DT (MBHx) 87.7 59.3 DT (MBHy) 86.7 60.8
– Outstanding performance rate with IDT 93.7% on NTSEL and 77.9% on Near-miss DR dataset
Spatio-temporal analysis • Using iDT, temporal direction is analyzed – Fewer frames are better in the space-time – Sudden motion should be recognized
Demonstration • Fine-grained ped. activity recognition on NTSEL dataset – Improved Dense Trajectories (93.7%)
Conclusion • Fine-grained walking activity analysis for the new step of pedestrian intention understanding – State-of-the-art motion analysis algorithms are implemented – High-performance localization and recognition on the traffic datasets – Pedestrian analysis are executed in detail
• More flexible models and intention understanding – We need more data in learning step – Transition model or more strong temporal feature should be implemented