personal driving diary: constructing a video archive of everyday driving events

Personal Driving Diary:Constructing a Video Archive of Everyday Driving

Events

IEEE workshop on Motion and Video Computing ( WMVC) 2011

IEEE Workshop on Applications of Computer Vision (WACV) 2011

Electronics and Telecommunications Research InstituteM. S. Ryoo, Jae-Yeong Lee, Ji Hoon Joung, Sunglok Choi, and Wonpil Yu

Introduction

• It illustrates important driving events of the user.– Enable interactive search of video segments– Help the user to analyze his/her driving habits and

patterns • The objective is to construct a system that

automatically annotates and summarizes videos.

Framework

geometry component(1/2)

• visual odometry [9]– To measure the self-motion of the camera.

geometry component(2/2)

• visual odometry– Feature (SIFT) detection for each frame– Matching is performed using KLT optical flows

• Estimating a ground plane using regular patterns on the ground (e.g. lane and crosswalk)– It enables global localization of other objects on

it.

Detection component(1/3)

Detection component(2/3)

• Detect pedestrians– Adopt histogram of oriented gradients (HOG)

features [3] and apply a sliding windows method– Filtering out windows with little vertical edges

Detection component(3/3)

• Vehicle detection– Apply the Viola and Jones’ method [15] to detect

rear-view of appearing vehicles

[15] P. Viola and M. Jones. Rapid object detection using a boostedcascade of simple features. In CVPR, 2001.

Rectangle features

Tracking component

• A single hypothesis for each object• Relies on color appearance model of the

object– Each object hypothesis is computed using its

position, size, and color histogram

Event analysis component

• The role is to label all ongoing events of the vehicle given a continuous video sequence.– They are recognized by hierarchically analyzing the

relationships among the detected sub-events.– Spatio-Temporal Relationship Decision Tree.

Decision Trees

• Rules for classifying data using attributes.• The tree consists of decision nodes and

leaf nodes.– A decision node has two (or more branches),

each representing values for the attribute tested.

– A leaf node attribute produces a homogeneous result (all in one class), which does not require additional classification testing.

intermediate node

Decision Tree Example

overcast

high normal falsetrue

sunnyrain

No NoYes Yes

Yes

Outlook

HumidityWindy

feature

event

result

Entropy

Entropy =

-1*(0.5log2(0.5) + 0.5log2(0.5)) = +1

Entropy =

-1*(0.1log2(0.1) + 0.9log2(0.9)) = 0.47

Entropy: a formula to calculate the homogeneity of a sample.

Maximizes the gain

E(Current set) – E(All child sets)

Spatio-Temporal Relationship Decision Tree

elementary sub-events

car passing another

car passed by another

car is at front of another

car at behind of another

cars side-by-side

accelerating

decelerating

vehicle stopped

pedestrian in front

Describing a condition of a particular sub-event (e.g. its duration greater than a certain threshold)

Binary decision tree

Spatio-Temporal Relationship Decision Tree

• The system recognizes the sub-events using four types of features. – Extracted from local 3-D XYT trajectories.

• Time intervals of all occurring sub-events are recognized, and are provided to the system for the further analysis.– Describing a condition of a particular sub-event– A relationship between two sub-events

orientation velocity acceleration relative XY coordinate

Experiments

• Dataset of driving events

• The dataset is segmented into 52 scenes, where each of them contains 0 to 3 events.

long stoppingovertakeovertakensudden accelerationsudden stop - pedestriansudden stop - vehicle