PowerPoint

SakuraSensor: Quasi-Realtime Cherry-Lined Roads Detection through Participatory Video Sensing by CarsShigeya Morishita, Shogo Maenaka, Daichi NagataMorihiko Tamai, Keiichi Yasumoto, Toshinobu Fukukura, Keita SatoNara Institute of Science and TechnologyDENSO CORPORATION

Thank you chairperson.Good afternoon, everyone.My name is Shigeya Morishita from Nara Institute of Science and Technology.I am very happy to see all of you today.Today, I would like to present our research named sakura sensor.1

Latest car navigation systemsHelp drivers search comfortable & efficient routes

CriteriaTraveling distanceTraveling timeToll/Toll-freeFuel efficiencyScenic beauty2

NAVITIME(http://products.navitime.co.jp/function/2519.html)Toll-freeFuel-efficientMinimum distanceToll Scenic

Latest car navigation systems help drivers with comfortable and efficient driving. With these systems, we can search routes by various () criteria.Among these criteria, we focus on scenic beauty.2

Scenic route searchProblems of existing servicesInformation is edited manuallySmall number of scenic spotsLow update frequencyScenery information consists of only texts and imagesinsufficient for users

3Our approachUse participatory sensing by carsCollect and share videos of scenic spotsExample of scenic spot info.

However, existing scenic route search services have some problems.First, information is edited manually.Second, scenery information consists of only texts and images.To solve these problems, approach, we use participatory sensing by cars and automatically collect and share videos () of scenic spots.3

Related work4MethodProposed methodParkNet [12]SignalGuru [15]Nericell [3]Participatory sensingCooperative sensingReal-timeInformation detection from videos (ultrasound signals)(traffic signals) (horn sounds)

[12] ParkNet: Drive-by Sensing of Road-Side Parking Statistics, MobiSys10[15] SignalGuru: Leveraging Mobile Phones for Collaborative Traffic Signal Schedule Advisory, MobiSys11[11] Nericell: Rich Monitoring of Road and Traffic Conditions using Mobile Smartphones, SenSys08Many existing studies on participatory sensing (PS) by carsNo studies use both PS and real-time video sensing

There are many existing studies on participatory sensing by cars.However, as long as we know, no studies use both participatory sensing and real-time video sensing.

4

SakuraSensor: automatically identifies scenic spots location and collects videos using PS

we target cherry-lined roads automatically collect and update scenic information gathering videos of scenic locationThe best period of flowering cherriesis short and uncertain from year toyear and from place to place

We propose Sakura Sensor, which automatically identifies scenic spots location and collects videos using participatory sensing.SakuraSensor targets flowering cherries called SAKURA in Japanese, sincethe best period of flowering cherries is short and uncertain from year to year and from place to place.So, up-to-date information is mandatory.5

SakuraSensor App for iOS devices6Full size video - https://youtu.be/2pRfDS7DeAcDemo at Hall C No.20

We have developed SakuraSensor application for iOS devices.Ill show a demo video of sakurasensor.We are also demonstrating SakuraSensor at hall C number twenty.

6

Key Idea7

Cloud

Cars with SmartphoneToo much cost for cellular bandwidth & computation resource at cloudRecording videoAnalyzing & sharingvideo with cherriesUpload whole recorded video

Recording videoAnalyzingvideoUpload only video with flowering cherriesSharingvideo with cherries

One possible approach to realize SakuraSensor is as follows. Cars with smartphone record videos and upload the whole recorded video to cloud server for analysis and sharing.However, this approach takes too much cost for cellular bandwidth and computation resource at cloud.The key idea of SakuraSensor is analyzing video at smartphones so that only video with flowering cherries are uploaded to the cloud and shared.

Challenges and key ideas of SakuraSensor

7

Technical ChallengesTC1: Real-Time flowering cherry detection by smartphone

TC2: Efficient load distribution among cars8

We have two technical challenges to realize SakuraSensor.First challenge is how to realize real-time flowering cherry detection by smartphone.Second challenge is how to realize efficient load distribution among cars.

8

TC1: Real-time Cherry DetectionEmploy simple computer vision techniquesSmart phone has lower computation power than PC/Cloud

Basic approachCount cherry-like color pixels in each image Identify amount of flowering cherry as cherry intensity

Problem to solveArtificial objects with similar color must be removed9

For the first challenge, we employ simple computer vision techniques since smart phone has lower computation power.So, our basic approach is just to count chery-like color pixels in each image and identify amount of flowering cherry In each image called cherry intensity ().Here, the problem to solve is that artificial objects with similar color must be removed.9

Step1: Removing Artificial Objects

An input imageBinary image after edge detection

box counting method [5]fractal dimensions10Employ fractal analysisNote: natural objects has higher fractal dimension

To remove artificial objects in each image, we employ fractal dimension analysis.Here, note that natural objects has higher fractal dimension. So, to an input image, we apply edge detection algorithm, and box counting methodto calculate fractal dimension of each square region.10

Real-time fractal dimension calculation11Red regions show natural objects

This is Real-time fractal dimension calculation.Here red color regions show natural objects.11

Step2: Detecting Cherry by Color Analysis

12Used 148 regions extracted from various scenesCreated color histogram of flowering cherry in HSV color space

Then we detect flowering cherry by color analysis.We created color histogram of flowering cherry in HSV color space. Here, we used 148 regions extracted from various scenes.These are part of the regions.12

HSV color spaceHHueSSaturationVValue of Brightness

From http://en.wikipedia.org/wiki/HSL_and_HSV

characterizes the color significantly varies depending on the lighting condition13Our approachused only H-S color space

HSV color space consists of Hue, Saturation and Value of brightness.From preliminary experiment, we found that V significantly varies () depending on the lighting condition. So, we used only H-S color space.13

H-S histogram for flowering cherry

HS01790255

Created from total of 148 cherry regions

The value at each coordinate is normalized between 0 and 114

This is the H-S histogram created from 148 cherry regions.The value at each coordinate is normalized between 0 and 1.

14

Step3: Calculating cherry intensity of an image

HS00Pixels (H, S)=(30, 20)

The value of (30, 20) is 0.816

An input imageCherry intensity = average value of all pixelsUse Backprojection method [6]

Then we calculate cherry intensity of an image by using backprojection method.For each pixel, the value is retrieved in the H-S histogram Finally, cherry intensity of the image is calculated as the average value of all pixels.15

Real-time cherry intensity calculation16Red boxes show high cherry intensity regions

This is real-time cherry intensity calculation.Here, red boxes show high cherry intensity regions.16

TC2: Load Distribution among CarsWhen all cars always conduct image analysis & uploadstoo much cost (battery consumption, bandwidth, etc)

Possible approacheach car senses at a fixed intervalmay miss PoI (cherry locations)17

The second technical challenge is load distribution among cars.When all cars always conduct image analysis and uploads of videos, the cars will take too much cost.Possible approach is that each car senses at a fixed interval.However, it may miss PoI.17

k-stage sensing18location where sensingis performed

Narrows sensing interval step-by-step when new PoI is found

Fixed interval(1st stage)PoI is detected!The preceding car

So we propose k-stage sensing which narrows sensing interval step-by-step when new PoI is found by preceding cars.This is the example of k-stage sensing.The preceding car travels and sensing is performed at an initial fixed interval.18

k-stage sensing19

Shorter Interval(2nd stage)

PoI is detected!Sensing is performed in this Radius

PoI detected by preceding carThe following car traveling the same roadNarrows sensing interval step-by-step when new PoI is found

location where sensingis performed

After that, when a following car enters the same road.The car narrows its sensing interval and radius, respectively, because a PoI is found on the road.Then, this car performs sensing at the shorter interval while the car is in the circle centered at the PoI with radius. 19

Evaluation of SakuraSensorInvestigate effectiveness of cherry intensityCompare the results of manual classification and automatic classification by cherry intensity

Videos

manual classification(used as ground truth)classificationby cherry intensityCompute accuracy by comparison 20

We conducted some experiments to evaluate Sakura Sensor.The first experiment is to investigate the accuracy of cherry intensity.We compare the result of manual classification and automatic classification by cherry intensity.20

Videos used for experimentsRecorded videos in 8 different scenes (routes) using SakuraSensor app for iOS by multiple carsscene namedatevehicleareaLength (min.)S1Mar. 31V1Aichi Pref.17S2Apr. 5V2Nara Pref.12S3Apr. 10V2Nara Pref.66S4Apr. 10V3Nara Pref.261S5Apr. 10V4Nara Pref.186S6Apr. 11V1Gifu Pref.72S7Apr. 12V2Osaka Pref.137S8Apr. 18V1Aichi Pref.89

extracted 1-second videos at random starting time from each scene21

We recorded videos in 8 different scenes using SakuraSensor application for iOS devices by multiple cars.We extracted 1-second videos at randomly selected starting time from each scene.

21

1-Second Videos Manual ClassificationClass nameCriteriaC1cherry ratio (in image) < 5%C25% cherry ratio < 25%C325% cherry ratio

SceneC1C2C3S1791710S2931017S3372433S416139645S5116760S62614772S788810S8521107Total4994230154

22Classification results with the same decision by two persons were used

We defined three classes where C1s cherry ratio in image is less than 5%, C2 between 5 and 25%, C3 more than 25%.Here, only classification results with the same decision by two persons were used.22

Videos of each class23

C1 (ratio < 5%)C2 (5% ratio < 25%)C3 (25% ratio)

These are example Videos of class C123

Evaluation Methodology

Dividing videos of each class into halvesTraining setTest set24Set of 1 second videos

Manual classification by human

First, we divided the set of classified videos in each class to the training set and the test set.

24

Evaluation Methodology

Training setTest setMedian of cherry intensity: M1 0.00033Median of cherry intensity: M2 0.00791Median of cherry intensity: M3 0.03326

ViCherry intensity

A video25

Then, from training set, we calculated median of cherry intensity for each class.Using the median values, 1-second videos in the test set are automatically classified.25

Classification Accuracy (1-second videos)26

precision recall

0.970.900.740.830.240.65

This figure shows the classification result by Sakura Sensor. We see that a good classification result is obtained for class C1 and C3 videos. On the other hand, for class C2 videos result is not so good.The main reason is that many videos included in class C1 were classified to class C2.26

Evaluation of k-stage sensing27

Simulation by 600 cars (k=3, 300m150m50m)smaller sensing times similar PoI discovery rate

We also evaluated the effectiveness of 3-stage sensing method.Evaluation was done with simulation by 600 cars.These results show the k-stage sensing achieves good PoI discovery rate with smaller sensing times.

We just adopted hulistic. Or empilically

27

ConclusionsSakuraSensorParticipatory video sensing system by carsConsisting of two key techniques

Flowering cherry detection by in-vehicle smartphoneColor histogram analysis for identifying cherry-blossomsFractal dimension analysis for removing artificial objects other than flowering cherryCherry detection accuracy (C3) with 0.7 of Precision and 0.8 of Recall

k-stage sensingDistribute sensing load among carsSimilar PoI discovery rate with about half sensing times compared with the fixed interval sensing method28

ConclusionsWe proposed sakura sensor which is a Participatory video sensing system by cars.As two key techniques, we proposed flowering cherry detection by in-vehicle smart phone andK-stage sensing.

this system consisting of two key techniquesFirst is flowering cherry detection by in-vehicle smartphone.Color histogram analysis for identifying cherry-blossomsFractal dimension analysis for removing artificial objects other than flowering cherryCherry detection accuracy with 0.7 of Precision and 0.8 of RecallSecond is k-stage sensingthis method distributes sensing load among cars.Similar PoI discovery rate with about half sensing times compared with the fixed interval sensing method.28

29Thank you!Demonstration at Hall C No.20

We also demonstrate our system.Please wat

29