continuous automatic identification and classification of ... · case 2: video classification of...
TRANSCRIPT
8/22/2019
1
8/22/2019 11
Continuous Automatic Identification and Classification of Trash on Street using Smartphone
I3 Conference on Aug. 19, 2019
Seon Ho Kim, Ph.D.
Integrated Media Systems Center
Viterbi School of Engineering
University of Southern California
8/22/2019 22
8/22/2019
2
8/22/2019 33
So many photos of Los Angeles downtown.How can we efficiently manage and learn from them?
8/22/2019 44
Motivation
• More and more visual data are generated
– More amount than human can physically watch machine watches
• Still, visual data collection is expensive and in adhoc manner
– Size of data, orchestration of collection
• Utilization of visual data for city, especially smart city
– Automation, efficiency, cost effectiveness
• Preparation for AI and machine learning is needed
8/22/2019
3
8/22/2019 55
Working with City of Los Angeles
• Beginning…– LA Mayor Eric Garcetti's Clean Streets initiative: monitor street
cleanliness
• Currently, data are manually collected and evaluated: inefficient, costly automate!
• In collaboration with the Sanitation Department of LA, automatically detect the cleanliness of streets as well as any special objects in need of removal.
8/22/2019 66
Case 1: Image Classification of Street Cleanliness
Images collected by the Sanitation Department, City of Los Angeles
8/22/2019
4
8/22/2019 77
Image Based Classifier
Published in IEEE Big Multimedia, 2018
• Achieved 80 – 90% accuracy (depending on class): stable and practical
• The more images in an area, the higher the accuracy becomes: promising
8/22/2019 88
Integration with I3(Intelligent IoT Integrator)
Seller: MyLA311 Mobile App (images) or Sanitation Dept. Truck (videos)
Broker: Classifier provided by IMSC
Buyer: City of Los Angeles or any Web Applications
Images need to be manually collected any way of automatic collection?
8/22/2019
5
8/22/2019 99
• 700+ garbage collection trucks are being operated
• Through four cameras (three attached on the sides on the truck and one on the front, each with a resolution of 704*480 and a frame rate of 29 fps), a truck operating for five hours (7 AM – 12 PM) generates videos whose size is around 19 GB.
• We extract one keyframe (i.e., an image) per second from videos.
Case 2: Video Classification of Street Cleanliness
8/22/2019 1010
Video Frame Examples and classification
Bulky Item Illegal Dumping
Encampment Clean
0
10
20
30
40
50
60
70
80
90
100
Bulky Item Clean Encampment Extra Vegetation Illegal Dumping
Mea
sure
men
t Sc
ore
Precision
Recall
F1 score
Video data transfer is too expensive any way to be affordable?
• Achieved comparable accuracy using the model learned from images
• For videos, higher performance is needed to be practical
8/22/2019
6
8/22/2019 1111
Case 3: Mobile Realtime Classification
Implemented and tested to simulate real time trash detection and classification with video camera on garbage collection truck: feasible!
Continuous Classification
Using Smartphone
Report result with high confidence only
Report result with high confidence only
8/22/2019 1212
Edge Computing
• Train machine learning models on the server (initial model)
• Distribute the models to edge devices (e.g., smartphones, smart cameras)
• Inference happens on edge devices using CPU on edge
• Report selected results to involved agencies
• Improve models iteratively
Framework to train, distribute and adapt models
To appear in IEEE BigMM, Sep. 2019
8/22/2019
7
8/22/2019 1313
Challenges
• Edge devices exhibit different capabilities (Hardware)
– CPU/GPU, Memory, Bandwidth
• May have power limitations (Mobile)
• May have limited availability/connectivity (Bandwidth)
• Need to consider: the model and binary size, application speed and model loading speed, performance
• Most of the existing frameworks train a single model on the server side and distribute it to edge devices
8/22/2019 1414
• Any other useful application of collected data?
• The back-camera attached on truck cab be a visual source for road damage monitoring.
– All LA streets can be monitored once a week!
Case 4: Road Damage Recognition and Classification
Real Images from LASAN
8/22/2019
8
8/22/2019 1515
IEEE Big Data Challenge Cup Competition (Dec. 2018)Classification of Road Damages
Defined by Japan Road Association [*]
[*] Maintenance and Repair Guide Book of the Pavement 2013, 1st ed. Tokyo, Japan: Japan Road Association, 04 2017.
Road damage detection from LA garbage trucks can be feasible and practical too.We won Bronze Prize (3rd place) out of 59 teams from 15 countries
Using images recorded in Japan using smartphones
8/22/2019 1616
Case 5: Identify Surface Material Beneath Graffiti
• Cleaning crew need to know what material beneath graffiti is
Metal Telephone Pole Wood Telephone Pole
Class
Wall
Brick Wall
Concrete Wall
Wood Wall
Stucco Wall
Sound Wall
Door
Metal Door
Wooden Door
Barricade Fence
Metal Fence
Wooden Fence
Fire HydrantPedestrian Furniture
Pole
Metal Pole
Wooden Pole
Granite Pole
Mailbox Sidewalk
walkway
Curb
StairsTraffic
Control BoxSign
Street Signs
Traffic Signs
Tree Windows
8/22/2019
9
8/22/2019 1717
Classes and Classification Result
To appear in IEEE Image Processing, Sep. 2019
• Fundamentally challenging problem due to multistep learning
• As a research result, achieved an excellent result (50-60% accuracy)
• But, maybe not enough to be practical yet: need to enhance accuracy further
8/22/2019 1818
Visual Analysis for Smart City
Trash
Graffiti
Parking
Road Damage
Thermal Images
(Fire)
Edge Computing
I3 as IoT Platform
Visual Data Analysis
Action
Camera as sensor
8/22/2019
10
8/22/2019 1919
Discussion• Next steps? How to use the techniques for LA?• Continue our research
– Image analysis needs more data: trash, graffiti, road damage– Sharing through I3
• Open datasets– Make available to the public: trash and graffiti
• Real world test– Mobile app (MyLA311), Smart camera on garbage collection truck
• Funding, Industry partners– Thank Oracle for providing generous cloud resources
8/22/2019 2020
Q & A
Seon Ho [email protected]