challenges in real-time event detection in...
TRANSCRIPT
![Page 1: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/1.jpg)
Advanced
Technology
Labs – Brasil
Challenges in real-time event detection in video
Leonardo Nunes
![Page 2: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/2.jpg)
Objective
• Motivation on real-time event detection in video
• Challenges when analyzing real-world data
• Some lessons learned
• Future research directions
![Page 3: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/3.jpg)
Advanced Technology Lab, Brazil
ATL CAIRO
ATL EUROPA
MSR CAMBRIDGE
MSR INDIA
MSR ASIA
MSRSTATION Q
MSR REDMONDFUSE LABSRESEARCH
CONNECTIONS
ATL ISRAEL
MSR NEW ENGLAND
MSR NEW YORK
ATL BRAZILATL Brazil – Rio de Janeiro, Brazil
Focus:• Real time Video Analytics
![Page 4: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/4.jpg)
Motivation
Source: 1 IHS Technology; 2 MarketsandMarkets
Video analytics market will
grow from US$1.5B to
>US$4B/year in 5 years
(>20% CAGR) 2
![Page 5: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/5.jpg)
Event Detection
• Event/Action recognition:
• Triggered when something of interest happens in a video
• Caused by either a human, vehicle, or an object
• Major applications:
• Surveillance
• Mobility
• Retail
![Page 6: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/6.jpg)
Constraints
Real-time Cloud-based COTS sensors
![Page 7: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/7.jpg)
Microsoft Cognitive Services
Cognitive Services API Collection
![Page 8: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/8.jpg)
Applications
Additional Scenarios:
Retail Analytics
Automotive Safety
Industrial Safety
Home Automation
Customer Recognition
Traffic Flow Measurement
Satisfaction Assessment
Fire/Smoke/Flood Detection
People Counting
Automated Media Indexing
Parking Analytics
Natural Disaster Detection
Vandalism Detection
Surveillance Scenarios:
Accident Detection
Asset Protection
Crowd Management
Perimeter Intrusion
Traffic Event Detection
Unattended Object Detection
Person-of-Interest Detection
Suspicious Activity Detection
Retail Loss Prevention
![Page 9: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/9.jpg)
Example Applications
People Vehicles Objects
![Page 10: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/10.jpg)
Challenges
![Page 11: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/11.jpg)
Video Quality
• Low-quality sensors
• Severe compression artifacts
• Video information is not reliable!
• Repeated frames
• Upsampled images
![Page 12: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/12.jpg)
Small Targets
• Usually small targets
• Example:
• Motorcycles
• Single camera
• Off the shelf detectors might not work
![Page 13: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/13.jpg)
Enviromental Conditions
![Page 14: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/14.jpg)
Events of Interest are Rare
• Events of interest are rare
• E.g.: crimes, accidents, person identification
• No training example available
• False alert rate vs Probability of detection
• Example (mobility):
• Jaywalking: 3 events/hour
• Motorcycle stopped: 0.2 events/hour
![Page 15: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/15.jpg)
Infrastructure Issues
• Variable bit-rate
• As low as 200 kbps
• Network issues
• Camera movement
• Variable CPU time
• ... all leading to lost frames
![Page 16: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/16.jpg)
Some Lessons Learned
![Page 17: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/17.jpg)
Pipeline System
Target Generation
Object Tracking
Event Detection
Property Estimation
Frames EventsTargets Tacks Tacks
![Page 18: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/18.jpg)
Reliable Detectors
• Low-quality / small objects
• Robust background subtraction algorithm
• Better quality / larger objects
• Deep object recognition algorithms
• Most expensive component!
• Not executed every frame: new targets are infrequent
• Only when and where there is motion
![Page 19: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/19.jpg)
Tracking by Detection
• Object detector executed:
• Old tracks are updated with new targets
• Similarity: distance and appearance
• Object detector not executed:
• Tracks updated using Kernelized Correlation Filter
• Net effect: not every pixel is analyzed every frame
![Page 20: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/20.jpg)
Ancillary information helps
• Perspective calibration
• Unsupervised learning of target sizes
![Page 21: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/21.jpg)
Need to monitor network/hardware conditions
• Scene conditions detection:
• Anomalous luminosity
• Number of discarded frames
• Camera movement
• Need paths to halt and re-initialize different components
![Page 22: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/22.jpg)
Future
![Page 23: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/23.jpg)
Video vs Image Algorithms
• When to use image vs video detectors?
• Deep object recognizers for video
• Improvements for small objects
• End to end learning
![Page 24: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/24.jpg)
Unsupervised adaptation
• When to run target detectors?
• How to adapt detectors to camera?
• How to adapt tracker parameters?
• How to integrate deep architectures into the pipeline?
![Page 25: Challenges in real-time event detection in videogibis.unifesp.br/sibgrapi16/eproceedings/upload/wia/it2.pdfVideo analytics market will grow from US$1.5B to >US$4B/year in 5 years (>20%](https://reader035.vdocuments.us/reader035/viewer/2022070923/5fbc033150783102bb5f82c0/html5/thumbnails/25.jpg)
Summary
• Lots of challenges
• Specific tools, models for different tasks
• Lack of data, heuristics are still needed
• Unsupervised adaptation for different conditions