object detection based on virtually trained ... - tesis.de · ©tesis gmbh dipl.-ing. ronnie...
TRANSCRIPT
© TESIS GmbH ◼ www.tesis.de
Dipl.-Ing. Ronnie Dessort – Senior Simulation Consultant at TESIS GmbH
Autonomous Vehicle Software Symposium, Stuttgart (June 6, 2018)
Object detection based on virtually trained neural networks
2
Motivation
1https://medium.com/@Synced/the-humans-behind-artificial-intelligence-3ff578cfcc60
1. Reduce manual work by using virtual ground truth data.
◼ AI algorithms need labelled data as ground truth for training.
◼ This work is tedious and costly: a person labels 10-40 images
per day1.
→ Increased frontloading for object detection by using virtual
objects
2. Complex traffic scenarios are crucial for testing vehicle
automation.
◼ Real-world test drives are expensive and inefficient for early
development stages.
◼ Small scenario numbers and deterministic behavior can
introduce an undesired bias.
→ Increased frontloading by testing in complex virtual traffic
scenarios
3
◼ Resources for training and testing
◼ Virtual training approach
◼ Evaluation of different detection models
◼ Testing object detection in complex virtual
traffic scenarios
◼ Summary & Conclusion
Agenda
4
Resources for virtual object detection training and testing
DYNA4
Simulation framework Visualization
DYNAanimation
Object catalogue for training Arbitrary scenes for testing
Publicly available image databases
for object detection and recognition
1https://arxiv.org/pdf/1405.0312.pdf2http://www.cvlibs.net/publications/Geiger2012CVPR.pdf3http://host.robots.ox.ac.uk/pascal/VOC/pubs/everingham15.pdf4https://arxiv.org/pdf/1409.0575.pdf5https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research
1
2
3
4
… and many more …5
▪ Vehicles
▪ Pedestrians
▪ Wildlife
▪ Road signs
▪ Buildings
▪ Vegetation
▪ Weather
▪ … and much more …
How to link real
and virtual
world?
5
Virtual training approach
Creating images under clinical conditions
En
vir
on
men
t
Automatic image database
generation by varying
▪ object type:
sedan, sports car, SUV, etc.
male, female, adult, child
▪ object color:
red, green, blue, etc.
different clothing (e.g.
doctor, construction worker)
▪ zoom:
object details, full size,
stamp size
▪ perspective:
360 degree panoramic view,
inclined camera
environment without relevant
objects to provide negligible
image background information
Automatic generation of annotation files based on
PASCAL VOC definition
Veh
icle
sP
ed
estr
ian
s
Training and
evaluation
with
TensorFlow
class:
Vehicle
class:
dontcare
class:
Pedestrian
6
◼ All self-trained models …
◼ … are based on Faster R-CNN + ResNet-101
◼ … used a pre-trained model (KITTI dataset) as initial solution
◼ … were trained for > 300,000 epochs
◼ Global training database consists of 14,568 virtual images:
◼ 4,488 vehicle images (14 types from sports car to SUV)
◼ 5,400 pedestrian images (12 types of male/female child/adult)
◼ 4,680 environment images (urban, rural, parking garage)
◼ Investigation of models trained on and detecting various sets of classes:
◼ V only based on vehicle images
◼ VE considering also environment images
◼ PE recognizing pedestrians in arbitrary environments
◼ VPE recognizing vehicles and pedestrians in arbitrary environments
Detector models
https://arxiv.org/pdf/1506.01497.pdf
7
◼ Detector model based on training with virtual data of Vehicles
Results of detector model „V“
Virtual test set data Validation with realistic scenes
8
◼ Detector model based on training with virtual data of Vehicles and Environment
Results of detector model „VE“
Virtual test set data Validation with realistic scenes
9
◼ Detector model based on training with virtual data of Pedestrians and Environment
Results of detector model „PE“
Virtual test set data Validation with realistic scenes
10
◼ Detector model based on training with virtual data of Vehicles, Pedestrians and Environment
Results of detector model „VPE“
Virtual test set data Validation with realistic scenes
11
Testing detectors in a virtual environment
Creation of virtual ground truth data with DYNAanimation
◼ Intrinsically available information: object class per pixel of each frame
◼ Semantic segmentation can be used for creating image annotations
automatically
◼ Scenes with arbitrary complexity can be analyzed and relevant objects
marked with ground-truth labels
Complex traffic scenarios are crucial for testing advanced algorithms for vehicle automation
Solution: Co-Simulation of DYNA4 and SUMO1
1http://sumo.dlr.de/
Change of seed
and
environment settings
12
◼ Traffic sign detection and recognition is influenced by
◼ weather conditions → fog, rain, snow
◼ color fading → long sun and rain exposure
◼ illumination variations → daytime sunshine vs. car light at night
◼ vehicle motion → image blur
Outlook: road sign detection
Apply weather
and lighting
13
Summary & Conclusion
DYNA4 SUMO
DYNAanimation
TensorFlow
◼ Object detection with deep neural networks
◼ Training with virtual data avoiding manual labelling
◼ Creation of virtual ground truth data for evaluation of
detection quality
◼ Co-Simulation of the virtual vehicle in virtual traffic
with complex traffic scenarios
◼ One-click scenario variation for stochastic, but
reproducible testing
Validation of conventionally trained object
detection in virtual traffic and environment &
virtual training for use with real-world data
Facilitate frontloading for ADAS/AD functions
by virtual testing
14
Thank you for your attention!
Object detection based on virtually
trained neural networks.
For more information visit our booth #AV801
and our talk
Virtual testing by coupling system simulation with SUMO traffic flow simulation
at theTest & Development Symposium (today at 17:30)
Dipl.-Ing. Ronnie Dessort
e-mail: [email protected]
phone: +49 89 747377-58