training and validating automated driving ......infrared, v2x, gps, etc. •interfaces with 3rd...
TRANSCRIPT
![Page 1: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/1.jpg)
www.tassinternational.com
TRAINING AND VALIDATING AUTOMATED DRIVING APPLICATIONS USING PHYSICS-BASED SENSOR SIMULATION
NVIDIA GTC Europe – October 11th 2017
Martijn Tideman – Product Director
![Page 2: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/2.jpg)
Confidential
Validation & Certification
Testing & Verification
Engineering & Development
Research & Concepts
TASS International supports the automotive industry in making vehicles safer and smarter by offering software and services
for development and validation of Automated Driving and Integrated Safety systems
2
TASS International: Connecting Simulation & Testing
![Page 3: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/3.jpg)
Confidential 3
Per September 1st 2017, TASS International and Siemens joined forces Offering a complete development chain for mechanics and electronics
Integrated solutions for verification and validation of automated driving systems AI and Deep Learning are key focus points
TASS International & Siemens
![Page 4: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/4.jpg)
Confidential
Testing needed to verify and demonstrate that the physical product complies to specific requirements and quality standards
(often in an emulated environment representing a subset of real-life use-cases)
Why testing?
4
Connecting Simulation & Testing
![Page 5: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/5.jpg)
Confidential
Simulation needed to make quick and cost-effective design iterations and validate the product against all relevant real-life use-cases
in an environment which is safe and offers perfectly reproducible conditions
Why simulation? Connecting Simulation & Testing
5
![Page 6: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/6.jpg)
Confidential
TASS International Simulation Solutions
World & Sensor Modelling
Environmental sensors perceiving the world
and delivering input to Automated Driving
decision & control logic
Tyre Modelling
Tyres transferring Automated Driving control commands
to the road
Human Modelling
Human drivers and passengers
traveling safely and comfortably from A to B
V2X Modelling
Receivers and transmitters
facilitating wireless communication
6
![Page 7: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/7.jpg)
Confidential
Simulation Platform: PreScanTM
World & Sensor Modelling
V2X Modelling
Environmental sensors perceiving the world
and delivering input to Automated Driving
decision & control logic
Receivers and transmitters
facilitating wireless communication
7
![Page 8: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/8.jpg)
Confidential
Main capabilities
• Easy world modeling, scenario building & import
• Extensive sensor model library
- Camera, radar, lidar, ultrasone, infrared, V2X, GPS, etc.
• Interfaces with 3rd party solutions
- Vehicle dynamics, maps, traffic, etc.
Real scenario
Virtual scenario
Virtual camera image
Workflow
Example
Physics based camera
PreScanTM Simulation Platform
Sensor models with varying fidelity
8
![Page 9: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/9.jpg)
Confidential
PreScanTM Application Examples
Adaptive Cruise Control Pedestrian AEB based on radar-camera fusion
9 Lane Keeping Assistance Parking Assistance
![Page 10: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/10.jpg)
Confidential
Deep Learning is increasingly being applied for ADAS and HAD
Almost all big OEMs / Tiers have
established dedicated teams for
Deep Learning
It is widely recognized that simulation is necessary to train HAD algorithms
Especially for the “corner cases”
(i.e. critical situations with low
probability)
Deep Learning currently mainly applied on camera data, but industry also looking at:
Using radar and lidar data
Raw sensor data fusion
10 Source: DFKI
Example: logos of companies recently
presenting about deep learning at conferences
Application area: Deep Learning
Deep learning is gaining momentum
![Page 11: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/11.jpg)
Confidential
Needed for successfully applying Deep Learning:
1. Lots of training data (e.g. camera, radar, lidar data, etc.)
• There is lots of real-world data available about high-probability cases, but insufficient real-world data available for critical situations with low-probability (“corner-cases”)
• Simulation can provide this data very easily!
2. Reference (ground truth) data: aka “labels” or “tags”
• Manually “tagging/labeling” images is an expensive and boring process (even if outsourced to low wage countries)
• Simulation solves this by providing a “free” ground-truth signal!
3. Test coverage & final validation/certification
• High Performance Clusters (HPCs) capable of running large numbers of scenarios & variations for validation purposes
• Open question: are we able to develop a virtual homologation methodology & environment?
11
![Page 12: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/12.jpg)
Confidential
Needed for successfully applying Deep Learning:
1. Lots of training data (e.g. camera, radar, lidar data, etc.)
• There is lots of real-world data available about high-probability cases, but insufficient real-world data available for critical situations with low-probability (“corner-cases”)
• Simulation can provide this data very easily!
2. Reference (ground truth) data: aka “labels” or “tags”
• Manually “tagging/labeling” images is an expensive and boring process (even if outsourced to low wage countries)
• Simulation solves this by providing a “free” ground-truth signal!
3. Test coverage & final validation/certification
• High Performance Clusters (HPCs) capable of running large numbers of scenarios & variations for validation purposes
• Open question: are we able to develop a virtual homologation methodology & environment?
12
![Page 13: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/13.jpg)
Confidential
PreScanTM Physics Based Camera (PBC)
13
PreScanTM PBC during night-time driving
PreScanTM PBC during tunnel entrance/exit
![Page 14: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/14.jpg)
Confidential
The PreScanTM Physics Based Camera offers: Full-spectrum world simulation (incl. non-visual wavelengths such as IR) Camera component models (e.g. lens, filters, imager)
14
PreScanTM Physics Based Camera (PBC)
![Page 15: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/15.jpg)
Confidential
PreScanTM Physics Based Radar (PBR)
15
Note: this is a 12s scenario, played 5x slower. The radar has a much wider field of view than the camera.
Camera image from the “radar’s point-of-view”
PreScanTM PBR simulated radar data, processed to Range-Doppler
![Page 16: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/16.jpg)
Confidential
PreScan TM Physics Based Radar (PBR) Capabilities
Multipath simulation up to any number of bounces.
Multistatic antenna configurations (MIMO).
Fully customizable waveforms (FMCW, Fast Chirp Modulation, etc).
Physical material properties, including polarization effects.
Clutter simulation.
Micro-doppler effects.
Interference between different radar sets.
Non-perfect component behaviour.
Configurable tradeoff between fidelity and performance.
16
![Page 17: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/17.jpg)
Confidential
PreScanTM LIDAR model
17
Example: PreScan LIDAR model simulating a Velodyne LIDAR sensor
![Page 18: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/18.jpg)
Confidential
Needed for successfully applying Deep Learning:
1. Lots of training data (e.g. camera, radar, lidar data, etc.)
• There is lots of real-world data available about high-probability cases, but insufficient real-world data available for critical situations with low-probability (“corner-cases”)
• Simulation can provide this data very easily!
2. Reference (ground truth) data: aka “labels” or “tags”
• Manually “tagging/labeling” images is an expensive and boring process (even if outsourced to low wage countries)
• Simulation solves this by providing a “free” ground-truth signal!
3. Test coverage & final validation/certification
• High Performance Clusters (HPCs) capable of running large numbers of scenarios & variations for validation purposes
• Open question: are we able to develop a virtual homologation methodology & environment?
18
![Page 19: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/19.jpg)
Confidential
• PreScan’s Image Segmentation Sensor (ISS) generates segmented images
• Two modes:
1. Object mode: each object gets unique ID, name, color
2. Type mode: objects are grouped according to object-type
PreScanTM Image Segmentation Sensor (ISS)
19
Camera image ISS image based on object types ISS image based on unique objects
ISS can be combined with other “reference sensors” (e.g. bounding boxes, depth cameras)
Not only for camera simulation, but also usable for radar and lidar simulation!
![Page 20: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/20.jpg)
Confidential
Image Segmentation Sensor: Example Application
20
![Page 21: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/21.jpg)
Confidential 21
Joint projects with DFKI & Siemens
Main questions:
1. Are synthetic camera images generated by PreScan suitable for training deep-learning based classifiers? What criteria do they need to comply to?
2. Does addition of synthetic images to a set of real images offer added value?
Approach:
• Training based on Convolutional Neural Networks (CNNs) for:
Image segmentation
Driving scenario classification
• Different models were trained based on real and synthetic data, mixed in various ratios
• Performance evaluated on a set of real test images using confusion matrices and Intersection over Union (IoU) criteria
Using PreScanTM data for deep learning
![Page 22: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/22.jpg)
Confidential 22
Real images from automotive camera
Synthetic images from PreScan Physics Based Camera (PBC) model
Joint projects with DFKI & Siemens
Segmented images from PreScan Image Segmentation Sensor (ISS)
Using PreScanTM data for deep learning
![Page 23: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/23.jpg)
Confidential 23
Real images from automotive camera
Joint projects with DFKI & Siemens
Using PreScanTM data for deep learning
Synthetic images from PreScan Physics Based Camera (PBC) model
Segmented images from PreScan Image Segmentation Sensor (ISS)
![Page 24: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/24.jpg)
Confidential 24
Real images from automotive camera
Joint projects with DFKI & Siemens
Using PreScanTM data for deep learning
Synthetic images from PreScan Physics Based Camera (PBC) model
Segmented images from PreScan Image Segmentation Sensor (ISS)
![Page 25: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/25.jpg)
Confidential 25
Real images from automotive camera
Joint projects with DFKI & Siemens
Using PreScanTM data for deep learning
Synthetic images from PreScan Physics Based Camera (PBC) model
Segmented images from PreScan Image Segmentation Sensor (ISS)
![Page 26: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/26.jpg)
Confidential 26
Some results & findings:
• Training based on only synthetic data yields models that don’t perform very well in the real world.
• Adding synthetic data to real training data increases the quality of the model, compared to using only real training data.
• Models based on higher number of synthetic images performed better, provided that the synthetic input is balanced against reality.
Artefacts and imperfections seen in the real world should also be present in the synthetic data (both in the environment model and in the sensor model).
Joint projects with DFKI & Siemens
Using PreScanTM data for deep learning
Note: these are first steps for PreScan in the field of deep-learning… … many more to follow!
![Page 27: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/27.jpg)
Confidential
Needed for successfully applying Deep Learning:
1. Lots of training data (e.g. camera, radar, lidar data, etc.)
• There is lots of real-world data available about high-probability cases, but insufficient real-world data available for critical situations with low-probability (“corner-cases”)
• Simulation can provide this data very easily!
2. Reference (ground truth) data: aka “labels” or “tags”
• Manually “tagging/labeling” images is an expensive and boring process (even if outsourced to low wage countries)
• Simulation solves this by providing a “free” ground-truth signal!
3. Test coverage & final validation/certification
• High Performance Clusters (HPCs) capable of running large numbers of scenarios & variations for validation purposes
• Open question: are we able to develop a virtual homologation methodology & environment?
27
![Page 28: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/28.jpg)
Confidential
Field Data Analysis
6-10-2017 28
>90% test coverage?
Simulation of “Corner Cases”
SW u
pd
ate
+ coverage of rare & critical scenarios?
Virtual homologation methodology & environment
Massive Physics Based Parametric Cluster Scenario Evaluation
Scenario Generation
Data Analysis
Mis
sin
g
Mis
sin
g
Goal: “guaranteed” 100% test coverage! basis for homologation/certification
![Page 29: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/29.jpg)
Confidential
PreScanTM simulation platform connected to NVIDIA Drive PX for verification and validation
29
PreScan PC
CAN
Injection of PreScanTM synthetic sensor data into NVIDIA Drive PX as an alternative or addition to road testing with real sensors
Virtual verification of algorithms for environmental perception
Virtual validation of control logic
Closed-loop real-time HIL simulation
PreScan synthetic
sensor data injection
The PreScanTM - Drive PX injection setup is demonstrated at our booth
![Page 30: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/30.jpg)
Confidential 30
Virtual validation of environmental perception algorithms running on the Drive PX
PreScanTM simulation platform connected to NVIDIA Drive PX for verification and validation
![Page 31: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/31.jpg)
Confidential 31
PreScanTM simulation platform connected to NVIDIA Drive PX for verification and validation
![Page 32: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/32.jpg)
Confidential
Needed for successfully applying Deep Learning:
1. Lots of training data (e.g. camera, radar, lidar data, etc.)
• There is lots of real-world data available about high-probability cases, but insufficient real-world data available for critical situations with low-probability (“corner-cases”)
• Simulation can provide this data very easily!
2. Reference (ground truth) data: aka “labels” or “tags”
• Manually “tagging/labeling” images is an expensive and boring process (even if outsourced to low wage countries)
• Simulation solves this by providing a “free” ground-truth signal!
3. Test coverage & final validation/certification
• High Performance Clusters (HPCs) capable of running large numbers of scenarios & variations for validation purposes
• Open question: are we able to develop a virtual homologation methodology & environment?
32
![Page 33: TRAINING AND VALIDATING AUTOMATED DRIVING ......infrared, V2X, GPS, etc. •Interfaces with 3rd party solutions - Vehicle dynamics, maps, traffic, etc. Real scenario Virtual scenario](https://reader036.vdocuments.us/reader036/viewer/2022062606/5fe4a349ccd72268432ae8a9/html5/thumbnails/33.jpg)
Confidential
Outlook
In addition to PreScan’s camera model, also using PreScan’s physics-based radar and lidar models to generate synthetic input for deep learning purposes
Establishing PreScan injection setups for deep-learning based on raw sensor data fusion
Using trained neural networks to automatically generate virtual PreScan scenarios and the corresponding synthetic sensor data
Using the latest HPC and GPU technologies to maximize the amount of “virtual miles” driven per hour/day/week/month/year
33