combining ros and ai for fail-operational automated driving...© virtual vehicle comet k2-center...
Post on 06-Jul-2020
0 Views
Preview:
TRANSCRIPT
© VIRTUAL VEHICLE 1
Combining ROS and AI for
fail-operational automated driving
Prof. Dr. Daniel WatzenigVirtual Vehicle Research Center, Graz, AustriaandInstitute of Automation and Control at Graz University of Technology
© VIRTUAL VEHICLE
COMET K2-Center
December 2017 Software Defined Vehicles 2
AUTOMOTIVE RAIL AEROSPACE
Shareholder:
� Gegründet: 2002
� Mitarbeiter: 204
� Umsatz: 20,3 Mio. EUR
� Standort: Graz
� Website: www.v2c2.at
Dr. Jost BernaschGeschäftsführer
Prof. Hermann SteffanWissenschaftlicher Leiter
© VIRTUAL VEHICLE
Our automated driving research activities
Embedded control and software functions
Real-time sensor fusion
Sensor self-diagnostics and fail-operational architectures
Dependable computing and reliable in vehicle control (strong multi-core expertise in both SW and HW)
Functional safety analyses (ISO PAS 21448)
Virtual prototyping of automated driving functions
Validation of real driving scenarios (real-time co-simulation)
Traffic simulation (micro, macro) / infrastructure integration
AI in both function development and vehicle operation
December 2017 Software Defined Vehicles 3
© VIRTUAL VEHICLE
Outline
� Motivating example
� The challenge “fail-operational”
� ROS and AI integration• Simulation, visualization, and communication
� AD demo car
� AI example for active safety design
� Summary
December 2017 Software Defined Vehicles 4
© VIRTUAL VEHICLE
Motivating example
Virtual Vehicle
� Research Center in Graz (about 200 employees)
� Expertise in simulation and experimental verification• Focus on Automated Driving
Dependable Systems Group
� Functional safety
� Verification and validation
December 2017 Software Defined Vehicles 5
© VIRTUAL VEHICLE
Motivating example
Motivating example – Uber car
� Safety is top priority for user acceptance of automated driving
http://orf.at/stories/2384924/2384925/
December 2017 Software Defined Vehicles 6
© VIRTUAL VEHICLE
Motivating example
Motivating example – consequences
� Achieve user acceptance• Avoid accidents and hazards (see example)
� Keep development and verification efforts reasonable• Easy to use simulation environment during development
• Verify software on different test platforms
− MiL, SIL, HiL, and vehicle
� Simple goals, hard to achieve…
December 2017 Software Defined Vehicles 7
© VIRTUAL VEHICLE
Technical challenges
Reference architecture
Sensoryprocessing
Environmentmodel
Behaviourgeneration
Valuejudgement
Sensors Actuators
Perceived objects and events
Situationevaluation
Predicted Input
Plan
State
Plan evaluation
Sensorinputs
Planresults
Update
Actions
ANSI Reference architecture of intelligent systems
December 2017 Software Defined Vehicles 8
Information and data uncertainties!
© VIRTUAL VEHICLE
Technical challenges
Vehicle technologies
� Sensory processing• Camera, RADAR, LiDAR, …
� Value judgement and environment model• High performance computing (HPC)
• Segmentation (using AI)
• Situation identification
• Behaviour estimation of other traffic participants
� Behaviour generation• Electronic Control Units (ECU)
• Vehicle control algorithms
Camera
LiDAR
RADAR
ECUHPC
Steering
Accelaration/Brake
December 2017 Software Defined Vehicles 9
© VIRTUAL VEHICLE
Limits of sensors
December 2017 Software Defined Vehicles 10
As effective sensors are, they have some drawbacks• Limited range• Performance is susceptible to common environmental conditions (rain, fog,
varying lighting conditions) • “False positives”• Range determination not as accurate as required• The use of several sensor types can ensure a higher level of confidence in
target detection and characterization
� Robust sensors and sensor self-diagnosis� Redundancy in HW and SW (“fail-operational”)� Sensor fusion (raw data? objects? hybrid?)
© VIRTUAL VEHICLE
• Use of “heat maps” (buffer where each pixel of successfully detected window adds “1”)
• Average the buffer from the last 10-20 frames
• Use only pixels that survived a certain amount of frames
• Extract component bounding blocks � vehicle(s)
Vehicle detection: false positives
December 2017 Software Defined Vehicles 11
© VIRTUAL VEHICLE
Enhancement of view
December 2017 Software Defined Vehicles 12
Austrian test region ALP.Lab
© VIRTUAL VEHICLE
Enhancement of view
December 2017 Software Defined Vehicles 13
© VIRTUAL VEHICLE
System architecture – automated driving
December 2017 14Software Defined Vehicles
plan safeactions under
currentconditions
steer functionsand vehicle
unpredictableconditions
fuse all input to
one model
Reliable sensor data processing and fusion• Raw data analysis of all sensors• Deliver consistent environmental model
Scene understanding, driver monitoring, decision making, and planning• Situation and behaviour prediction• Planning of provably safe trajectories• Handover/takeover planning
Fail-operational X-by-wire actuation (low level control)
System Performance and Driver Monitoring
• Out-of-position• Warning• Intervention• Measure reliability
and uncertainty• Detection and
decision on functionavailability
• System degradation• Sensor self-diagnosis• Ensuring fail-
operational behaviour• …
Radar LIDAR Cameras Infrared Ultrasonic V2XOther
Sensors
[Source: based on ECSEL Project RobustSense, 2015]
© VIRTUAL VEHICLE
Towards fail-operational: e.g. power steering
December 2017 Software Defined Vehicles 15
• Fail-safe (what we have now)• No emergency operation necessary
• Safe state: system off, driver immediately in control loop
• High-availability
• Safe state: system off, driver immediately in control loop
• Emergency operation is desirable but not required
• Minimize hazardous situation in case of potential misuse
• Fail-operational
• Emergency operation is required (10-15s)
• Eyes-off, brain-off
• Achieved by adding measures to all vital parts
[Source: Steindl, Miedl, Safetronic, 2015]
© VIRTUAL VEHICLE
Homogeneous vs. diversity concept
December 2017 Software Defined Vehicles 16
• Homogeneous redundancy
• uses a minimum of two equal instances in parallel
• the effort in development can be reduced due to the identical components
• because of the equality, this approach only protects against random faults caused by aging, deterioration or bit flips
• probability for a complete system crash is higher than in approaches with diverse components
• Redundancy by diversity (avionics)
• The calculating components are heterogeneous, e.g. from different manufacturers
• System SW of each unit is different or uses at least different HW components
• this system SW diversity complements functional diversity of the application SW
• Different implementations result in a lower probability of failure for the system due to the lower probability that the diverse components show the same misbehavior at the same time
© VIRTUAL VEHICLE
Fail-operational architecture
• EC Project “PRYSTINE” (2018 to 2021, Programmable systems for intelligence in automobiles)
• Infineon, Scania, Ford, BMW, Virtual Vehicle, TU Graz…
December 2017 Software Defined Vehicles 17
© VIRTUAL VEHICLE
Fail-operational architecture
• Massive redundancy will not be the solution (2x and 3x, STANAG 4626)
• We need “smart” solutions!
• Mechanism for HW fault detection (e.g. BIST, built-in self tests)
• HW extensions for predictive diagnosis
• Reconfiguration strategies (isolation of faulty function and “shift of functions”)
• Degradation strategies (critical vs. non-critical functions)
• …
December 2017 Software Defined Vehicles 18
© VIRTUAL VEHICLE
PRYSTINE
December 2017 Software Defined Vehicles 19
© VIRTUAL VEHICLE
ROS Introduction
December 2017 Software Defined Vehicles 20
© VIRTUAL VEHICLE
Robot Operating System (ROS)
� Open-source, meta-operating system for robots• Originally developed by Stanford AI Lab and Willow Garage in 2007
• Maintained by the Open Source Robotics Foundation (OSRF)
• Runs on top of e.g. Ubuntu/Linux
� Designed for many kinds of robots• Provides tools for building/running code
− Including software libraries
• Hardware abstraction
• Allows for low-level device control
• Provides communication system
ROS Introduction
December 2017 Software Defined Vehicles 21
© VIRTUAL VEHICLE
ROS Introduction
ROS Communication
� Peer-to-peer communication
� Central “service” registration
� Supports UDP and TCP
Advantages
� Flexible configuration
� Fast communication
� Simple distribution of functions on computationplatforms
December 2017 Software Defined Vehicles 22
© VIRTUAL VEHICLE
ROS Introduction
Integration of common solutions
� Sensor data processing
− Library for 2D/3D image and point cloud processing
− Filtering, feature detection, …
− Tracking on camera data
� Motion planning• Support of multiple planning algorithms
� 3D simulation and visualization• Dynamics, kinematics, sensors, …
� Data transformation• Coordination system transformations for sensor data
December 2017 Software Defined Vehicles 23
© VIRTUAL VEHICLE
ROS Introduction
ROS Simulation
� 3D environment
� Sensor simulation• Laser scanner built in
• Camera available
• Extension possible
� Vehicle simulation• Position
• Orientation
• Speed
• Steering
• …
December 2017 Software Defined Vehicles 24
Sensorsimulation
© VIRTUAL VEHICLE
ROS Introduction
Driving Simulation
� Detailed street layouts• Number of lanes
• Street markings
� Traffic management• Traffic signs and signals
� Different weather conditions• Sunshine, snow, rain, …
� Sensor simulation• Multiple sensor types available
• Extension possible
December 2017 Software Defined Vehicles 25
https://vires.com/vtd-vires-virtual-test-drive/
Kom
ple
xe S
traß
enve
rläufe
Änderu
ngen im
Str
aß
enve
rlauf
Ändern
de U
mw
eltb
edin
gungen
Unte
rsch
iedlic
he
Str
aß
enbesc
haff
enheit
© VIRTUAL VEHICLE
ROS – Function Implementation
AI Integration
Simulation
� Exchangeability of simulation and real environment• Replacement of sensor data format
� Software deployment can be freely chosen
December 2017 Software Defined Vehicles 26
DrivingSimulation
Real worlddata
Sensor dataprocessing
Behaviourgeneration
Segmentation(AI function)
Decisionmaking
ROS –Simulation
ROS –Converter
ROSmessages
HPC #1 HPC #2 ECU
Sensordata
available available
© VIRTUAL VEHICLE
AI inference
Standard Software
AI Integration
December 2017 Software Defined Vehicles 27
AI function control flow
- Read image
- Copy to GPU memory
- Matrix operations(multiply and add)
- Mass non-linear function(ReLU, sigmoid, …)
call
return result
CPU GPU
- Read result
- Send message
ROS specific part can be implemented “as usual”
© VIRTUAL VEHICLE
Our automated drive test vehicle
ROS Deployment
� Use ROS function implementation in vehicle• No change of SW needed
� Integrated sensors• Six radar sensors
− Four long range, two short range
• 6 Cameras
− Front, two side, back
• One Mobile Eye
� Full vehicle control• Steering, acceleration, brake, …
December 2017 Software Defined Vehicles 28
Virtual Vehicle – Automated Drive test vehicle
Virtual Vehicle – AD test vehicle trunk
© VIRTUAL VEHICLE
Active safety design using ML techniques
December 2017 Software Defined Vehicles 29
© VIRTUAL VEHICLE
Motivation: Development process active safety systems
December 2017 30
Problem: Complexity and high variation of accidents
� Safety functions for every combination of accident type and cause
� Function design based an quantitative (e.g. DESTATIS) and qualitative accident databases (e.g. GIDAS Pre-Crash-Matrix)
Software Defined Vehicles
First 50%: 26 types and causes of accidents
Last 50%: 5287 types and causes of accidents
An exponential effort in the development of individual active safety systems
is required to cover only significant increase in the addressing of accidents!
Type:
Cause: speeding, slippery road, etc
© VIRTUAL VEHICLE
Function development based on real world data
December 2017 31
Method analysis: From the accident to the active safety system
� Accident recording by highly automated vehicles (SAE Level 3)
Data recording: Course of the ego vehicle and other traffic participantsroad geometry, driver behavior
1) Evaluation of active safety systems with recorded traffic data
2) Learning the function behavior of active safety systems directly from recorded data
Software Defined Vehicles
[Here] [Kostal][BMW]
Backend
� 360° environment recognition� High-precision digital maps incl.
localization� Driver monitoring� Backend communication
[Source: BMW and Virtual Vehicle, cooperative R&D project, 2017]
© VIRTUAL VEHICLE
Vision: Optimization based on real world traffic data
December 2017 32Software Defined Vehicles
� 360°sensors� High-precision maps � Driver monitoring� Backend
Crossing scenarios Total scenarios Accidents
Training data 1840 242
Test data 1829 243
Project use case: Crossing pedestrian (75% of all pedestrian accidents)
� Generated pedestrian scenarios from the effectiveness analysis
© VIRTUAL VEHICLE
Function development active safety system
December 2017 Software Defined Vehicles 33
© VIRTUAL VEHICLE
Simulation results: False positives vs. speed reduction
December 2017 Software Defined Vehicles 34
Neural Network
0 5 10 15 20 25 30 35 400
10
20
30
40
50
60
70
80
90
100
False positives
Sp
ee
d r
ed
uct
ion
[%]
Reference algorithm
NN
NN (Extended Features)
NN (Reduced Training Data)
NN (Extended Features / Reduced Training Data)
0 5 10 15 20 25 30 35 400
10
20
30
40
50
60
70
80
90
100
False positives
Sp
ee
d r
ed
uct
ion
[%]
Reference algorithm
RF
RF (Extended Features)
RF (Reduced Training Data)
RF (Extended Features / Reduced Training Data)
Random Forest
Variation by the algorithm evaluation:1) Feature set (feature variation)
2) Training data (reduced speed range pedestrian)
Speed reduction
Reference Implementation 73.7%
Random Forest 83.4%
Neural Network 92.0%
© VIRTUAL VEHICLE
AI-based systems – challenges to be faced
December 2017 Software Defined Vehicles 35
• Requirements engineering (Machine Learning algorithms)
• Identification of Key Performance Indicators for ML algorithms• e.g. accuracy (fault tolerances but also mis-detection), speed, etc.
• HW requirements for different DNN structures
• Identification of relevant safety analyses according to ISO 26262 and ISO PAS 21448• Determination of safety measures, time tolerances for detection etc.
• How to measure code and structure coverage of DNNs?
• Selection of training and test data
• Work out criteria based on statistical methods, quality, and versatility
• SW unit tests and integration test
• Verification and validation of SW safety requirements
© VIRTUAL VEHICLE
Summary
� Automated driving requires redundancy (SW and HW)
• Fail-operational architectures � ISO PAS 21448
• Minimum redundancy but maximum reliability
• Homogeneous redundancy vs. redundancy by diversity � trade-off
� AI functions have already proven their usefulness• Object detection and localization (segmentation)
• End-to-End driving vs. modular approach
• AI in function development
� ROS is already widely used
• Sound basis for rapid prototyping
• Seamless transition from simulation to real hardware
• ROS is a suitable development and test environment for AI functions
December 2017 Software Defined Vehicles 36
© VIRTUAL VEHICLEDecember 2017 Software Defined Vehicles 37
Prof. Dr. Daniel WatzenigEmail: daniel.watzenig@v2c2.at
Thank you for your attention.
top related