adas/ad - nvidia · avl drive for adas kpi sc vehicle under test proving ground very close to the...

AVL List GmbH Tobias Düser | | 11 Oktober 2018 | 1Public

Technical• Functional safety & Reliability in all

conditions (Traffic, Weather & Road)• Infinite high number of test cases• Complex and new technology• Time & Cost pressure during development

Public relations• Public & politic reactions in

case of accidents• Customer acceptance• Cyber security• Ethic

Legislative• Homologation criteria• Responsibility in case of accident

The rocky road to

ADAS/ADDevelopment, Validation and Homologation


Level 5 Future

Now

Responsibility

Dri

ver’s

Lic

ense

Regula

tions

?

?


“The critical path to introduce autonomous driving vehicles will not be the technology but the development of a metric which empowers for an approval”

Prof. Dr. rer. nat. H. Winner, November 2013

A lot of progress concerning the

TECHNOLOGYis visible, but:

… and the next question will be how to execute scenarios, test cases etc. of this metric in a limited time


“The biggest hurdle is validation to confirm that the system does not cause failures. One has to execute 250 million test kilometers”

Bosch Executive Director.

Same approaches as 36 years ago?

Mercedes-Benz 190 E (1982) - Demonstration and advantages of ABS

New and Innovative Approaches in

Testing and Validation

are necessary!


Facing the Challenges of Development and Homologation of Autonomous Driving Using Virtual

Approaches

In cooperation with research partners

Driving Software and Digital Driver

Legislation, Regulations and

Conformity

Vehicle Validation and Self Driving

Vehicle

§


Background and Homologation

Dr. Houssem Abdellatif

Global Head Autonomous Driving and ADAS

Driving Test: Human vs. Machine

718-10-11TÜV SÜD Auto Service GmbH | GTC Europe 2018

Human Machine

Level 3:

Highly

automated

Level 4:

Fully

automated

Level 5:

Driverless

Level 0:

Driver Only L1: Assisted

L2: Partially

automated

!

Today

Responsibility

Tomorrow

http://www.onlinewahn.de/generator/f-maker.htm http://www.onlinewahn.de/generator/f-maker.htm

Homologation and Type Approval


Self certification vs. type approval 3rd party principle

DefinitionHomologation refers to the certification process of a product (vehicle) granting that it complies with all local standards and legal regulations such as safety and environmental regulation.

No homologation No CoC No sales

Type

Approvalconcept

phase

vehicle development cycle

X

• European Union: Directive 2007/46/EC Type approval, tests are based on

United Nations Economic Commission for Europe (UN/ECE) procedures;

• North America: Federal Motor Vehicle Safety Standards (FMVSS)

regulations released by the NHTSA;

• Australian Design Rules (ADR) regulations;

• Japan follows UN/ECE regulations and their own Test Requirements and

Instructions for Automobile Standards (TRIAS) regulations;

• Other countries that accept or base their own regulation on those mentioned

above, following the latest release or previous versions of the regulations.

Type Approval in vehicle development• Last step of

development• Accomplishment of the

v-cycle• legal and technical

approval of the concept

UN/ECE-Regulation 140: Homologation of ESC Systems


Sine with Dwell (SwD) maneuver

© Pearson Education, Ltd., Upper Saddle River, NJ 07458

Approval testing & collecting data

real TestsVehicle

response

Vehicle

responsesimulation

Inputs

Simulation

results

correlationmodel

adjustment

Model validationApproval & assessment of variants by

using validated simulation model

Inputs

N Simulation resultsN vehicle/model variants

„Where a vehicle has been physically tested in accordance with […],

the compliance of versions or variants of that same vehicle type may be demonstrated by a computer simulation […].”

Generate the vehicle simulation

and correlate the obtained data

02Define vehicle representative of

the type to be homologated, and

test it under dynamic maneuvers

01

Simulate critical variants

03

ESC vs. Automated Driving


Sense and control of internal vehicle

Sense and control in external environment

✓ Similar homologation / validation architecture:

▪ Environment

▪ Sensor – Function / active system – Vehicle (dynamics)

▪ (Driver)

✓ Methodology can be reused and extended to automated driving

Simulation aided / supported homologation of automated driving functions

vehicle dynamicsdriver

active system

environment

information

physics

perception

interpretation

reasoning

acting

executing

driver

vehicleInternal sensors

for vehicle states

vehicle dynamicsdriver

active system

environment

information

physics

information

perception

interpretation

reasoning

acting

executing

driver

vehicle

external sensors

paradigm shift

Combining Tools (Tool Chain) for Concise Approval


Scenario data base• description• parameters• fail/pass criteria (KPI)

subset for approval

subset for physical testing

subset for simulation testing

real vehicle

virtual vehicle

results: m vehicle variants X nscenario variants X …

test track

simulation

test bench

subset for testbench testing

Virtual-based Assessment for L3+AVs


Requirements

LegislationRegulations

Norms

AV System Specification

Assessment Database:

Scenario, KPIsand Criteria

Trustworthiness of Simulation via Validation

Tool ChainValidation

Vehicle Model Validation

World Model Validation

Sensor & PerceptionValidation

Evaluation

Methods

KPIs

CriteriaTesting

SimulationFramework

Proving GroundFieldTests

Assessment Selection

SafetyKnow-How

CriticalityAnalysis

FunctionalDecomposition

18-09-28

AV Assessment

Component System Vehicle


Virtual Testing Methods

Prof. Bernhard Schick

Professor for Vehicle Dynamics and Advanced Driver Assistance Systems

11.10.2018

14

GTC 2018

11.10.2018

GTC 2018

14

Positive driving experience remain important!

11.10.2018

15

GTC 2018

11.10.2018

GTC 2018

15

Object Hopping

Object Lost

Sudden System Drop-Out

1

2

3

11.10.2018

16

GTC 2018

Challenges

Subjects feel more stressed and unsaved with LKAS instead of without LKAS

Statistical Analysis >50 Measurements1

2

3

4

5

120 kph 160 kph

Subjectively perceived stress "with" versus "without" lane-keeping assistant

with LKAS early with LKAS late without LKAS

goo

db

ad

65 %

105 %

11.10.2018

17

GTC 2018

11.10.2018

GTC 2018

17

"The most important thing about a ADAS/AD

is that it is common used.

And it's only used when it's experienced as

safe, usefulness and ease of use.“

People want a perceived safety of 100%.

11.10.2018

18

GTC 2018

Paradigm change

Fun to be Driven Fun to DriveTestingSimulation

Team Play

Development Efficiency Driving Experience

How to transfer todays “fun-to-drive” into future “fun-to-be-driven” within a attribute driven development?

11.10.2018

19

GTC 2018

11.10.2018

GTC 2018

19

We need a very high level of environment knowledge and anticipatory understanding of the situation!

11.10.2018

20

GTC 2018

Rule based vs. artificial intelligence

How complex, reproducible and understandable are the methods?

Steering Angle

Folding LevelSensor Input

(Image)

Pooling Level Fully Connected Level Output Vector

ADAS/AD

FunctionSteering Torque

Maneuver InterpretationSensor Input

(Fused Sensors)Decision Tree Function Output Vector

11.10.2018

21

GTC 2018

11.10.2018

GTC 2018

21

Lateral control in complex environments!

• markings heavily weathered

• partly no center marking

• dirty roads

• no side lines in sections

• uneven road with inclination changes

• heavy vehicle response

11.10.2018

22

GTC 2018

11.10.2018

GTC 2018

22

Drift Weaving

Sudden System Drop-Out

1

2

11.10.2018

23

GTC 2018

Learning challenges of artificial intelligence

Real Ground Truth

Virtual Ground Truth

Detailing of labeling and the process to build up the simulation environment will have a effect to the results.

11.10.2018

24

GTC 2018

The Toolchain

Dr. Tobias Düser

Department ManagerAdvanced Solution Lab


A seamless but open toolchain with new approach is necessary

Simulation

Same Results

Virtual Testing Proving Ground Real World

Maneuver, Scenario, Test Case Abstraction incl. KPIs

Environment Abstraction

Vehicle Abstraction

The most efficient validation will be done by those who will use the best combination!


Reference: Proving Ground

NVIDIA DRIVE

AD Function

Vehicle Interface

Real Vehicle on Real Proving Ground

Real Vehicle on Real Proving Ground

Methodology, Test Catalogue and KPIsfor AD Function

AVL DRIVE for ADAS

KPI

SC

Vehicle Under Test

Proving Ground

very close to the real operation

expensive, high effort, less repeatability


Architecture Virtual Testing@ the DrivingCube

NVIDIA DRIVE

AD Function

Vehicle Interface

CAN Interface

TB Interface

AVL V

ECO

N

Real Vehicle on AVL DRIVINGCUBE™

Virtual Vehicle on Virtual ProvingGroundT

estb

ed.C

ON

NECT™

with M

odel.CO

NN

ECT™

NVIDIA DRIVE SIM

DRIVE SIM

Wrapper


AVL DRIVE for ADAS

DRIVE Interface

KPI

SC

Vehicle Under Test

Virtual Testing

close to real operation, chassis dynos are already established for homologation (emissions)

limited in terms of lateral dynamics


Architecture Simulation

NVIDIA DRIVE

AD Function

CAN Interface

VSM Wrapper

AVL V

SM

Virtual Vehicle in Vehicle Simulation

Virtual Vehicle on Virtual ProvingGroundM

odel.CO

NN

ECT™

-

NVIDIA DRIVE SIM

DRIVE SIM

Wrapper


AVL DRIVE for ADAS

DRIVE Interface

KPI

SC

Vehicle Under Test

Simulation

fast, flexible and cheap in operation, there are already ESC homologation

processes in simulation is only as good as the model(s)


First Results / First Comparison

0 5 10 15 20 25 30 35 40 45 50 55 60

Time / s

Velo

city T

SV /

(m

/s)

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Proving GroundVHiL Repetition 1

MiL

0 5 10 15 20 25 30 35 40 45 50 55 60

Time / s

Desir

ed A

ccele

ration /

(m

/s^

2)

-3,0

-2,5

-2,0

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

2,0

2,5

3,0


MiL

0 5 10 15 20 25 30 35 40 45 50 55 60

Time / s

Delta D

ista

nce /

m

0

5

10

15

20

25

30

35

40

45

50


MiL

0 5 10 15 20 25 30 35 40 45 50 55 60

Time / s

Velo

city /

(m

/s)

-2

0

2

4

6

8

10

12

14

16

Proving GroundVHiL Repetition 1MiL

upper & lower limit

Riedmaier, S.; Nesensohn, J.; Gutenkunst, C.; Düser, T.; Schick, B.; Abdellatif, H.:Validation of X-in-the-Loop Approaches for Virtual Homologation of Automated Driving Functions, GSVF Symposium 2018, Graz

11.10.2018

31

GTC 2018

Validation and Homologation of Autonomous Driving will be a

challenge

Partnering is extremely important to join the forces

The most efficient validation and homologation will be done by those

who will use the best combination of different test environments

NVIDIA, TUEV SUED AND AVL will push this topic together

11.10.2018

32

GTC 2018

Source: https://de.upost.info/31383037343032393936

… we can skip plan B

adas/ad - nvidia · avl drive for adas kpi sc vehicle under test proving ground very close to the...

Documents