dr. benny kneissl maneuver prediction using vehicle sensor ... · data innovation lab | maneuver...
TRANSCRIPT
![Page 1: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/1.jpg)
Maneuver prediction using vehicle sensor data
Team:Nils Sturma, Harshit Chopra, Anna Kuvakina
Mentors:Nico Epple,Dr. Benny Kneissl
TUM Data Innovation Lab
Chair for Applied and Numerical Analysis
Co-Mentor:Laure Vuaille
Project Lead:Dr. Ricardo Acevedo Cabra
Supervisor:Prof. Dr. Massimo Fornasier
![Page 2: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/2.jpg)
Motivation
2Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Range of functionality of ADAS systems [1]
“We focus on the intention of the driver and develop an approach for lane change prediction”
![Page 3: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/3.jpg)
Naturalistic driving studies
3Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Test vehicles collect real world data in different countries using
● multiple sensors ● front and rear view cameras
We work with signal and object data collected on German highways
![Page 4: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/4.jpg)
Overview
4Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 5: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/5.jpg)
Labeling
5Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Ego EgoEgo Ego
Ego
● automatic, rule-based algorithm● validation: videos● ground truth
![Page 6: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/6.jpg)
Prediction horizon
6Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
![Page 7: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/7.jpg)
Prediction horizon
7Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Used features● situation ● kinematics● surrounding objects
NOT used features● winker● distance to line left/right● steering wheel angle● lateral acceleration
Goal: predict driver’s intention
![Page 8: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/8.jpg)
Data workflow
8Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
feature extraction
ML models
feature engineering
final data and final model
Train Test
lane change samples 13416 887
non lane change samples 68393 4113
![Page 9: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/9.jpg)
Overview
9Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 10: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/10.jpg)
Surrounding objects
10Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
![Page 11: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/11.jpg)
Features
11Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Velocity
Blind spot detection (left
and right)
Road angleBoundary line type (left and
right)
Lane count
Acceleration
![Page 12: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/12.jpg)
Features
12Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Velocity
Blind spot detection (left
and right)
Road angleBoundary line type (left and
right)
Lane count
Acceleration
Distance to surrounding
objects
Absolute velocity of surrounding
objects
Absolute acceleration
of surrounding objects
![Page 13: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/13.jpg)
● Inverse time to collison
● Time headway
● Relative speed
● Handling missing objects
Feature engineering
13Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
E1 exists E2 exists E3 exists E4 exists
yes/no yes/no yes/no yes/no
![Page 14: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/14.jpg)
Overview
14Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 15: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/15.jpg)
Continuous features
15Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Mean acceleration of Ego Mean time headway between Ego and E1
Ego E1
![Page 16: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/16.jpg)
Continuous features
16Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Mean inverse time to collision between Ego and E2
Mean inverse time to collision between Ego and E2 with std. deviations
Ego
E2
![Page 17: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/17.jpg)
Categorical features
17Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Probability of lane change conditioned on the existence of vehicles
![Page 18: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/18.jpg)
Overview
18Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 19: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/19.jpg)
Performance metrics
19Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
TP TN FP FN
![Page 20: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/20.jpg)
Random Forest
20Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Random Forest [6]
![Page 21: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/21.jpg)
Random Forest
21Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
F1 score Accuracy
0.6864 0.7946Used features:
● no scaling
● fast training
![Page 22: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/22.jpg)
Support Vector Machine
22Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Linear SVM classifier [2]
● Same features as for Random Forest● Kernel Trick (RBF kernel)● Grid Search
○ Gamma○ Penalty parameter
● Forward Feature Selection● Backward Elimination
![Page 23: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/23.jpg)
Support Vector Machine
23Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
F1 score Accuracy
0.5642 0.8273
Best Features F1 score
vehicles right count 0.3659
boundary line left (type 3) 0.3823
longitudinal velocity 0.4018
longitudinal acceleration 0.4406
difference of velocity between Ego and E2 0.4863
boundary line left (type 3) start 0.4977
inverse time to collision between Ego and E1....
0.5107
Forward Feature Selection Algorithm
![Page 24: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/24.jpg)
● Learn relations between sequences of inputs● Input samples -
(no_of_timestamps * features)● LSTM/GRU● Loss: Binary cross entropy
Recurrent Neural Networks
24Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Many-to-one RNN network [4]
Unrolling in RNN [3]
![Page 25: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/25.jpg)
● Hyperparameter tuning● Optimizer - RMSProp● Other network features
○ Dropout○ Input weight sampling○ Weight regularization○ Reduce learning rate
● Final GRU structure:
Recurrent Neural Networks
25Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
F1 score variation with different number of neurons in recurrent layers
F1 score variation with initial learning rate
Recurrent 1 128
Recurrent 2 64
Dense 1 32
Dense 2 1
![Page 26: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/26.jpg)
Overview
26Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 27: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/27.jpg)
Results
27Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
F1 score Accuracy
0.7714 0.9264
Lane changes 887
Non lane changes 4113
F1 score Accuracy
0.7470 0.9014
Model which predicts direction of lane change:
![Page 28: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/28.jpg)
Visualization
28Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
![Page 29: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/29.jpg)
Edge cases
29Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
TN FP
FN TPDriver overtakes and does not go right again
TN FP
FN TPDriver changes lane to left with no objects around
![Page 30: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/30.jpg)
Clustering of drivers according to driving behavioural patterns
Further improvements
30Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Data improvements● Self-representation of Ego● Vehicles behind Ego
Clustering of people [6]
![Page 31: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/31.jpg)
Overview
31Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Preprocessing
Feature engineering
Data exploration
Algorithms
Results
Case studies
![Page 32: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/32.jpg)
Case studies
32Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
China data● Transfer our trained model directly● Test transferability to geographically
different environment
All features● Train new model using features that were
excluded like winker, distance to line, lateral acceleration
● Assert improvement by these features
Experiment F1 score Accuracy
Direct transfer China 0.2664 0.6276
Using all features 0.8828 0.9366
![Page 33: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/33.jpg)
Conclusion
33Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
Transferable?
● model (direct)
● approach
automatic rule-based
labeling
explore possible further
improvements
feature engineering on
large scale
lane change prediction
![Page 34: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/34.jpg)
[1] https://www.kisspng.com/png-car-advanced-driver-assistance-systems-driving-veh-3194115/preview.html [visited on 20/07/2019][2] https://michelleful.github.io/code-blog/2015/06/18/classifying-roads/ [visited on 30/07/2019][3] https://colah.github.io/posts/2015-08-Understanding-LSTMs/ [visited on 03/08/2019][4] https://discuss.pytorch.org/t/example-of-many-to-one-lstm/1728 [visited on 28/07/2019][5] https://towardsdatascience.com/k-means-clustering-identifying-f-r-i-e-n-d-s-in-the-world-of-strangers-695537505d [visited on 03/08/2019][6] https://www.linkedin.com/pulse/random-forest-algorithm-interactive-discussion-niraj-kumar/ [visited on 30/07/2019]
References
34Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019
![Page 35: Dr. Benny Kneissl Maneuver prediction using vehicle sensor ... · Data Innovation Lab | Maneuver prediction using vehicle sensor data | 06/08/2019 23 F1 score Accuracy 0.5642 0.8273](https://reader033.vdocuments.us/reader033/viewer/2022041415/5e1affcf061fdf6e1717ef16/html5/thumbnails/35.jpg)
35
Thank you for your attention! Questions?