autonomous driving revolution- trends, challenges and machine learning
TRANSCRIPT
![Page 1: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/1.jpg)
1
Autonomous driving revolution: trends, challenges and machine learning
Junli Gu
![Page 2: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/2.jpg)
2
Outline• Vehicle revolution• Challenges: when Big Data meets Machine Learning in the car• Sensor System collects Big Data• Machine Learning perceives the real world• Car is a Digital Agent
• Conclusions
![Page 3: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/3.jpg)
3
Transportation evolution history
P1: Analog device
P2: Digitalization
P3: Intelligence
1900 2020
P0: Primitive
From IHS
![Page 4: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/4.jpg)
4
A revolution from analog to digital to intelligence• All parts are digital
controllable• Complex sensor systems• World perception• Controlled by computers
• A car will become an agent• With autonomous behavior
![Page 5: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/5.jpg)
5
Big Data meets Machine Learning in the Car
World Perception GPS, mapping, localization
Motion Control Path Planning
• Real world is a Big Data problem• Driving in human world required intelligent perception of the world
![Page 6: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/6.jpg)
6
Hybrid sensor system collects Big Data• A sensor system composed of Cameras, Radar, Ultrasound, Lidar• Sensor fusion technology is not mature yet• Different sensor data is mostly computed separately
![Page 7: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/7.jpg)
7
Case study in combining sensors• Google self driving cars• Velodyne 64-beam laser + 4 radars+ 1 camera+ GPS• generates a 3D geometry map of the environment
• Radar info is too thin to differentiate objects of same size, same speed• Cameras see rich semantics. What you see is what you get. But no depth
What radar sees What camera sees
![Page 8: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/8.jpg)
8
Machine learning perceives the real world• Large scale of 2D object recognition is better than human• What objects are around
• 3D scene understanding and modeling• Where is the object
• Semantic segmentation• Extent of the obstacles
• Reinforcement learning• Policy (reward or penalty based learning)
• End to end learning• From raw data direct to behavior, like a robot
![Page 9: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/9.jpg)
Deep learning based 2D image recognition
9
97%
2016
• Large scale object recognition 97% accuracy• Lane detection, pedestrian detection, vehicle detection• Animal detection, and road surface detection etc.• Case: Mobile eye from traditional algorithm => Deep learning
DNN model
Big Data
Human 95%
![Page 10: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/10.jpg)
10
3D scene understanding• Real world is not flat. It is 3D.• Depth information is critical for driving. • 3D model is far more complicated.
Reference: Lin Yuanqing, “Building Blocks for Visual 3D Scene Understandingtowards Autonomous Driving”
![Page 11: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/11.jpg)
11
Semantic segmentation• Understand the extent of the object and each pixel of it• Advanced requirement based on recognition and detection• Eventually 3D world segmentation
Above: “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”Right: “Object Scene Flow for Autonomous Vehicles”
![Page 12: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/12.jpg)
12
Reinforcement learning• Goal: to win• Policy based learning (reward/penalty)
• Learning task: next move at arbitrary states• Algorithm: reinforcement learning + DNN• Similar methodology can be applied to driving• Driving Policy: Given safety, spend least time to get to destination• The car figures out how to drive by following the reward or penalty
Google Alpha go
![Page 13: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/13.jpg)
End to end learning• Use machine learning as the only step from raw data to control
• Without human interference noise in the middle steps
• DaveNet: AI teaches the car how to drive• NVidia GTC 2016 “after 3000 miles of supervised driving, their car can navigate safely on freeways and country
roads, even on rainy days.”
“End to End Learning for Self-Driving Cars”, arxiv.org ; Google “Human-level control through deep reinforcement learning”
13
![Page 14: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/14.jpg)
14
Other challenges in the Car• Embedded computers in the car• Cloud solution• Vehicle external connections (V2E)• Vehicle internal connection
![Page 15: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/15.jpg)
15
Embedded computing systems• Real time computing for the sensor array’s Big Data input• Heterogeneous SoC + HPC level computing (Tera flops)• Reliability is another key
ARM
DSP
GPU
accelerator
General purposeControl friendly
Customized processor machine learning
Computing throughput
Mobile eyeGoogle TPU
NVidia drive PXQualcomm snapdragon
![Page 16: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/16.jpg)
16
Cloud based solution• Agent+ system: cloud updates with traffic condition, bad weather, etc. (Toyota)• Connect smart home with car (Volks Wagen, with LG software)• Challenges: network reliability and real time response
cloud
Collect traffic info
broadcast info
Collect smart home
![Page 17: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/17.jpg)
17
V2E (vehicle to everything) network• Collectively become smarter: share learning on the network• Dephi: V2V(vehicle), V2B(building), V2P(pedestrain)• Tesla: fleet learning. Baidu: vehicle to road
![Page 18: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/18.jpg)
18
High speed internal interconnect• All digital components communicate• High speed due to many sensor• Reliability and redundant design
Credit to Texas Instruments
![Page 19: Autonomous driving revolution- trends, challenges and machine learning](https://reader035.vdocuments.us/reader035/viewer/2022062522/587dfbb91a28abab7e8b6e11/html5/thumbnails/19.jpg)
Conclusions
• Cars are transforming from analog to digitalization and intelligence• Future car will be a very complex digital autonomous device, similar to Robots
• When Big Data meets Machine learning in the car• Complex sensor system collects the Big Data info• Machine learning algorithms will be the key to perceive the real world• Computing in the car is another challenge• Cars will be connected to each other and the cloud
• Whoever addresses the technical challenge will harvest the influence