synthetic data / ai - nvidia › video › ... · 7 synthetic data real data is expensive,...
TRANSCRIPT
![Page 1: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/1.jpg)
Rev Lebaredian - Vice President, Simulation Technolgy
SYNTHETIC DATA / AI
![Page 2: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/2.jpg)
2
NEURAL NETWORKS NEED DATA
Deep Learning is amazing!
Require huge amounts of quality data
Data needs labeling
For some problems, data + labeling is available
Good data doesn’t exists for most problems
And Labels!
Image-Net (http://www.image-net.org/) example
![Page 3: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/3.jpg)
3
DOMINOS, ANYONE?Isaac at SIGGRAPH 2017
![Page 4: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/4.jpg)
4
NO EXISTING DATASET
![Page 5: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/5.jpg)
5
Labeled Data
1min per object
![Page 6: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/6.jpg)
7
SYNTHETIC DATA
Real data is expensive, sometimes dangerous
Synthetic labels are automatic and accurate
Useful for validation, in addition to training
![Page 7: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/7.jpg)
8
AV SYNTHETIC DATASETSVirtual KITTI
Adrien Gaidon, Qiao Wang, Yohann Cabon, Eleonora Vig: Virtual Worlds as Proxy for Multi-Object Tracking Analysis IEEE CVPR 2016
![Page 8: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/8.jpg)
9
DOMAIN RANDOMIZATIONExplore the gap using random cars, textures, camera, distractors, etc
![Page 9: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/9.jpg)
10
DOMAIN RANDOMIZATIONExample Scenes
![Page 10: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/10.jpg)
11
STRUCTURED DOMAIN RANDOMIZATIONPutting it all together
![Page 11: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/11.jpg)
12
SIMULATETRAIN MODELSCOLLECT DATA DRIVE
NVIDIA DRIVEEND-TO-END PLATFORM
Lanes Lights
Path
Signs
PedestriansCars
Lanes Lights
Path
Signs
PedestriansCars
![Page 12: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/12.jpg)
13
![Page 13: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/13.jpg)
14
![Page 14: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/14.jpg)
15
AlphaZero OpenAI Five
OpenAI, 2018Deepmind, 2018
REINFORCEMENT LEARNING SUCCESSES
![Page 15: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/15.jpg)
16
APPLICATIONS
Locomotion/Animation
Reinforcement Learning
Liang, Makoviychuk, Handa etc, 2018NVIDIA
![Page 16: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/16.jpg)
17
APPLICATIONS
Sim2Real Robotics
Robotics
Chebotar, Handa, Makoviychuk, etc, 2018NVIDIA
![Page 17: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/17.jpg)
18
EXAMPLESLocomotion
![Page 18: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/18.jpg)
19
EXAMPLESLocomotion + Physics
![Page 19: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/19.jpg)
20
EXAMPLESLocomotion + Physics
![Page 20: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/20.jpg)
21
ISAAC GYMToolkit for Parallel AI Learning Experiments
![Page 21: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/21.jpg)
22
PHYSICSPhysX
![Page 22: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/22.jpg)
23
PHYSICSFleX
• Multi-physics
• Rigid and FEM soft bodies
• Cloth, ropes
• Liquids
• Two-way coupling and force propagation between different phases
![Page 23: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/23.jpg)
24
CABLE ROBOT
![Page 24: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/24.jpg)
25
ROBOTS
LEARN& PLAN
SEE, HEAR,TOUCH
ACT
![Page 25: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/25.jpg)
26
![Page 26: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/26.jpg)
27
![Page 27: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/27.jpg)
28
BARTENDING ROBOTTrained in Isaac Gym
![Page 28: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/28.jpg)
29
ISAAC SIM
![Page 29: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/29.jpg)
30
![Page 30: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/30.jpg)
31
![Page 31: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/31.jpg)
32
![Page 32: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/32.jpg)
33
Our Reality Virtual Reality
Reality
Human
ArtificialIn
telli
gen
ce
![Page 33: SYNTHETIC DATA / AI - Nvidia › video › ... · 7 SYNTHETIC DATA Real data is expensive, sometimes dangerous Synthetic labels are automatic and accurate Useful for validation, in](https://reader036.vdocuments.us/reader036/viewer/2022081406/5f10471e7e708231d4485122/html5/thumbnails/33.jpg)