full stack deep learning · full stack deep learning (march 2019) pieter abbeel, sergey karayev,...
TRANSCRIPT
![Page 1: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/1.jpg)
Full Stack Deep LearningSetting up Machine Learning Projects
Josh Tobin, Sergey Karayev, Pieter Abbeel
![Page 2: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/2.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Goals for the lecture
1. Introduce our framework for understanding ML projects
2. Describe best practices for planning & setting up ML projects
!2
![Page 3: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/3.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Running case study - pose estimation
!3
(x, y, z)<latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit>
(�, ✓, )<latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit>
Orientation (L2 loss)
Xiang, Yu, et al. "PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes." arXiv preprint arXiv:1711.00199 (2017).
Position (L2 loss)
![Page 4: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/4.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Hypothetical Co. Full Stack Robotics (FSR) wants to use pose estimation to enable grasping
!4
(x, y, z)<latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit>
(�, ✓, )<latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit>
Grasp model Motor commands
![Page 5: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/5.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Goals for the lecture
1. Introduce our framework for understanding ML projects
2. Describe best practices for planning & setting up ML projects
!5
![Page 6: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/6.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!6
Planning & project setup
![Page 7: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/7.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!7
• Decide to work on pose estimation• Determine requirements & goals• Allocate resources• Etc.
Planning & project setup
![Page 8: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/8.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!8
Planning & project setup
Data collection & labeling
![Page 9: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/9.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!9
• Collect training objects• Set up sensors (e.g., cameras)• Capture images of objects• Annotate with ground truth (how?)
Planning & project setup
Data collection & labeling
![Page 10: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/10.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!10
Planning & project setup
Data collection & labeling
![Page 11: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/11.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!11
• Too hard to get data. • Easier to label a different task (e.g.,
hard to annotate pose, easier to annotate per-pixel segmentation)
Planning & project setup
Data collection & labeling
![Page 12: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/12.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!12
Planning & project setup
Data collection & labeling
Training & debugging
![Page 13: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/13.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!13
• Implement baseline in OpenCV• Find SoTA model & reproduce• Debug our implementation• Improve model for our task
Planning & project setup
Data collection & labeling
Training & debugging
![Page 14: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/14.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!14
Planning & project setup
Data collection & labeling
Training & debugging
![Page 15: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/15.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!15
• Collect more data• Realize data labeling is unreliable
Planning & project setup
Data collection & labeling
Training & debugging
![Page 16: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/16.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!16
Planning & project setup
Data collection & labeling
Training & debugging
![Page 17: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/17.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!17
• Realize task is too hard• Requirements trade off with each
other - revisit which are most important
Planning & project setup
Data collection & labeling
Training & debugging
![Page 18: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/18.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!18
• Pilot in grasping system in the lab• Write tests to prevent regressions• Roll out in production
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
![Page 19: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/19.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!19
• Doesn’t work in the lab - keep improving accuracy of model
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
![Page 20: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/20.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!20
• Fix data mismatch between training data and data seen in deployment
• Collect more data• Mine hard cases
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
![Page 21: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/21.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!21
• The metric you picked doesn’t actually drive downstream user behavior. Revisit the metric.
• Performance in the real world isn’t great - revisit requirements (e.g., do we need to be faster or more accurate?
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
![Page 22: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/22.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!22
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
![Page 23: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/23.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!23
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
![Page 24: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/24.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!24
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
Team & hiring
Infra & tooling
Cross-project infrastructure
![Page 25: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/25.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
What else do you need to know?
• Understand state of the art in your domain
• Understand what’s possible
• Know what to try next
• We will introduce most promising research areas
!25
![Page 26: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/26.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!26
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
Team & hiring
Infra & tooling
Cross-project infrastructure
![Page 27: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/27.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!27
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
Define project goals
Choose metrics
Evaluate baselines
Set up codebase
Ingest LabelingStrategy
Pilot in production Testing Deployment Monitoring
Choose si-mplest
Implement model
Debug model(s)
Look at train/ val/test
Prioritize improvement
Team & hiring
Infra & tooling
Cross-project infrastructure
![Page 28: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/28.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!28
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
Define project goals
Choose metrics
Evaluate baselines
Set up codebase
Ingest LabelingStrategy
Pilot in production Testing Deployment Monitoring
Choose si-mplest
Implement model
Debug model(s)
Look at train/ val/test
Prioritize improvement
Team & hiring
Infra & tooling
Cross-project infrastructure
![Page 29: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/29.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Outline of the rest of the lecture
1. Prioritizing projects & choosing goals
2. Choosing metrics
3. Choosing baselines
!29
![Page 30: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/30.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Outline of the rest of the lecture
1. Prioritizing projects & choosing goals
2. Choosing metrics
3. Choosing baselines
!30
![Page 31: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/31.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for prioritizing projects
A. High-impact ML problems
• Complex parts of your pipeline
• Places where cheap prediction is valuable
B. Cost of ML projects is driven by data availability, but accuracy requirement also plays a big role
!31
1. Prioritizing projects
![Page 32: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/32.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
A (general) framework for prioritizing projects
!32
Feasibility (e.g., cost)Low
High
Low High
High priority
1. Prioritizing projects
Impact
![Page 33: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/33.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Mental models for high-impact ML projects
1. Where can you take advantage of cheap prediction?
2. Where can you automate complicated manual software pipelines?
!33
1. Prioritizing projects
![Page 34: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/34.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Mental models for high-impact ML projects
!34
Prediction Machines: The Simple Economics of Artificial Intelligence (Agrawal, Gans, Goldfarb)
• AI reduces cost of prediction
• Prediction is central for decision making
• Cheap prediction means
• Prediction will be everywhere
• Even in problems where it was too expensive before (e.g., for most people, hiring a driver)
• Implication: Look for projects where cheap prediction will have a huge business impact
The economics of AI(Agrawal, Gans, Goldfarb)
1. Prioritizing projects
![Page 35: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/35.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Mental models for high-impact ML projects
!35
Software 2.0 (Andrej Karpathy): https://medium.com/@karpathy/software-2-0-a64152b37c35
Software 2.0(Andrej Karpathy)
1. Prioritizing projects
![Page 36: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/36.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Mental models for high-impact ML projects
!36
Software 2.0 (Andrej Karpathy): https://medium.com/@karpathy/software-2-0-a64152b37c35
• Software 1.0 = traditional programs with explicit instructions (python / c++ / etc)
• Software 2.0 = humans specify goals, and algorithm searches for a program that works
• 2.0 programmers work with datasets, which get compiled via optimization
• Why? Works better, more general, computational advantages
• Implication: look for complicated rule-based software where we can learn the rules instead of programming them
Software 2.0(Andrej Karpathy)
1. Prioritizing projects
![Page 37: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/37.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
A (general) framework for prioritizing projects
!37
Feasibility (e.g., cost)Low
High
Low High
High priority
1. Prioritizing projects
Impact
![Page 38: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/38.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!38
Data availability
Cost drivers
1. Prioritizing projects
![Page 39: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/39.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!39
Data availability
Accuracy requirement
Cost drivers
1. Prioritizing projects
![Page 40: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/40.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!40
Data availability
Accuracy requirement
Problem difficulty
Cost drivers
1. Prioritizing projects
![Page 41: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/41.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!41
Data availability
Accuracy requirement
Problem difficulty
Cost drivers Main considerations
• How hard is it to acquire data?• How expensive is data labeling?• How much data will be needed?
1. Prioritizing projects
![Page 42: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/42.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!42
Data availability
Accuracy requirement
Problem difficulty
Cost drivers Main considerations
• How hard is it to acquire data?• How expensive is data labeling?• How much data will be needed?
• How costly are wrong predictions?• How frequently does the system need to be
right to be useful?
1. Prioritizing projects
![Page 43: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/43.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!43
Data availability
Accuracy requirement
Problem difficulty
Cost drivers Main considerations
• How hard is it to acquire data?• How expensive is data labeling?• How much data will be needed?
• How costly are wrong predictions?• How frequently does the system need to be
right to be useful?
• Good published work on similar problems? (newer problems mean more risk & more technical effort)
• Compute needed for training?• Compute available for deployment?
1. Prioritizing projects
![Page 44: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/44.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Assessing feasibility of ML projects
!44
Data availability
Accuracy requirement
Problem difficulty
Cost drivers Main considerations
• How hard is it to acquire data?• How expensive is data labeling?• How much data will be needed?
• How costly are wrong predictions?• How frequently does the system need to be
right to be useful?
• Good published work on similar problems? (newer problems mean more risk & more technical effort)
• Compute needed for training?• Compute available for deployment?
1. Prioritizing projects
![Page 45: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/45.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why are accuracy requirements so important?
!45
Required accuracy
Project cost
50% 90% 99% 99.9% …
1. Prioritizing projects
![Page 46: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/46.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why are accuracy requirements so important?
!46
Required accuracy
Project cost
50% 90% 99% 99.9% …
ML project costs tend to scale super-linearly in the accuracy requirement
1. Prioritizing projects
![Page 47: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/47.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Product design can reduce need for accuracy
!47
See “Designing Collaborative AI” (Ben Reinhardt and Belmer Negrillo): https://medium.com/@Ben_Reinhardt/designing-collaborative-ai-5c1e8dbc8810
1. Prioritizing projects
![Page 48: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/48.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Another heuristic for assessing feasibility
!48
• Recognize content of images
• Understand speech
• Translate speech
• Grasp objects
• etc.
Examples Counter-examples?
• Understand humor / sarcasm
• In-hand robotic manipulation
• Generalize to new scenarios
• etc.
1. Prioritizing projects
![Page 49: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/49.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why is FSR focusing on pose estimation?
!49
• FSR’s goal is grasping - requires reliable pose estimation
• Traditional robotics pipeline uses hand-designed heuristics & online optimization• Slow• Brittle• Great candidate for Software 2.0!
Impact Feasibility
• Data availability
• Easy to collect data
• Labeling data could be a challenge, but can instrument lab with sensors
• Accuracy requirement
• Require high accuracy to grasp an object: <0.5cm
• However, low cost of failure - picks per hour important, not % successes
• Problem difficulty
• Similar published results exist but need to adapt to our objects and robot
1. Prioritizing projects
![Page 50: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/50.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for prioritizing projects
A. To find high-impact ML problems, look for complex parts of your pipeline and places where cheap prediction is valuable
B. The cost of ML projects is primarily driven by data availability, but your accuracy requirement also plays a big role
!50
1. Prioritizing projects
![Page 51: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/51.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Outline of the rest of the lecture
1. Prioritizing projects & choosing goals
2. Choosing metrics
3. Choosing baselines
!51
![Page 52: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/52.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for choosing a metricA. The real world is messy; you usually care about lots of
metrics
B. However, ML systems work best when optimizing a single number
C. As a result, you need to pick a formula for combining metrics
D. This formula can and will change!
!52
2. Choosing metrics
![Page 53: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/53.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Review of accuracy, precision, and recall
!53
n=100 Predicted: NO
Predicted: YES
Actual: NO 5 5 10
Actual: YES 45 45 90
50 50
Confusion matrix
2. Choosing metrics
![Page 54: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/54.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Review of accuracy, precision, and recall
!54
n=100 Predicted: NO
Predicted: YES
Actual: NO 5 5 10
Actual: YES 45 45 90
50 50
Confusion matrix
AccuracyCorrect
Total
50%
2. Choosing metrics
![Page 55: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/55.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Review of accuracy, precision, and recall
!55
n=100 Predicted: NO
Predicted: YES
Actual: NO 5 5 10
Actual: YES 45 45 90
50 50
Confusion matrix
Precisiontrue positives
true positives + false positives
90%
2. Choosing metrics
![Page 56: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/56.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Review of accuracy, precision, and recall
!56
n=100 Predicted: NO
Predicted: YES
Actual: NO 5 5 10
Actual: YES 45 45 90
50 50
Confusion matrix
Recalltrue positives
Actual YES
50%
2. Choosing metrics
![Page 57: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/57.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why choose a single metric?
!57
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
Which is best?Model 1
2. Choosing metrics
![Page 58: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/58.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to combine metrics
• Simple average / weighted average
• Threshold n-1 metrics, evaluate the nth
• More complex / domain-specific formula
!58
2. Choosing metrics
![Page 59: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/59.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!59
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
Model 1
2. Choosing metrics
![Page 60: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/60.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!60
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
Model 1
2. Choosing metrics
![Page 61: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/61.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!61
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
Model 1
2. Choosing metrics
![Page 62: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/62.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to combine metrics
• Simple average / weighted average
• Threshold n-1 metrics, evaluate the nth
• More complex / domain-specific formula
!62
2. Choosing metrics
![Page 63: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/63.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to combine metrics
• Simple average / weighted average
• Threshold n-1 metrics, evaluate the nth
• More complex / domain-specific formula
!63
2. Choosing metrics
![Page 64: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/64.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Thresholding metrics
Choosing which metrics to threshold
Choosing threshold values
!64
• Domain judgment (e.g., what is an acceptable tolerance downstream? What performance is achievable?)
• How well does the baseline model do?
• How important is this metric right now?
2. Choosing metrics
• Domain judgment (e.g., which metrics can you engineer around?)
• Which metrics are least sensitive to model choice?
• Which metrics are closest to desirable values?
![Page 65: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/65.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!65
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
Model 1
2. Choosing metrics
![Page 66: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/66.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!66
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
p @ (r > 0.6)
0.0
0.8
0.7
Model 1
2. Choosing metrics
![Page 67: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/67.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!67
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
p @ (r > 0.6)
0.0
0.8
0.7
Model 1
2. Choosing metrics
![Page 68: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/68.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to combine metrics
• Simple average / weighted average
• Threshold n-1 metrics, evaluate the nth
• More complex / domain-specific formula
!68
2. Choosing metrics
![Page 69: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/69.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to combine metrics
• Simple average / weighted average
• Threshold n-1 metrics, evaluate the nth
• More complex / domain-specific formula
!69
2. Choosing metrics
![Page 70: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/70.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Domain-specific metrics: mAP
The economics of AI(Agrawal, Gans, Goldfarb)
!70
Recall
Precision
0% 25% 50% 75%0%
25%
75%
100%
100%
2. Choosing metrics
![Page 71: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/71.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Domain-specific metrics: mAP
Recall
Precision
0% 25% 50% 75%0%
25%
75%
100%
100%
!71
mAP = mean AP, e.g., over classes
2. Choosing metrics
Average precision (AP) = area under the curve
![Page 72: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/72.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!72
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
p @ (r > 0.6)
0.0
0.8
0.7
Model 1
2. Choosing metrics
![Page 73: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/73.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!73
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
p @ (r > 0.6)
0.0
0.8
0.7
mAP
0.7
0.6
0.6
Model 1
2. Choosing metrics
![Page 74: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/74.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Combining precision and recall
!74
Model 2
Model 3
Precision Recall
0.9 0.5
0.8 0.7
0.7 0.9
(p + r) / 2
0.7
0.75
0.8
p @ (r > 0.6)
0.0
0.8
0.7
mAP
0.7
0.6
0.6
Model 1
2. Choosing metrics
![Page 75: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/75.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
(x, y, z)<latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit><latexit sha1_base64="VzuaXkC3pMHYMgeFzgbdHYPD3xo=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahQim7Iuix4MWTVLAf0i4lm2bb0CS7JFmxLv0VXjwo4tWf481/Y9ruQVsfDDzem2FmXhBzpo3rfju5ldW19Y38ZmFre2d3r7h/0NRRoghtkIhHqh1gTTmTtGGY4bQdK4pFwGkrGF1N/dYDVZpF8s6MY+oLPJAsZAQbK92XHytoXEFPp71iya26M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/ODp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjpp0zGiaGSzBeFCUcmQtPvUZ8pSgwfW4KJYvZWRIZYYWJsRgUbgrf48jJpnlU9t+rdnpdqN1kceTiCYyiDBxdQg2uoQwMICHiGV3hzlPPivDsf89ack80cwh84nz/sK480</latexit>
(�, ✓, )<latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit><latexit sha1_base64="BmkDtqmIRxqts5HvJACOgh3GSEs=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0WoICURQZcFN66kgn1AE8pkOmmGTiZh5kaoofgrblwo4tb/cOffOG2z0NYDl3s4517mzglSwTU4zre1tLyyurZe2ihvbm3v7Np7+y2dZIqyJk1EojoB0UxwyZrAQbBOqhiJA8HawfB64rcfmNI8kfcwSpkfk4HkIacEjNSzD6teGvEz7EHEgJiean7asytOzZkCLxK3IBVUoNGzv7x+QrOYSaCCaN11nRT8nCjgVLBx2cs0SwkdkgHrGipJzLSfT68f4xOj9HGYKFMS8FT9vZGTWOtRHJjJmECk572J+J/XzSC88nMu0wyYpLOHwkxgSPAkCtznilEQI0MIVdzcimlEFKFgAiubENz5Ly+S1nnNdWru3UWlflvEUUJH6BhVkYsuUR3doAZqIooe0TN6RW/Wk/VivVsfs9Elq9g5QH9gff4AtECUHw==</latexit>
!75
Orientation (L2 loss)
t<latexit sha1_base64="Wn3278THpwSxwzLAYeJjstf3jaA=">AAAB6HicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eJIW7Ae0oWy2k3btZhN2J0Ip/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemEphyPO+nbX1jc2t7cJOcXdv/+CwdHTcNEmmOTZ4IhPdDplBKRQ2SJDEdqqRxaHEVji6nfmtJ9RGJOqBxikGMRsoEQnOyEp16pXKXsWbw10lfk7KkKPWK311+wnPYlTEJTOm43spBROmSXCJ02I3M5gyPmID7FiqWIwmmMwPnbrnVum7UaJtKXLn6u+JCYuNGceh7YwZDc2yNxP/8zoZRTfBRKg0I1R8sSjKpEuJO/va7QuNnOTYEsa1sLe6fMg042SzKdoQ/OWXV0nzsuJ7Fb9+Va7e53EU4BTO4AJ8uIYq3EENGsAB4Rle4c15dF6cd+dj0brm5DMn8AfO5w/jxY0E</latexit><latexit sha1_base64="Wn3278THpwSxwzLAYeJjstf3jaA=">AAAB6HicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eJIW7Ae0oWy2k3btZhN2J0Ip/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemEphyPO+nbX1jc2t7cJOcXdv/+CwdHTcNEmmOTZ4IhPdDplBKRQ2SJDEdqqRxaHEVji6nfmtJ9RGJOqBxikGMRsoEQnOyEp16pXKXsWbw10lfk7KkKPWK311+wnPYlTEJTOm43spBROmSXCJ02I3M5gyPmID7FiqWIwmmMwPnbrnVum7UaJtKXLn6u+JCYuNGceh7YwZDc2yNxP/8zoZRTfBRKg0I1R8sSjKpEuJO/va7QuNnOTYEsa1sLe6fMg042SzKdoQ/OWXV0nzsuJ7Fb9+Va7e53EU4BTO4AJ8uIYq3EENGsAB4Rle4c15dF6cd+dj0brm5DMn8AfO5w/jxY0E</latexit><latexit sha1_base64="Wn3278THpwSxwzLAYeJjstf3jaA=">AAAB6HicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eJIW7Ae0oWy2k3btZhN2J0Ip/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemEphyPO+nbX1jc2t7cJOcXdv/+CwdHTcNEmmOTZ4IhPdDplBKRQ2SJDEdqqRxaHEVji6nfmtJ9RGJOqBxikGMRsoEQnOyEp16pXKXsWbw10lfk7KkKPWK311+wnPYlTEJTOm43spBROmSXCJ02I3M5gyPmID7FiqWIwmmMwPnbrnVum7UaJtKXLn6u+JCYuNGceh7YwZDc2yNxP/8zoZRTfBRKg0I1R8sSjKpEuJO/va7QuNnOTYEsa1sLe6fMg042SzKdoQ/OWXV0nzsuJ7Fb9+Va7e53EU4BTO4AJ8uIYq3EENGsAB4Rle4c15dF6cd+dj0brm5DMn8AfO5w/jxY0E</latexit><latexit sha1_base64="Wn3278THpwSxwzLAYeJjstf3jaA=">AAAB6HicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eJIW7Ae0oWy2k3btZhN2J0Ip/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemEphyPO+nbX1jc2t7cJOcXdv/+CwdHTcNEmmOTZ4IhPdDplBKRQ2SJDEdqqRxaHEVji6nfmtJ9RGJOqBxikGMRsoEQnOyEp16pXKXsWbw10lfk7KkKPWK311+wnPYlTEJTOm43spBROmSXCJ02I3M5gyPmID7FiqWIwmmMwPnbrnVum7UaJtKXLn6u+JCYuNGceh7YwZDc2yNxP/8zoZRTfBRKg0I1R8sSjKpEuJO/va7QuNnOTYEsa1sLe6fMg042SzKdoQ/OWXV0nzsuJ7Fb9+Va7e53EU4BTO4AJ8uIYq3EENGsAB4Rle4c15dF6cd+dj0brm5DMn8AfO5w/jxY0E</latexit>
Prediction timeXiang, Yu, et al. "PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes." arXiv preprint arXiv:1711.00199 (2017).
Position (L2 loss)
2. Choosing metrics
![Page 76: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/76.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
• Enumerate requirements
• Downstream goal is real-time robotic grasping
• Position error must be <1cm, not sure exactly how precise is needed
• Angular error <5 degrees
• Must run in 100ms to work in real-time
!76
2. Choosing metrics
![Page 77: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/77.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
• Enumerate requirements
• Evaluate current performance
• Train a few models
!77
2. Choosing metrics
![Page 78: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/78.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
• Enumerate requirements
• Evaluate current performance
• Compare current performance to requirements
• Position error between 0.75 and 1.25cm (depending on hyperparameters)
• All angular errors around 60 degrees
• Inference time ~300ms
!78
2. Choosing metrics
![Page 79: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/79.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
• Enumerate requirements
• Evaluate current performance
• Compare current performance to requirements
• Prioritize angular error
• Threshold position error at 1cm
• Ignore run time for now
!79
2. Choosing metrics
![Page 80: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/80.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Example: choosing a metric for pose estimation
• Enumerate requirements
• Evaluate current performance
• Compare current performance to requirements
• Revisit metric as your numbers improve
!80
2. Choosing metrics
![Page 81: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/81.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for choosing a metricA. The real world is messy; you usually care about lots of
metrics
B. However, ML systems work best when optimizing a single number
C. As a result, you need to pick a formula for combining metrics
D. This formula can and will change!
!81
2. Choosing metrics
![Page 82: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/82.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Outline
1. Prioritizing projects & choosing goals
2. Choosing metrics
3. Choosing baselines
!82
![Page 83: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/83.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for choosing baselines
A. Baselines give you a lower bound on expected model performance
B. The tighter the lower bound, the more useful the baseline (e.g., published results, carefully tuned pipelines, & human baselines are better)
!83
3. Choosing baselines
![Page 84: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/84.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why are baselines important?
!84
3. Choosing baselines
![Page 85: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/85.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Why are baselines important?Same model, different baseline —> different next steps
!85
3. Choosing baselines
![Page 86: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/86.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
!86
External baselines
3. Choosing baselines
![Page 87: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/87.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
• Published results
!87
External baselines
3. Choosing baselines
Make sure comparison is fair!
![Page 88: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/88.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
• Published results
• Scripted baselines
!88
External baselines
Internal baselines
3. Choosing baselines
![Page 89: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/89.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
• Published results
• Scripted baselines, e.g.,• OpenCV scripts• Rules-based methods
!89
External baselines
Internal baselines
3. Choosing baselines
![Page 90: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/90.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
• Published results
• Scripted baselines
• Simple ML baselines
!90
External baselines
Internal baselines
3. Choosing baselines
![Page 91: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/91.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to look for baselines• Business / engineering requirements
• Published results
• Scripted baselines
• Simple ML baselines, e.g., • Standard feature-based models (e.g., bag-of-words
classifier)• Linear classifier with hand-engineered features• Basic neural network model (e.g., VGG-like architecture
without batch norm, weight norm, etc.)
!91
External baselines
Internal baselines
3. Choosing baselines
![Page 92: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/92.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to create good human baselines
!92
Low
Quality of baseline
High
Ease of data collection
Low
Random people (e.g., Amazon Turk)
Ensemble of random people
Domain experts (e.g., doctors)
Deep domain experts (e.g., specialists)
Mixture of experts
3. Choosing baselines
![Page 93: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/93.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
How to create good human baselines• Highest quality that allows more data to be labeled easily
• More specialized domains need more skilled labelers
• Find cases where model performs worse and concentrate data collection
!93
More on labeling in data lecture!
3. Choosing baselines
![Page 94: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/94.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Key points for choosing baselines
A. Baselines give you a lower bound on expected model performance
B. The tighter the lower bound, the more useful the baseline (e.g., published results, carefully tuned pipelines, human baselines are better)
!94
3. Choosing baselines
![Page 95: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/95.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Lifecycle of a ML project
!95
Planning & project setup
Data collection & labeling
Training & debugging
Deploying & testing
Per-project activities
Define project goals
Choose metrics
Evaluate baselines
Set up codebase
Ingest LabelingStrategy
Pilot in production Testing Deployment Monitoring
Choose si-mplest
Implement model
Debug model(s)
Look at train/ val/test
Prioritize improvement
Team & hiring
Infra & tooling
Cross-project infrastructure
![Page 96: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/96.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Questions?
!96
![Page 97: Full Stack Deep Learning · Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects Lifecycle of a ML project!21 • The metric you picked doesn’t](https://reader030.vdocuments.us/reader030/viewer/2022040410/5ecbe959cce67259330eeab3/html5/thumbnails/97.jpg)
Full Stack Deep Learning (March 2019) Pieter Abbeel, Sergey Karayev, Josh Tobin L2: Projects
Where to go to learn more• Andrew Ng’s “Machine Learning Yearning”
• Andrej Karpathy’s “Software 2.0”
• Agrawal’s “The Economics of AI”
!97