deep learning part 2 other networks · slides by: joseph e. gonzalez, [email protected] deep...
TRANSCRIPT
Slides by:
Joseph E. Gonzalez,
Deep Learning Part 2Other networks
Quick Logistics
Ø Please sign up to present
Ø Complete reading questions before lecture
Ø Skim the reading for the lecture you are presenting (later in semester).Ø Is there a better paper?
http://bit.ly/aisys-sp19
Machine Learning ≈ Function Approximation
Label:Cat
Object Recognition Speech Recognition
“The cat in the hat”
Robotic Control
The cat in the hat.
El gato en el sombrero
Machine Translation
Last Time
Architectures for Different kinds of inputs
Convolutional Networks
The quick brown fox…
The cat in the hat.
El gato en el sombrero
Recurrent Networks
Sequential reasoning tasksspatial reasoning tasks
Speech recognition
ReinforcementLearning
Architectures for Different kinds of inputs
Convolutional Networks
The quick brown fox…
The cat in the hat.
El gato en el sombrero
Recurrent Networks
Sequential reasoning taskspatial reasoning tasks
Speech recognition
ReinforcementLearning
Graph Networks
Operating on graph data
Link Prediction
Graph Embedding
Supervised Machine LearningØ Given data containing the function inputs and outputs
Input Output
X1 Y1
X2 Y2
… …
Xn Yn
cat
baby
Data
baby
f✓(x) ! y<latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit>
Model
Parameters
Training (approximates the goal over training data):
Goal
✓⇤ = argmin✓
ED [L (f✓(x), y)]<latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit>
Loss
Over all future data
✓̂ = argmin✓
1
n
nX
i=1
L (f✓(xi), yi)<latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit>
Last Time
Learning without Labels
Ø Can we learn what inputs look like?Ø Useful inductive bias when training for a
later supervised task.Ø Often done when labeled data is
available but limited
Ø Convert to a supervised learningproblem:
Input Output
X1 Y1
X2 Y2
… …
Xn Yn
cat
baby
Data
baby
available but limited
Ø Convert to a supervised learningproblem:
Input Output
X1 Y1
X2 Y2
… …
Xn Yn
cat
baby
Data
baby InformationBottleneck
available but limited
Ø Convert to a supervised learningproblem:Input Output
X1 Y1
X2 Y2
… …
Xn Yn
cat
baby
Data
baby
InformationBottleneck