deep learning part 2 other networks · slides by: joseph e. gonzalez, [email protected] deep...

16
Slides by: Joseph E. Gonzalez, [email protected] Deep Learning Part 2 Other networks

Upload: ngohanh

Post on 19-May-2019

218 views

Category:

Documents


0 download

TRANSCRIPT

Slides by:

Joseph E. Gonzalez,

[email protected]

Deep Learning Part 2Other networks

Quick Logistics

Ø Please sign up to present

Ø Complete reading questions before lecture

Ø Skim the reading for the lecture you are presenting (later in semester).Ø Is there a better paper?

http://bit.ly/aisys-sp19

Machine Learning ≈ Function Approximation

Label:Cat

Object Recognition Speech Recognition

“The cat in the hat”

Robotic Control

The cat in the hat.

El gato en el sombrero

Machine Translation

Last Time

Architectures for Different kinds of inputs

Convolutional Networks

The quick brown fox…

The cat in the hat.

El gato en el sombrero

Recurrent Networks

Sequential reasoning tasksspatial reasoning tasks

Speech recognition

ReinforcementLearning

Architectures for Different kinds of inputs

Convolutional Networks

The quick brown fox…

The cat in the hat.

El gato en el sombrero

Recurrent Networks

Sequential reasoning taskspatial reasoning tasks

Speech recognition

ReinforcementLearning

Graph Networks

Operating on graph data

Link Prediction

Graph Embedding

Supervised Machine LearningØ Given data containing the function inputs and outputs

Input Output

X1 Y1

X2 Y2

… …

Xn Yn

cat

baby

Data

baby

f✓(x) ! y<latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit><latexit sha1_base64="cuKQEy7Pm3MosVAVLe4DhhYax+Y=">AAACAnicbVDLSsNAFJ3UV62vqCtxM1iEuimJCLosunFZwT6gCWEynTRDJw9mbtQSiht/xY0LRdz6Fe78G6dtFtp64MLhnHu59x4/FVyBZX0bpaXlldW18nplY3Nre8fc3WurJJOUtWgiEtn1iWKCx6wFHATrppKRyBes4w+vJn7njknFk/gWRilzIzKIecApAS155kHgORAyILWHE+xIPgiBSJnc45FnVq26NQVeJHZBqqhA0zO/nH5Cs4jFQAVRqmdbKbg5kcCpYOOKkymWEjokA9bTNCYRU24+fWGMj7XSx0EidcWAp+rviZxESo0iX3dGBEI1703E/7xeBsGFm/M4zYDFdLYoyASGBE/ywH0uGQUx0oRQyfWtmIZEEgo6tYoOwZ5/eZG0T+u2VbdvzqqNyyKOMjpER6iGbHSOGugaNVELUfSIntErejOejBfj3fiYtZaMYmYf/YHx+QPDc5cD</latexit>

Model

Parameters

Training (approximates the goal over training data):

Goal

✓⇤ = argmin✓

ED [L (f✓(x), y)]<latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit><latexit sha1_base64="Nhfau74CPzVJyPBXfLHwayNIkiQ=">AAACP3icbVBNSyNBEO3x26y6UY9eGsNCFAkzi6AXQVwFDx4UTBQy49DTqUkae3qG7hoxDPlnXvYvePPqxYMiXr3ZSebg6j5o6vFeFdX1okwKg6774ExMTk3PzM7NV34sLC79rC6vtEyaaw5NnspUX0bMgBQKmihQwmWmgSWRhIvo+s/Qv7gBbUSqzrGfQZCwrhKx4AytFFZbPvYA2dUm3aM+013qJ0KFY9Fyhr0oKo4GYXE48CXE2KYno1qPy6b67cYWpX1fi24PN+i4BmG15jbcEeh34pWkRkqchtV7v5PyPAGFXDJj2p6bYVAwjYJLGFT83EDG+DXrQttSxRIwQTG6f0B/WaVD41Tbp5CO1M8TBUuM6SeR7RxeZL56Q/F/XjvHeDcohMpyBMXHi+JcUkzpMEzaERo4yr4ljGth/0p5j2nG0UZesSF4X0/+Tlq/G57b8M62a/sHZRxzZI2skzrxyA7ZJ8fklDQJJ3fkkTyTF+ev8+S8Om/j1gmnnFkl/8B5/wD3Nq7j</latexit>

Loss

Over all future data

✓̂ = argmin✓

1

n

nX

i=1

L (f✓(xi), yi)<latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit><latexit sha1_base64="1nEudtxg0hj2bQCWcBS1v3yjo6I=">AAACQnicbZBPaxQxGMYztWpd/6167CV0EbYgy0wR9FIoevHgoQW3rWzW4Z3sOzuhSWZI3hGXYT6bFz+BNz+AFw+W4tVDs7tzqK0PBB6e531J8ssqrTzF8Y9o49bm7Tt3t+717j94+Ohx/8nTY1/WTuJYlrp0pxl41MrimBRpPK0cgsk0nmRnb5f9yWd0XpX2Ay0qnBqYW5UrCRSitP9RFECNoAIJWr7PBbg5F0bZdJ1xkTuQTdI2thW+Nmmj9pP2k+Xvhcachnk3N/ySqt0XnC9SJZyaF7Sb9gfxKF6J3zRJZwas02Ha/y5mpawNWpIavJ8kcUXTBhwpqbHtidpjBfIM5jgJ1oJBP21WCFr+PCQznpcuHEt8lV7daMB4vzBZmDRAhb/eLcP/dZOa8tfTRtmqJrRyfVFea04lX/LkM+VQkl4EA9Kp8FYuCwjMKFDvBQjJ9S/fNMd7oyQeJUcvBwdvOhxbbJvtsCFL2Ct2wN6xQzZmkn1lP9lvdh59i35FF9Gf9ehG1O08Y/8o+nsJv0Cw0Q==</latexit>

Last Time

Learning without Labels

Ø Can we learn what inputs look like?Ø Useful inductive bias when training for a

later supervised task.Ø Often done when labeled data is

available but limited

Ø Convert to a supervised learningproblem:

Input Output

X1 Y1

X2 Y2

… …

Xn Yn

cat

baby

Data

baby

available but limited

Ø Convert to a supervised learningproblem:

Input Output

X1 Y1

X2 Y2

… …

Xn Yn

cat

baby

Data

baby InformationBottleneck

available but limited

Ø Convert to a supervised learningproblem:Input Output

X1 Y1

X2 Y2

… …

Xn Yn

cat

baby

Data

baby

InformationBottleneck

Xor Perceptrons

Perceptron Not Gate

x

-1

1

y

1

-1

Perceptron

And Gate Perceptron

Perceptron y

-1

-1

-1

1

x1 x2-1 -1

-1 1

1 -1

1 1

Or Gate Perceptron

Perceptron y

-1111

x1 x2-1 -1-1 11 -11 1

XOr Gate Perceptron

Perceptron y

-111-1

x1 x2-1 -1-1 11 -11 1

Not+1 -1

And

+1

-1

-1

-1

+1

-1

+1

+1

-1

-1

+1

+1

Or XOr

No separating hyperplane

Using one hidden layer

1

-1

1

-1-1

-1

1

1 0

y

-1

1

1

-1

x1 x2-1 -1

-1 1

1 -1

1 1