neural networks part i the multilayer perceptron
TRANSCRIPT
![Page 1: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/1.jpg)
Neural Networks Part I
The Multilayer Perceptron
![Page 2: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/2.jpg)
Neural Networks History Lesson
2
1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
o First neuron-based learning algorithm
o Allegedly “could learn anything that you could program”
![Page 3: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/3.jpg)
Neural Networks History Lesson
3
1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
o First neuron-based learning algorithm
o Allegedly “could learn anything that you could program”
1969: Minsky & Papert, Perceptron: An Introduction to Computational Geometry
o First real complexity analysis
o Showed, in principle, many things that perceptrons can't learn to do
o Shut down any interest in neural networks
![Page 4: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/4.jpg)
Neural Networks History Lesson
4
1986: Rumelhart, Hinton & Williams, Back Propagation
o Overcame many difficulties raised by Minsky, et al
o Neural Networks wildly popular again (for a while)
![Page 5: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/5.jpg)
Neural Networks History Lesson
5
1999-2005:
Shift to Bayesian Methods: o Best ideas from neural networks
o Direct statistical computing
Support Vector Machines:o Nice mathematical properties
o Global optima vs local optima (like NNs)
![Page 6: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/6.jpg)
Neural Networks History Lesson
6
1999-2005:
Shift to Bayesian Methods: o Best ideas from neural networks
o Direct statistical computing
Support Vector Machines:o Nice mathematical properties
o Global optima vs local optima (like NNs)
A few people still playing with NNs: o Hinton (Univ. Toronto and Google)
o LeCun (NYU and Facebook)
o Bengio (Univ. Montreal)
![Page 7: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/7.jpg)
Neural Networks History Lesson
7
2005-2010:
o Core group of people continue to make improvements
o Developed various tricks to make NN learning practical
2010-Present:o Went back to the NN methods of the 1980s
o Have been wildly successful
![Page 8: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/8.jpg)
Neural Networks History Lesson
8
2005-2010:
o Core group of people continue to make improvements
o Developed various tricks to make NN learning practical
2010-Present:o Went back to the NN methods of the 1980s
o Have been wildly successful
Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?
![Page 9: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/9.jpg)
Neural Networks History Lesson
9
Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?
Efficient algorithms (Back Propagation)
Raw computing power
Massive amounts of training data:o Deep Neural Nets in particular have a massive amount of parameters
o Need tons of data to effectively train accurate models
![Page 10: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/10.jpg)
Neural Networks History Lesson
10
The history of ML has been very cyclic
But will they be 20 years from now?
¯\_(�)_/¯
![Page 11: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/11.jpg)
The Perceptron
11
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
Given some training data, learn weights associated with edges that make predictions accurate
o →
2→ Go , I }
-3( →
![Page 12: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/12.jpg)
The Perceptron
12
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
y =
⇢0 if wTx threshold1 if wTx > threshold
<latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit>
WTX = THRESH
WTX - THRESH = o
o→ -
b
¥s y
= 0I →
![Page 13: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/13.jpg)
The Perceptron
13
In its simplest form, a perceptron takes some binary inputs, and predicts a binary output
y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
ob
: v
![Page 14: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/14.jpg)
The Perceptron
14
Simple Example: Learning ORy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 OR x2 0 1 1 1<latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit>
ow=f, }b=o
I
1 wt⇒ly1t,b=t'1511,1+0--2>0
![Page 15: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/15.jpg)
The Perceptron
15
Simple Example: Learning ANDy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 AND x2 0 0 0 1<latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit>: w=[1,1 & b= -1
![Page 16: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/16.jpg)
The Perceptron
16
Simple Example: Learning NANDy =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 NAND x2 1 1 1 0<latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit>I wtxtbs El - 1 3 'Ll, ) t 1.5
![Page 17: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/17.jpg)
The Perceptron
17
Simple Example: Can we learn XOR?y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
1= =
41 wtx = +1 )
L
Wtxtb EO
![Page 18: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/18.jpg)
The Perceptron
18
Simple Example: Can we learn XOR?
NOPE!
y =
⇢0 if wTx+ b 01 if wTx+ b > 0
<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>
x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
![Page 19: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/19.jpg)
The Perceptron
19
Question: Why can’t we learn XOR?
OR AND
NAND XOR
W
![Page 20: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/20.jpg)
The Perceptron
20
Question: Why can’t we learn XOR?
Answer: The perceptron is a linear classifier. Can only learn things that are linearly separable
Question: How can we fix this?
![Page 21: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/21.jpg)
The Multilayer Perceptron
21
Question: How can we fix this?
Answer: The Multilayer Perceptron
Anatomy: o Input Layer
o Hidden Layer
o Output Layera.12 bz
onW 22=
![Page 22: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/22.jpg)
The Multilayer Perceptron
22
Anatomy: Input Layer to Hidden Layer
o Define intermediate activities z
o Define activations a
The indicator I is the activation function
z1 = w111x1 + w1
12x2 + b11, a1 = I(z1 > 0)
z2 = w121x1 + w1
22x2 + b12, a2 = I(z2 > 0)<latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit>
a
:#=¥ I
![Page 23: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/23.jpg)
The Multilayer Perceptron
23
Vectorized Anatomy: Input Layer to Hidden Layer
becomes
where
z1 = w111x1 + w1
12x2 + b11, a1 = I(z1 > 0)z2 = w1
21x1 + w122x2 + b12, a2 = I(z2 > 0)
<latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit>
z2 = W 1x1 + b1
a2 = I(z2 > 0) = I(W 1x1 + b1 > 0)<latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit>
W 1 =
w1
11 w112
w121 w1
22
�, b1 =
b11b12
�
<latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit>
b2,
=
× - [ It ] WE
WE
[ wiwii ]b2= Ibf }
![Page 24: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/24.jpg)
The Multilayer Perceptron
24
Vectorized Anatomy: Hidden Layer to Output Layer
where
z3 = W 2a2 + b2
y = I(z3 > 0) = I(W 2a2 + b2 > 0)<latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit>
W 2 =⇥w2
11 w212
⇤, b2 =
⇥b21
⇤<latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit>
zs = W2A2+b2T
a3=yn=I( Z ?> 0) p.ee#svres NEWFEATVRS
( Inputs ) ( HIDDEN )
![Page 25: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/25.jpg)
The Multilayer Perceptron
25
Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1
x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>
![Page 26: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/26.jpg)
The Multilayer Perceptron
26
Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
\ µaBor : T 1 11
T°3=Wl ,bl
-
ORnano : [t.IT/l.5/
/ And :[ 1 1 ) [ -1 ]
NAND W2 b2
![Page 27: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/27.jpg)
The Multilayer Perceptron
27
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
O112.2 > o )
0 0 0
0 1.5 g
°
, o o
![Page 28: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/28.jpg)
The Multilayer Perceptron
28
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
O
1 c (0 05 1
'
, 1 1
![Page 29: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/29.jpg)
The Multilayer Perceptron
29
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
![Page 30: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/30.jpg)
The Multilayer Perceptron
30
Example: XOR Check
x1 0 1 0 1x2 0 0 1 1
(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>
W 1 =
1 1
�1 �1
�, b1 =
0
1.5
�
<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>
W 2 =⇥1 1
⇤, b2 =
⇥�1
⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>
z2 = W 1x1+b1 =
1 1
�1 �1
� �+
0
1.5
�=
�, a2 = I(z2 > 0) =
�
<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>
z3 = W 2a2 + b2 =⇥1 1
⇤ �+⇥�1
⇤=
⇥ ⇤, y = I(z3 > 0) =
⇥ ⇤
<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>
1 2 1
1 - .5 0
f o o
![Page 31: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/31.jpg)
The Multilayer Perceptron
31
OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??
![Page 32: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/32.jpg)
The Multilayer Perceptron
32
OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??
We’re pretty much already there.
Things we’ll explore moving forward:
o Non-Binary Features
o Better activation functions
o How to choose architectures
o How to train these things
o Regression or Classification (Answer: Yes)
![Page 33: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/33.jpg)
33
Neural Networks I Wrap-Up
33
o Simplest network is just a regular old perceptron (a linear classifier)
o We can learn non-linear decision boundaries by chaining perceptrons
o We can represent these complicated interactions using linear algebra
Next Time: o The Feed-Forward Neural Network
![Page 34: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/34.jpg)
34
![Page 35: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/35.jpg)
35
![Page 36: Neural Networks Part I The Multilayer Perceptron](https://reader031.vdocuments.us/reader031/viewer/2022012023/6169d0b811a7b741a34bac1f/html5/thumbnails/36.jpg)
36