elec 576: neural networks & backpropagation lecture 3 · 2017. 9. 13. · elec 576: neural...
TRANSCRIPT
![Page 1: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/1.jpg)
ELEC 576: Neural Networks & Backpropagation
Lecture 3Ankit B. Patel
Baylor College of Medicine (Neuroscience Dept.) Rice University (ECE Dept.)
09-06-2016
![Page 2: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/2.jpg)
Administrivia• We’ve increased the limit in the class (again)
• Assignment #0 will be available on website this evening (not for credit but very very important!)
• Register for the class on Gradescope (detailed instructions in Asst #0)
• Let’s take a picture to let the Registrar know we need more space!
![Page 3: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/3.jpg)
Latest News
![Page 4: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/4.jpg)
TF-Slim: A High Level Library to Define Complex Models in Tensorflow
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim
![Page 5: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/5.jpg)
Baidu Release Its Open-source Platformhttps://github.com/baidu/paddle
![Page 6: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/6.jpg)
Self-Driving Cars: Tesla Autopilot
• NYTimes Video
• https://www.youtube.com/watch?v=5TjbqVartjM
![Page 7: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/7.jpg)
Tesla Autopilot Accident…due to a misclassification error
![Page 8: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/8.jpg)
Why?• Mis-Classification: Visual perception system misclassified tractor
trailer as billboard/overhead road sign
• Depth Ambiguity: radar system also ignored truck because otherwise it leads to too many false positive braking events (overhead road signs
• Video Tracking/Dynamical Modeling: system did not translate knowledge from earlier in time (that there was indeed a tractor trailer moving across field of view) into current understanding
![Page 9: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/9.jpg)
Potential Solutions (Project Ideas)• Lesson: Good performance on tasks is not sufficient. It is absolutely
critical that we understand why our NNs make the decisions they do — we need human-interpretable explanations for their decisions.
• Project Ideas:
• How to probe trained NNs to see what factors led it to its decision? Applications in Self-Driving Cars, Medical Imaging, etc.
• Dynamical Modeling/Video Understanding with Convnets
• Depth Perception Jointly with Image Classification
![Page 10: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/10.jpg)
Outline• Neural Networks
• Definition of NN and terminology
• Review of (Old) Theoretical Results about NNs
• Intuition for why compositions of nonlinear functions are more expressive
• Expressive power theorems [McC-Pitts, Rosenblatt, Cybenko]
• Backpropagation algorithm (Gradient Descent + Chain Rule)
• History of backdrop summary
• Gradient descent (Review).
• Chain Rule (Review).
• Backprop
• Intro to Convnets
• Convolutional Layer, ReLu, Max-Pooling
![Page 11: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/11.jpg)
Neural Networks
![Page 12: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/12.jpg)
Neural Network
![Page 13: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/13.jpg)
Expressive Power Theorems
![Page 14: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/14.jpg)
Compositions of Nonlinear Functions are more expressive
[Yoshua Bengio]
![Page 15: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/15.jpg)
McCulloch-Pitts Neurons
![Page 16: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/16.jpg)
Expressive Power of McCulloch-Pitts Nets
![Page 17: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/17.jpg)
The Perceptron (Rosenblatt)
![Page 18: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/18.jpg)
Limitations of Perceptron• Rosenblatt was overly enthusiastic about the perceptron and made the ill-timed
proclamation that:
• "Given an elementary α-perceptron, a stimulus world W, and any classification C(W) for which a solution exists; let all stimuli in W occur in any sequence, provided that each stimulus must reoccur in finite time; then beginning from an arbitrary initial state, an error correction procedure will always yield a solution to C(W) in finite time…” [4]
• In 1969, Marvin Minsky and Seymour Papert showed that the perceptron could only solve linearly separable functions. Of particular interest was the fact that the perceptron still could not solve the XOR and NXOR functions.
• Problem outlined by Minsky and Papert can be solved by deep NNs. However, many of the artificial neural networks in use today still stem from the early advances of the McCulloch-Pitts neuron and the Rosenblatt perceptron.
![Page 19: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/19.jpg)
Universal Approximation Theorem [Cybenko 1989, Hornik 1991]
• https://en.wikipedia.org/wiki/Universal_approximation_theorem
![Page 20: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/20.jpg)
Universal Approximation Theorem• https://en.wikipedia.org/wiki/Universal_approximation_theorem
• Shallow neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters.
• Proved by George Cybenko in 1989 for sigmoid activation functions.[2]
• Kurt Hornik showed in 1991[3] that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators.
![Page 21: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/21.jpg)
Backpropagation
![Page 22: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/22.jpg)
Gradient Descent
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 23: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/23.jpg)
How to Learn NNs? History of the Backpropagation Algorithm (1960-86)
• Introduced by Henrey J. Kelley (1960) and Arthur Bryson (1961) in control theory, using Dynamic Programming
• Simpler derivation using Chain Rule by Stephen Dreyfus (1962)
• General method for Automatic Differentiation by Seppo Linnainamaa (1970)
• Using backdrop for parameters of controllers minimizing error by Stuart Dreyfus (1973)
• Backprop brought into NN world by Paul Werbos (1974)
• Used it to learn representations in hidden layers of NNs by Rumelhart, Hinton & Williams (1986)
![Page 24: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/24.jpg)
Gradient Descent
![Page 25: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/25.jpg)
Gradient Descent
![Page 26: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/26.jpg)
Interesting Question:What kind of problems might you run into with Gradient Descent? (2 min)
![Page 27: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/27.jpg)
Learning Rate Needs to Be Carefully Chosen
![Page 28: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/28.jpg)
Global Optima is not Guaranteed
![Page 29: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/29.jpg)
Chain Rule
![Page 30: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/30.jpg)
Chain Rule
![Page 31: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/31.jpg)
Chain Rule
![Page 32: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/32.jpg)
Fun Exercise:Do Chain Rule on nested function (5
min)
![Page 33: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/33.jpg)
Interesting Question:What could go wrong with very deep
functions? (3 min)
![Page 34: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/34.jpg)
Neural Network: Definitions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
Input Units
Output Units
Net Input (Output)
Activation
![Page 35: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/35.jpg)
Neural Networks: Activation Functions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
![Page 36: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/36.jpg)
Neural Networks: Definitions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
Feedforward Propagation: Scalar Form
Input Units
Output Units
Net Input
(Output) Activation
Hidden Units
![Page 37: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/37.jpg)
Neural Networks: Definitions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
Feedforward Propagation: Vector Form
Input Units
Output Units
Net Input
(Output) Activation
Hidden Units
![Page 38: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/38.jpg)
Neural Networks: Definitions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
Deep Feedforward Propagation: Vector Form
![Page 39: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/39.jpg)
Neural Networks: Definitions
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
The Training Objective
![Page 40: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/40.jpg)
Pseudo-Code for Backprop: Scalar Form
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
![Page 41: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/41.jpg)
Pseudo-Code for Backprop: Matrix-Vector Form
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
![Page 42: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/42.jpg)
Gradient Descent for Neural Networks
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
![Page 43: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/43.jpg)
Backpropagation: Network View
![Page 44: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/44.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 45: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/45.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 46: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/46.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 47: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/47.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 48: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/48.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 49: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/49.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 50: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/50.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 51: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/51.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 52: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/52.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 53: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/53.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 54: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/54.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 55: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/55.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 56: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/56.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 57: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/57.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 58: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/58.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 59: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/59.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 60: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/60.jpg)
Example
[Fei-Fei Li, Andrej Karpathy, Justin Johnson]
![Page 61: ELEC 576: Neural Networks & Backpropagation Lecture 3 · 2017. 9. 13. · ELEC 576: Neural Networks & Backpropagation Lecture 3 Ankit B. Patel Baylor College of Medicine (Neuroscience](https://reader034.vdocuments.us/reader034/viewer/2022052016/602e56cc0c5ad44e67505f77/html5/thumbnails/61.jpg)
Classification with 2-Layer Neural Network
• http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html