neural networks tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · csc411 tutorial...
TRANSCRIPT
![Page 1: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/1.jpg)
CSC411 Tutorial #5 Neural Networks
Oct, 2017 Shengyang Sun
*Based on the lectures given by Professor Sanja Fidler and the prev. tutorial by Boris Ivanovic, Yujia Li.
![Page 2: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/2.jpg)
High-Level Overview
• A Neural Network is a function! • It (generally) comprised of: – Neurons which pass input values through
functions and output the result –Weights which carry values between neurons
• We group neurons into layers. There are 3 main types of layers: – Input Layer – Hidden Layer(s) – Output Layer
![Page 3: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/3.jpg)
High-Level Overview
![Page 4: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/4.jpg)
Neuron Breakdown
neuron
weight
activation
x1
X2
x3
w1
w2w3
![Page 5: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/5.jpg)
Neuron Breakdown
b
![Page 6: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/6.jpg)
Activation Functions
Most popular recently for deep learning
![Page 7: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/7.jpg)
Representation Power
![Page 8: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/8.jpg)
What does this mean?
• Neural Networks are POWERFUL, it’s exactly why with recent computing power there was a renewed interest in them.
BUT • “With great power comes great overfitting.” – Boris Ivanovic, 2016
• Last slide, “20 hidden neurons” is an example.
![Page 9: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/9.jpg)
How to mitigate this?
• Stay Tuned!
• First, how do we even use or train neural networks?
![Page 10: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/10.jpg)
Training Neural Networks (Key Idea)
(Regression)
(Classification)
![Page 11: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/11.jpg)
Training Compared to Other Models
• Training Neural Networks is a NON-CONVEX OPTIMIZATION PROBLEM.
• This means we can run into many local optima during training.
Weight
![Page 12: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/12.jpg)
Training Neural Networks (Implementation)
• We need to first perform a forward pass • Then, we update weights with a
backward pass
![Page 13: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/13.jpg)
Forward Pass (AKA “Inference”)
![Page 14: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/14.jpg)
Backward Pass (AKA “Backprop.”)
![Page 15: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/15.jpg)
Learning Weights during Backprop
• Do exactly what we’ve been doing! • Take the derivative of the error/cost/loss
function w.r.t. the weights and minimize via gradient descent!
![Page 16: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/16.jpg)
Useful Derivatives
![Page 17: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/17.jpg)
Preventing Overfitting
Weight decay
Early stop
![Page 18: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/18.jpg)
Limiting the Size of the Weights
![Page 19: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/19.jpg)
The Effects of Weight-Decay
![Page 20: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/20.jpg)
Early Stopping
![Page 21: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/21.jpg)
Why Early Stopping Works
![Page 22: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/22.jpg)
Neural Network Visualizations
• http://playground.tensorflow.org • http://scs.ryerson.ca/~aharley/vis/conv/
![Page 23: Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun ssy@cs.toronto.edu *Based on the lectures](https://reader030.vdocuments.us/reader030/viewer/2022021715/5c8034ae09d3f257328c5c10/html5/thumbnails/23.jpg)
Questions