introduction to machine learning10701/slides/8_perceptron.pdf · 2017. 9. 28. · introduction to...
TRANSCRIPT
![Page 1: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/1.jpg)
Introduction to Machine Learning
Perceptron
Barnabás Póczos
![Page 2: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/2.jpg)
2
Contents
History of Artificial Neural Networks
Definitions: Perceptron, Multi-Layer Perceptron
Perceptron algorithm
![Page 3: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/3.jpg)
3
Short History of Artificial Neural Networks
![Page 4: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/4.jpg)
4
Progression (1943-1960)
• First mathematical model of neurons
▪ Pitts & McCulloch (1943)
• Beginning of artificial neural networks
• Perceptron, Rosenblatt (1958)
▪ A single neuron for classification
▪ Perceptron learning rule
▪ Perceptron convergence theorem
Degression (1960-1980)
• Perceptron can’t even learn the XOR function
• We don’t know how to train MLP
• 1963 Backpropagation… but not much attention…
Bryson, A.E.; W.F. Denham; S.E. Dreyfus. Optimal programming problems with inequality constraints. I: Necessary conditions for extremal solutions. AIAA J. 1, 11 (1963) 2544-2550
Short History
![Page 5: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/5.jpg)
5
Progression (1980-)
• 1986 Backpropagation reinvented:
▪ Rumelhart, Hinton, Williams: Learning representations by back-propagating errors. Nature, 323, 533—536, 1986
• Successful applications:
▪ Character recognition, autonomous cars,…
• Open questions: Overfitting? Network structure? Neuron number? Layer number? Bad local minimum points? When to stop training?
• Hopfield nets (1982), Boltzmann machines,…
Short History
![Page 6: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/6.jpg)
6
Degression (1993-)
• SVM: Vapnik and his co-workers developed the Support Vector Machine (1993). It is a shallow architecture.
• SVM and Graphical models almost kill the ANN research.
• Training deeper networks consistently yields poor results.
• Exception: deep convolutional neural networks, YannLeCun 1998. (discriminative model)
Short History
![Page 7: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/7.jpg)
7
Short History
Deep Belief Networks (DBN)
• Hinton, G. E, Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554.
• Generative graphical model
• Based on restrictive Boltzmann machines
• Can be trained efficiently
Deep Autoencoder based networks
Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H. (2007). Greedy Layer-Wise Training of Deep Networks, Advances in Neural Information Processing Systems 19
Convolutional neural networks running on GPUsAlex Krizhevsky, Ilya Sutskever, Geoffrey Hinton, Advances in Neural Information Processing Systems 2012
Progression (2006-)
![Page 8: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/8.jpg)
8
The Neuron
![Page 9: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/9.jpg)
9
The Neuron
– Each neuron has a body, axon, and many dendrites
– A neuron can fire or rest
– If the sum of weighted inputs larger than a threshold, then the neuron
fires.
– Synapses: The gap between the axon and other neuron’s dendrites. It
determines the weights in the sum.
![Page 10: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/10.jpg)
10
The Mathematical Model of a Neuron
![Page 11: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/11.jpg)
11
• Identity function
• Threshold function
(perceptron)
• Ramp function
Typical activation functions
![Page 12: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/12.jpg)
12
• Logistic function
Typical activation functions
• Hyperbolic tangent function
![Page 13: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/13.jpg)
13
Typical activation functions
• Rectified Linear Unit (ReLU)
• Exponential Linear Unit
• Softplus function(This is a smooth approximation of ReLU)
• Leaky ReLU
![Page 14: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/14.jpg)
14
![Page 15: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/15.jpg)
15
![Page 16: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/16.jpg)
16
Structure of Neural Networks
![Page 17: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/17.jpg)
17
Input neurons, Hidden neurons, Output neurons
Fully Connected Neural Network
![Page 18: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/18.jpg)
18
Layers, Feedforward neural networks
Convention: The input layer is Layer 0.
![Page 19: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/19.jpg)
19
• Multilayer perceptron: Connections only between Layer i and Layer i+1
• The most popular architecture.
Multilayer Perceptron
![Page 20: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/20.jpg)
20
![Page 21: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/21.jpg)
21
Recurrent Neural Networks
Recurrent NN: there are connections backwards too.
![Page 22: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/22.jpg)
22
The Perceptron
![Page 23: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/23.jpg)
23
The Training Set
![Page 24: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/24.jpg)
24
The Perceptron
![Page 25: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/25.jpg)
25
1
-1
The Perceptron
![Page 26: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/26.jpg)
Matlab: opengl hardwarebasic, nnd4pr
![Page 27: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/27.jpg)
27
Matlab demos: nnd3pc
![Page 28: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/28.jpg)
28
The Perceptron Algorithm
![Page 29: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/29.jpg)
The Perceptron algorithm
29
The perceptron learning algorithm
![Page 30: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/30.jpg)
30
The perceptron algorithm
Observation
![Page 31: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/31.jpg)
The Perceptron Algorithm
31
How can we remember this rule?
An interesting property:
we do not require the learning rate to go to zero!
![Page 32: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/32.jpg)
The Perceptron Algorithm
32
![Page 33: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/33.jpg)
Perceptron Convergence
33
![Page 34: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/34.jpg)
Perceptron Convergence
34
![Page 35: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/35.jpg)
Perceptron Convergence
35
Lemma Using this notation, the update rule can be written as
Proof
![Page 36: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/36.jpg)
Perceptron Convergence
36
Lemma
![Page 37: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/37.jpg)
Perceptron Convergence
37
![Page 38: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/38.jpg)
Lower bound
38
![Page 39: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/39.jpg)
Upper bound
39
Therefore,
![Page 40: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/40.jpg)
Upper bound
40
Therefore,
![Page 41: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/41.jpg)
The Perceptron Algorithm
41
![Page 42: Introduction to Machine Learning10701/slides/8_Perceptron.pdf · 2017. 9. 28. · Introduction to Machine Learning Perceptron Barnabás Póczos. 2 Contents History of Artificial Neural](https://reader035.vdocuments.us/reader035/viewer/2022071503/612398879d50d25615505b15/html5/thumbnails/42.jpg)
Take me home!
History of Neural Networks
Mathematical model of the neuron
Activation Functions
Perceptron definition
Perceptron algorithm
Perceptron Convergence Theorem
42