artificial neural networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf ·...
TRANSCRIPT
![Page 1: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/1.jpg)
Artificial Neural Networks Part 2/3 – Perceptron
Slides modified from Neural Network Design
by Hagan, Demuth and Beale
Berrin Yanikoglu
![Page 2: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/2.jpg)
Perceptron
• A single artificial neuron that computes its weighted input and uses a threshold activation function.
• It effectively separates the input space into two categories by the hyperplane:
wTx + b = 0
![Page 3: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/3.jpg)
Decision Boundary
The weight vector is orthogonal to the decision boundary The weight vector points in the direction of the vector which
should produce an output of 1 • so that the vectors with the positive output are on the right side of
the decision boundary – if w pointed in the opposite direction, the dot products of all input
vectors would have the opposite sign – would result in same classification but with opposite labels
The bias determines the position of the boundary
• solve for wTp+b = 0 using one point on the decision boundary to find b.
![Page 4: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/4.jpg)
Two-Input Case
w1 1, 1= w1 2, 2=
Decision Boundary: all points p for which wTp + b =0
+
-
a = hardlim(n) = [1 2]p + -2
If we have the weights and not the bias, we can take a point on the decision boundary, p=[2 0]T, and solving for [1 2]p + b = 0, we see that b=-2.
![Page 5: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/5.jpg)
Decision Boundary
wT1 p b+ 0= wT1 p bœ=
• All points on the decision boundary have the same inner product (= -b) with the weight vector
• Therefore they have the same projection onto the weight vector; so they must lie on a line orthogonal to the weight vector
wT.p = ||w||||p||Cosθ
proj. of p onto w = ||p||Cosθ
= wT.p/||w||
θ
p w
proj. of p onto w
ADVANCED
-b
![Page 6: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/6.jpg)
An Illustrative Example
![Page 7: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/7.jpg)
Boolean OR
p100
= t1 0=,⎩ ⎭⎨ ⎬⎧ ⎫ p2
01
= t2 1=,⎩ ⎭⎨ ⎬⎧ ⎫ p3
10
= t3 1=,⎩ ⎭⎨ ⎬⎧ ⎫ p4
11
= t4 1=,⎩ ⎭⎨ ⎬⎧ ⎫
Given the above input-output pairs (p,t), can you find (manually) the weights of a perceptron to do the job?
![Page 8: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/8.jpg)
Boolean OR Solution
w10.50.5
=
wT1 p b+ 0.5 0.5 00.5
b+ 0.25 b+ 0= = = b 0.25œ=⇒
2) Weight vector should be orthogonal to the decision boundary.
3) Pick a point on the decision boundary to find the bias.
1) Pick an admissable decision
boundary
![Page 9: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/9.jpg)
Matrix Form
![Page 10: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/10.jpg)
Multiple-Neuron Perceptron
W
w T 1
w T 2
w T S
=
ai hardlim ni( ) hardlim wTi p bi+( )= =
W
w 1 1 , w 1 2 , … w 1 R ,
w 2 1 , w 2 2 , … w 2 R ,
w S 1 , w S 2 , … w S R ,
=
w i
w i 1 ,
w i 2 ,
w i R ,
= i
2x1
3x2
weights of one neuron in one row of W.
![Page 11: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/11.jpg)
Multiple-Neuron Perceptron
Each neuron will have its own decision boundary.
wTi p bi+ 0=
A single neuron can classify input vectors into two categories.
An S-neuron perceptron can potentially classify input vectors into 2S categories.
![Page 12: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/12.jpg)
Perceptron Limitations
![Page 13: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/13.jpg)
Perceptron Limitations
• A single layer perceptron can only learn linearly separable problems. – Boolean AND function is linearly separable,
whereas Boolean XOR function is not.
Boolean AND Boolean XOR
![Page 14: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/14.jpg)
AND Network
x1
x2
X0=1
W0 = -0.8
W1=0.5
W2=0.5 Σ
![Page 15: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/15.jpg)
Perceptron Limitations
wT1 p b+ 0=
Linear Decision Boundary
Linearly Inseparable Problems
![Page 16: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/16.jpg)
Perceptron Limitations
For a linearly not-separable problem: – Would it help if we use more layers of neurons? – What could be the learning rule for each neuron?
Solution: Multilayer networks and the backpropagation learning algorithm
![Page 17: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/17.jpg)
• More than one layer of perceptrons (with a hardlimiting activation function) can learn any Boolean function.
• However, a learning algorithm for multi-layer perceptrons has not been developed until much later – backpropagation algorithm – replacing the hardlimiter in the perceptron with a sigmoid
activation function
![Page 18: Artificial Neural Networkspeople.sabanciuniv.edu/berrin/cs512/lectures/7-nn2-perceptron.ppt.pdf · Artificial Neural Networks Part 2/3 – Perceptron Slides modified from Neural Network](https://reader033.vdocuments.us/reader033/viewer/2022042103/5e81022959172a03b461f4a4/html5/thumbnails/18.jpg)
Summary
• So far we have seen how a single neuron with a threshold activation function separates the input space into two.
• We also talked about how more than one nodes may indicate convex (open or closed) regions
• The next slides = Backpropagation algorithm to learn the weights automatically