perceptrons gary cottrell. cognitive science summer school 2 perceptrons: a bit of history frank...

Perceptrons

Gary CottrellGary Cottrell

Cognitive Science Summer School2

Perceptrons: A bit of Perceptrons: A bit of historyhistory

Frank Rosenblatt studied a simple version of a neural net called a perceptron:

A single layer of processing Binary output Can compute simple things like (some)

boolean functions (OR, AND, etc.)

Cognitive Science Summer School3

Perceptrons: A bit of Perceptrons: A bit of historyhistory

Computes the weighted sum of its inputs (the net input, compares it to a threshold, and “fires” if the net is greater than or equal than the threshold.

(Note in the picture it’s “>”, but we will use “>=“ to make sure you’re awake!)

Cognitive Science Summer School 4

The Perceptron Activation Rule

This is called a linear threshold unit

net inputnet input

output output


QuizQuiz

Assume: FALSE == 0, TRUE==1, so if X1 is false, it is 0.

Can you come up with a set of weights and a threshold so that a two-input perceptron computes OR?

X1 X2 X1 OR X2

0 0 0

0 1 1

1 0 1

1 1 1


QuizQuiz

Assume: FALSE == 0, TRUE==1

Can you come up with a set of weights and a threshold so that a two-input perceptron computes AND?

X1 X2 X1 AND X2

0 0 0

0 1 0

1 0 0

1 1 1


QuizQuiz

Assume: FALSE == 0, TRUE==1Can you come up with a set of weights and a threshold so that a two-input perceptron computes XOR?

X1 X2 X1 XOR X2

0 0 0

0 1 1

1 0 1

1 1 0


PerceptronsPerceptrons

The goal was to make a neurally-inspired machine that could categorize inputs – and learn to do this from examples

Learning: A bit of historyLearning: A bit of history

Rosenblatt (1962) discovered a Rosenblatt (1962) discovered a learning rule for perceptrons called learning rule for perceptrons called the the perceptron convergence perceptron convergence procedureprocedure..

Guaranteed to learn anything Guaranteed to learn anything computable (by a two-layer computable (by a two-layer perceptron)perceptron)

Unfortunately, not everything was Unfortunately, not everything was computable (Minsky & Papert, 1969)computable (Minsky & Papert, 1969)

Cognitive Science Summer School904/21/23

10

Perceptron LearningPerceptron Learning It is It is supervised learningsupervised learning::

There is a set of input patternsThere is a set of input patterns and a set of desired outputs (the and a set of desired outputs (the targetstargets or or

teachingteaching signalsignal)) The network is presented with the inputs, The network is presented with the inputs,

and FOOMP, it computes the output, and and FOOMP, it computes the output, and the output is compared to the target.the output is compared to the target.

If they don’t match, it changes the weights If they don’t match, it changes the weights and threshold so it will get closer to and threshold so it will get closer to producing the target next time.producing the target next time.

Cognitive Science Summer School


Perceptron Learning Perceptron Learning

First, get a training set - let’s choose ORFirst, get a training set - let’s choose ORFour “patterns”:Four “patterns”:

X1 X2 X1 OR X2

0 0 0

0 1 1

1 0 1

1 1 1


Perceptron Learning Made Perceptron Learning Made SimpleSimple

Output activation rule: Output activation rule: First, compute the First, compute the output of the networkoutput of the network::

Learning rule:Learning rule:If output is If output is 11 and should be and should be 00, then , then lower lower weights weights

to active inputs and to active inputs and raiseraise the threshold θ the threshold θ If output is If output is 00 and should be and should be 11, then , then raise raise weights weights

to active inputs and to active inputs and lowerlower the threshold θ the threshold θ

((““active inputactive input”” means x means xi i = = 11, not , not 0)0)


Perceptron Learning Perceptron Learning First, get a training set - let’s choose OR Four “patterns”:

Now, randomly present these to the network, apply the learning rule, and continue until it doesn’t make any mistakes.

X1 X2 X1 OR X2

0 0 0

0 1 1

1 0 1

1 1 1


STOP HERE FOR TRAINING DEMO of OR.

Cognitive Science Summer School

15

Characteristics of Characteristics of perceptron learningperceptron learning

Supervised learning: Gave it a set of input-output examples for it to model the function (a teaching signal)

Error correction learning: only corrected it when it is wrong (never praised! ;-))

Random presentation of patterns. Slow! Learning on some patterns ruins

learning on others. This can explain the U-shaped learning of

the past tense in English.

04/21/23


Perceptron Learning Made Perceptron Learning Made Simple for Computer Simple for Computer

ScienceScience Output activation rule:

First, compute the output of the network:

Learning rule:If output is 1 and should be 0, then lower weights

to active inputs and raise the threshold θIf output is 0 and should be 1, then raise weights

to active inputs and lower the threshold θ(“active input” means xi = 1, not 0)


Perceptron Learning Made Perceptron Learning Made SimpleSimple

for Computer Sciencefor Computer Science Learning rule:If output is 1 and should be 0, then lower weights to active

inputs and raise the threshold If output is 0 and should be 1, then raise weights to active

inputs and lower the threshold Easy one-line program to do this:

wi(t+1) = wi(t) + α(teacher – output)xi

(α is the learning rate) This is known as the delta rule because learning is based on

the delta (difference) between what you did and what you should have done: δ = (teacher - output)


Let’s convince ourselves Let’s convince ourselves these are the same…these are the same…

Learning rule:If output is 1 and should be 0, then lower weights to active

inputs and raise the threshold


wi(t+1) = wi(t) + α(0 – 1)1 (“xi is an active input” means xi = 1)

= wi(t) – α lower weight

What if xi is inactive?


wi(t+1) = wi(t) + α(teacher – output)0

= wi(t) (no change)


Let’s convince ourselves Let’s convince ourselves these are the same…these are the same…

Learning rule:If output is 0 and should be 1, then raise weights to active

inputs and lower the threshold


wi(t+1) = wi(t) + α(1 – 0)1 = wi(t) + α (raise weight)


LetLet’’s convince ourselves s convince ourselves these are the same…these are the same…

What about the threshold? We just treat θ as a weight from a unit that is always a constant -1. If output is 1 and should be 0, then lower weights to active

inputs and raise the threshold Learning rule:


I.e.: θ(t+1) = θ(t) + α(0 – 1)(-1) θ(t+1) = θ(t) + α(–1)(-1) θ(t+1) = θ(t) + α raise θ


LetLet’’s convince ourselves s convince ourselves these are the same…these are the same…

What if we get it right??? Then teacher = output, and …

Learning rule:


wi(t+1) = wi(t) + α(0)xi

= wi(t) (no change)


But wait! There’s more!!!But wait! There’s more!!! Learning rule:


(α is the learning rate)

In fact, this learning rule works for arbitrary numeric inputs.

I.e., the xi’s don’t need to be binary – they can be real numbers, or integers (like pixel values!).

So, I can train a perceptron to be a “Gary detector,” since an image is just a table of pixel values.

(Illustrate on board)


But wait! There’s more!!!But wait! There’s more!!! Learning rule:


(α is the learning rate)

In fact, this learning rule works for linear outputs I.e., we can use this to do linear regression! (Illustrate on board) The learning rule itself can be derived as

minimizing the euclidean distance between the desired outputs (the teaching signal) and the actual outputs.


But wait! There’s more!!!But wait! There’s more!!! A simple variant gives us logistic regression Logistic regression, if you don’t know what that is,

is when you have a nominal output variable, e.g., “give this guy a loan? Or not?”

This ends up being a kind of “soft” perceptron. It is more useful than a perceptron because it gives

a probability of the output being true. (Do Demo, if possible)

perceptrons gary cottrell. cognitive science summer school 2 perceptrons: a bit of history frank...

Documents