biological and artificial neuron
DESCRIPTION
Neural Networks. Learning Processes. Biological and Artificial Neuron. Weights, need to be determined. Biological neuron. Bias, need to be determined. Artificial neuron. Neural Networks. Learning Processes. Application of Neural Networks. Function approximation and prediction - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/1.jpg)
Lecture 2
Introduction to Neural Networksand Fuzzy Logic
President University Erwin Sitompul NNFL 2/1
Dr.-Ing. Erwin SitompulPresident University
http://zitompul.wordpress.com
2 0 1 3
![Page 2: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/2.jpg)
President University Erwin Sitompul NNFL 2/2
soma
synapse
dendrite
axon
( )y f net( )f
1w
2w
3w
b
1y
3y
2y
1
net
Weights, need to be determinedBiological neuron
Artificial neuron
Bias, need to be determined
Learning ProcessesNeural Networks
Biological and Artificial Neuron
![Page 3: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/3.jpg)
President University Erwin Sitompul NNFL 2/3
Learning ProcessesNeural Networks
Application of Neural NetworksFunction approximation and predictionPattern recognitionSignal processingModeling and controlMachine learning
![Page 4: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/4.jpg)
President University Erwin Sitompul NNFL 2/4
Building a Neural NetworkSelect Structure: design the way that the neurons are
interconnected.Select weights: decide the strengths with which the
neurons are interconnected.Weights are selected to get a “good match” of
network output to the output of a training set.Training set is a set of inputs and desired outputs.The weight selection is conducted by the use of a
learning algorithm.
Learning ProcessesNeural Networks
![Page 5: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/5.jpg)
President University Erwin Sitompul NNFL 2/5
Stage 1: Network Training
Training Data
Stage 2: Network Validation
Artificial neural network
Input and output sets, adequate coverage
Learning Process
In the form of a set of optimized synaptic weights and biases
Unseen Data
From the same range as the training data
Artificial neural network
ImplementationPhase
Learning ProcessesNeural Networks
Learning Process
Knowledge
Output Prediction
![Page 6: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/6.jpg)
President University Erwin Sitompul NNFL 2/6
Learning ProcessLearning is a process
by which the free parameters of a neural network are adapted through a process of stimulation by the environment in which the network is embedded.
In most cases, due to complex optimization plane, the optimized weights and biases are obtained as a result of a number of learning iterations.
[w,b]x y
[w,b]0x y(0)
Initialize: Iteration (0)
[w,b]1x y(1)
Iteration (1)
[w,b]nx y(n) ≈ d
Iteration (n)
ANN
d : desired output
…
Learning ProcessesNeural Networks
![Page 7: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/7.jpg)
President University Erwin Sitompul NNFL 2/7
Learning RulesLearning ProcessesNeural Networks
Error Correction LearningDelta Rule or Widrow-Hoff Rule
Memory Based LearningNearest Neighbor Rule
Hebbian LearningSynchronous activation increases the synaptic
strengthAsynchronous activation decreases the synaptic
strengthCompetitive LearningBoltzmann Learning
![Page 8: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/8.jpg)
President University Erwin Sitompul NNFL 2/8
wk1(n)x1
x2
xm
Inp
uts
Synapticweights
Bias
Activation
function
wk2(n)
wkm(n)
SOutputyk (n)
Desired outputdk (n)
ek (n)
+
f(.)
bk(n)
1
-
Errorsignal
Learning ProcessesNeural Networks
Error-Correction Learning
S
LearningRule
![Page 9: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/9.jpg)
President University Erwin Sitompul NNFL 2/9
Learning ProcessesNeural Networks
Delta Rule (Widrow-Hoff Rule)
Minimization of a cost function (or performance index)
21( ) ( )
2 kn e nE
![Page 10: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/10.jpg)
President University Erwin Sitompul NNFL 2/10
wkj(0) = 0
yk(n) = S [wkj(n) xj(n)]
wkj(n+1) = wkj(n) + h [dk(n) – yk(n)] xj(n)
h : learning rate, [0…1]
n = n+1
( 1) ( )kj kjkj
w n w nw
E
n = 0 Least Means Squares Rule
Learning ProcessesNeural Networks
Delta Rule (Widrow-Hoff Rule)
![Page 11: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/11.jpg)
President University Erwin Sitompul NNFL 2/11
Learning ProcessesNeural Networks
Learning Paradigm
SANN
Error
Desired
Actual
+-
Environment(Data)
Teacher(Expert)
Supervised Unsupervised
Environment(Data)
Delay
ANN
Delayed Reinforcement
Learning
CostFunction
![Page 12: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/12.jpg)
President University Erwin Sitompul NNFL 2/12
Single Layer PerceptronsNeural Networks
Single Layer Perceptrons
• Output unit is independent of the others.• Analysis can be limited to
single output perceptron.
Single-layer perceptron network is a network with all the inputs connected directly to the output(s).
![Page 13: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/13.jpg)
President University Erwin Sitompul NNFL 2/13
Single Layer PerceptronsNeural Networks
Derivation of a Learning Rule for Perceptrons
w1w2
E(w
)
Key idea: Learning is performed by adjusting the weights in order to minimize the sum of squared errors on a training.
Weights are updated repeatedly (in each epoch/iteration).
Sum of squared errors is a classical error measure (e.g. commonly used in linear regression).
Learning can be viewed as an optimization search problem in weight space.
![Page 14: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/14.jpg)
President University Erwin Sitompul NNFL 2/14
Single Layer PerceptronsNeural Networks
Derivation of a Learning Rule for Perceptrons The learning rule performs a search within the
solution's vector space towards a global minimum.
The error surface itself is a hyper-paraboloid but is seldom as smooth as is depicted below.
In most problems, the solution space is quite irregular with numerous pits and hills which may cause the network to settle down in a local minimum (not the best overall solution).
Epochs are repeated until stopping criterion is reached (error magnitude, number of iterations, change of weights, etc).
![Page 15: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/15.jpg)
President University Erwin Sitompul NNFL 2/15
Single Layer PerceptronsNeural Networks
Derivation of a Learning Rule for Perceptrons
Widrow [1962]
x1
x2
xm
wk1
wk2
wkm
.
.
.
xwTk xwT
kky
Adaline(Adaptive Linear Element)
Goal: Tk kky d w x
![Page 16: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/16.jpg)
President University Erwin Sitompul NNFL 2/16
Least Mean Squares (LMS)Single Layer PerceptronsNeural Networks
2
1
1( ) ( ) ( )
2
p
k k ki
E d i y i
w
2T
1
1( ) ( )
2
p
k k ki
d i i
w x
2
1 1
1( ) ( )
2
p m
k kj ji j
d i w x i
The following cost function (error function) should be minimized:
![Page 17: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/17.jpg)
President University Erwin Sitompul NNFL 2/17
Single Layer PerceptronsNeural Networks
Least Mean Squares (LMS)
12
21
k k kkm
mk kdw dwdf w
f
w wd
f f
w
2
T
1, , ,k k kmdw dw dww
T
1 2
, , ,k k km
f f ff
w w w
Letting f(wk) = f (wk1, wk2,…, wkm) be a function over Rm, then
Defining
df f w
![Page 18: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/18.jpg)
President University Erwin Sitompul NNFL 2/18
fw fw
df : positive df : zero df : negativego uphill plain go downhill
fw
To minimize f , we choose
df f w
f w
Single Layer PerceptronsNeural Networks
Gradient Operator
![Page 19: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/19.jpg)
President University Erwin Sitompul NNFL 2/19
Single Layer PerceptronsNeural Networks
Adaline Learning Rule
With
2
1 1
1( ) ( ) ( ) ,
2
p m
k k kj ji j
E d i w x i
w
T
1 2
( ) ( ) ( )( ) , , ,k k k
kk k km
E E EE
w w w
w w ww then
As already obtained before,
( )k kE w w Weight Modification Rule
1
( )( ) ( )
pk
k jikj
Ei x i
w
w
( ) ( ) ( )k k ki d i y i Defining
we can write
![Page 20: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/20.jpg)
President University Erwin Sitompul NNFL 2/20
Single Layer PerceptronsNeural Networks
Adaline Learning Modes
Batch Learning Mode
1
( ) ( )ki
kj
p
jx iw i
Incremental Learning Mode
k jkjw x
![Page 21: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/21.jpg)
President University Erwin Sitompul NNFL 2/21
-Learning Rule LMS Algorithm Widrow-Hoff Learning Rule
Single Layer PerceptronsNeural Networks
Adaline Learning Rule
Tˆ ˆ ˆ( 1) ( ) ( ) ( ) ( ) ( )n n n d n n n w w x x wˆ ˆ( 1) ( ) ( )( ( ) ( ))n n n d n y n w w xˆ ˆ( 1) ( ) ( ) ( )n n n e n w w x
![Page 22: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/22.jpg)
President University Erwin Sitompul NNFL 2/22
Single Layer PerceptronsNeural Networks
Generalization and Early Stopping By proper training, a neural
network may produce reasonable output for inputs not seen during training Generalization
Generalization is particularly useful for the analysis of a “noisy” data (e.g. time–series)
“Overtraining” will not improve the ability of a neural network to produce good output.
On the contrary, it will try to take noise as the real data and lost its generality.
![Page 23: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/23.jpg)
President University Erwin Sitompul NNFL 2/23
Generalization and Early Stopping Single Layer PerceptronsNeural Networks
Learning data set
Modelin
g e
rror
Number of iteration in optimization
Validation data set
Early stopping area
Overfitting vs Generalization
![Page 24: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/24.jpg)
President University Erwin Sitompul NNFL 2/24
Homework 1Single Layer PerceptronsNeural Networks
Given a function y = 4x2, you are required to find the value of x that will result y = 2 by using the Least Mean Squares method.Use initial estimate x0 = 1 and learning rate η = 0.01.Write down the results of the first 10 epochs/iterations.Give conclusion about your result.
Note: Calculation can be done manually or using Matlab.
![Page 25: Biological and Artificial Neuron](https://reader035.vdocuments.us/reader035/viewer/2022062301/56815ab7550346895dc86ee7/html5/thumbnails/25.jpg)
President University Erwin Sitompul NNFL 2/25
Homework 1ASingle Layer PerceptronsNeural Networks
Given a function y = 2x3 + cos2x, you are required to find the value of x that will result y = 5 by using the Least Mean Squares method.Use initial estimate x0 = 0.2*Student ID and learning
rate η = 0.01.Write down the results of the first 10 epochs/iterations.Give conclusion about your result.
Note: Calculation can be done manually or using Matlab/Excel.