![Page 1: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/1.jpg)
UNIT ‐II
BPN AND BAM
![Page 2: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/2.jpg)
Back Propagation Network
• Rumelhart (early 80’s), Werbos (74),…, explosion of neural net interest
• Multi‐layer supervised learning• Able to train multi‐layer perceptrons (and other topologies)
• Uses differentiable sigmoid function which is the smooth (squashed) version of the threshold function
• Error is propagated back through earlier layers of the network
![Page 3: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/3.jpg)
Multi-Layer Perceptrons
In contrast to perceptrons, multilayer networks can learn not only multiple decisionboundaries, but the boundaries may be nonlinear. The typical architecture of a multi-layerperceptron (MLP) is shown below.
Output nodes
Internal nodes
Input nodesTo make nonlinear partitions on the space we need to define each unit as a nonlinearfunction (unlike the perceptron). One solution is to use the sigmoid unit. Another reasonfor using sigmoids are that they are continuous unlike linear thresholds and are thusdifferentiable at all points.
x1x2
xn
w1
w2
wn
Σ
w0O = σ(net) = 1 / 1 + e -net
net
![Page 4: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/4.jpg)
Back-Propagation Algorithm
Multi-layered perceptrons can be trained using the back-propagation algorithm describednext.
Goal: To learn the weights for all links in an interconnected multilayer network.
We begin by defining our measure of error:
E(W) = ½ Σd Σk (tkd – okd) 2
k varies along the output nodes and d over the training examples.
The idea is to use again a gradient descent over the space of weights to find a globalminimum (no guarantee).
Algorithm:
1. Create a network with nin input nodes, nhidden internal nodes, and nout outputnodes.
2. Initialize all weights to small random numbers.3. Until error is small do:
For each example X do
••
Propagate example X forward through the networkPropagate errors backward through the network
Internal
Forward Propagation
Given example X, compute the output of every node until we reach th
Compute sigmoidfunction
Input
Example
![Page 5: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/5.jpg)
Backward Propagation
A. For each output node k compute the error:δk = Ok (1-Ok)(tk – Ok)
B. For each hidden unit h, calculate the error:δh = Oh (1-Oh) Σk Wkh δk
C. Update each network weight:
C. Wji = Wji + ΔWji
where ΔWji = η δj Xji
(Wji and Xji are the input and weight of node i to node j)
A momentum term, depending on the weight value at last iteration, may also be added tothe update rule as follows. At iteration n we have the following:
ΔWji (n) = η δj Xji + αΔWji (n)
Where α ( 0 <= α <= 1) is a constant called the momentum.
1. It increases the speed along a local minimum.2. It increases the speed along flat regions.
Remarks on Back-propagation
1.2.3.4.
It implements a gradient descent search over the weight space.It may become trapped in local minima.In practice, it is very effective.How to avoid local minima?
b) Use stochastic gradient descent.c) Use different networks with different initial values for the weights.
Multi-layered perceptrons have high representational power. They can represent thefollowing:
1. Boolean functions. Every boolean function can be represented with a networkhaving two layers of units.
2. Continuous functions. All bounded continuous functions can also beapproximated with a network having two layers of units.
3. Arbitrary functions. Any arbitrary function can be approximated with a networkwith three layers of units.
![Page 6: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/6.jpg)
Use a validation set and stop until the error is small in this set.
Use 10 fold cross validation.
Use weight decay; the weights are decreased slowly on each iteration.
Applications of Neural Networks
Neural networks have broad applicability to real world business problems. They havealready been successfully applied in many industries.
Since neural networks are best at identifying patterns or trends in data, they are wellsuited for prediction or forecasting needs including:
sales forecastingindustrial process controlcustomer researchdata validationrisk managementtarget marketing
Generalization and overfitting:
One obvious stopping point for backpropagation is to continue iterating until the error isbelow some threshold; this can lead to overfitting.
Validation setError
Training set
Number of weight
Overfitting can be avoided using the following strategies.
![Page 7: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/7.jpg)
Applications of BPN‐Paint Quality Assessment
![Page 8: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/8.jpg)
Applications of BPN‐4:1 Video Data Compression
![Page 9: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/9.jpg)
Applications of BPN‐4:1 Video Data Compression
![Page 10: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/10.jpg)
Backpropagation Learning• Notations:
– Weights: two weight matrices:
from input layer (0) to hidden layer (1)
from hidden layer (1) to output layer (2)
weight from node 1 at layer 0 to node 2 in layer 1
– Training samples: pair of
so it is supervised learning
– Input pattern:
– Output pattern:
– Desired output:
– Error: error for output j when xp is applied
sum square error
This error drives learning (change and )
),...,( ,1, nppp xxx =),...,( ,1, kppp ooo =
),...,( ,1, kppp ddd =
)0,1(w)1,2(w)0,1(
1,2w},...,1),{( Ppdx pp =
jpjpjp dol ,,, −=
2,
1 1)( jp
P
p
K
jl∑∑
= ==
)0,1(w )1,2(w
![Page 11: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/11.jpg)
Backpropagation Learning• Sigmoid function again:
– Differentiable:
– When |net| is sufficiently large, it moves into one of the two saturation regions, behaving like a threshold or ramp function.
• Chain rule of differentiation
))(1)((
111
)()1(
1
)'1()1(
1)('1
1)(
2
2
xSxS
ee
e
ee
ee
xSe
xS
x
x
x
xx
xx
x
−=
+⋅
+=
−⋅+
−=
+⋅+
−=+
=
−
−
−
−−
−−
−
Saturationregion
Saturationregion
)(')(')(' then)(),(),( if thxgyfdtdx
dxdy
dydz
dtdzthxxgyyfz =⋅⋅====
![Page 12: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/12.jpg)
Backpropagation Learning
• Forward computing:– Apply an input vector x to input nodes
– Computing output vector x(1) on hidden layer
– Computing the output vector o on output layer
– The net is said to be a map from input x to output o
• Objective of learning:– reduce sum square error
for the given P training samples as much as possible (to
zero if possible)
)()( )1()1,2(,
)2( ∑==j
jjkkk xwSnetSo
2,
1 1)( jp
P
p
K
jl∑∑
= =
)()( )0,1(,
)1()1( ∑==i
iijjj xwSnetSx
![Page 13: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/13.jpg)
Backpropagation Learning• Idea of BP learning:
– Update of weights in w(2, 1) (from hidden layer to output layer): delta rule as in a single layer net using sum square error
– Delta rule is not applicable to updating weights in w(1, 0) (from input and hidden layer) because we don’t know the desired values for hidden nodes
– Solution: Propagating errors at output nodes down to hidden nodes, these computed errors on hidden nodes drives the update of weights in w(1, 0) (again by delta rule), thus called error BACKPROPAGATION (BP) learning
– How to compute errors on hidden nodes is the key
– Error backpropagation can be continued downward if the net has more than one hidden layer
– Proposed first by Werbos (1974), current formulation by Rumelhart, Hinton, and Williams (1986)
![Page 14: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/14.jpg)
Backpropagation Learning• Generalized delta rule:
– Consider sequential learning mode: for a given sample (xp, dp)
– Update of weights by gradient descent
For weight in w(2, 1):
For weight in w(1, 0):
– Derivation of update rule for w(2, 1):
since E is a function of lk = dk – ok, dk – ok is a function ,
and is a function of , by chain rule
)/( )1,2(,
)1,2(, jkjk wEw ∂−∂∝Δ
∑= k kplE 2, )(
)/( )0,1(,
)0,1(, ijij wEw ∂−∂∝Δ
)1,2(, jkw
)2(knet
)2(knet
![Page 15: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/15.jpg)
Backpropagation Learning
– Derivation of update rule for
consider hidden node j:
weight influences
it sends to all output nodes
∴ all K terms in E are functions of
)1(jnet
)0,1(,ijw
)0,1(,ijw
)( )1(jnetS
)0,1(,ijw
)1(,
)1(
)1(
)1(
)1(
)2(
)2(
)2( )(
ij
j
j
j
j
k
k
k
k wnet
netx
xnet
netnetS
oE
∂
∂
∂
∂
∂∂
∂∂
∂∂
i
j
ok
)0,1(,ijw
)1,2(, jkw
∑∑∑
==
==−=
i ijijjj
j jkjkkkk kk
wxnetnetSxwxnetnetSoodE
)0,1(,
)1()1()1(
)1,2(,
)1()2()2(2
),(,),(,)(
by chain rule
![Page 16: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/16.jpg)
Backpropagation Learning
– Update rules:
for outer layer weights w(2, 1) :
where
for inner layer weights w(1, 0) :
where
)(')( )2(kkkk netSod −=δ
)(')( )1()1,2(, jk jkkj netSw∑= δμ
Weighted sum of errors from output layer
![Page 17: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/17.jpg)
Note: if S is a logistic function, then S’(x) = S(x)(1 – S(x))
![Page 18: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/18.jpg)
Backpropagation Learning
• Pattern classification: – Two classes: 1 output node
– N classes: binary encoding (log N) output nodes
better using N output nodes, a class is represented as
(0,..., 0,1, 0,.., 0)
– With sigmoid function, nodes at output layer will never be 1 or 0, but either 1 – ε or ε.
– Error reduction becomes slower when moving into saturation regions (when ε is small).
– For fast learning, for a given error bound ε, set error lp,k = 0 if |dp,k – op,k| ≤ ε
– When classifying a input x using a trained BP net, classify it to the kth class if with dk. > dl for all l != k
![Page 19: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/19.jpg)
Backpropagation Learning
• Pattern classification: an example– Classification of myoelectric signals
• Input pattern: 3 features (NIF, VT, RR), normalized to real values between 0 and 1
• Output patters: 2 classes: (success, failure)
– Network structure: 2‐5‐3• 3 input nodes, 2 output nodes, • 1 hidden layer of 5 nodes • η = 0.95, α = 0.4 (momentum)
– Error bound ε = 0.05– 332 training samples– Maximum iteration = 20,000 – When stopped, 38 patterns remain misclassified
![Page 20: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/20.jpg)
Hopfield Memories is of 2 typesDiscrete Hopfield NNsContinuous Hopfield NNs
Associative Memories is of 2 types
Hopfield MemoryBidirection Memory
![Page 21: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/21.jpg)
BPN Architecture
![Page 22: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/22.jpg)
BPN Data Structure
![Page 23: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/23.jpg)
Feedforward/Feedback NNs
Feedforward NNsThe connections between units do not form cycles.Usually produce a response to an input quickly.Most feedforward NNs can be trained using a wide variety of efficient algorithms.
Feedback or recurrent NNsThere are cycles in the connections. In some feedback NNs, each time an input is presented, the NN must iterate for a potentially long time before it produces a response.Usually more difficult to train than feed forward NNs.
![Page 24: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/24.jpg)
Supervised‐Learning NNs
• Feedforward NNs– Perceptron – Adaline, Madaline – Backpropagation (BP)– Artmap – Learning Vector Quantization (LVQ) – Probabilistic Neural Network (PNN) – General Regression Neural Network (GRNN)
• Feedback or recurrent NNs– Brain-State-in-a-Box (BSB) – Fuzzy Congitive Map (FCM) – Boltzmann Machine (BM) – Backpropagation through time (BPTT)
![Page 25: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/25.jpg)
Unsupervised‐Learning NNs
• Feedforward NNs– Learning Matrix (LM) – Sparse Distributed Associative Memory (SDM) – Fuzzy Associative Memory (FAM) – Counterprogation (CPN)
• Feedback or Recurrent NNs– Binary Adaptive Resonance Theory (ART1) – Analog Adaptive Resonance Theory (ART2, ART2a) – Discrete Hopfield (DH) – Continuous Hopfield (CH)– Discrete Bidirectional Associative Memory (BAM) – Kohonen Self-organizing Map/Topology-preserving map
(SOM/TPM)
![Page 26: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/26.jpg)
The Hopfield NNs
• In 1982, Hopfield, a Caltech physicist, mathematically tied together many of the ideas from previous research.
• A fully connected, symmetrically weighted network where each node functions both as input and output node.
• Used for– Associated memories
Combinatorial optimization
![Page 27: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/27.jpg)
Hamming Distance
![Page 28: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/28.jpg)
Hopfield Memory for TSP
![Page 29: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/29.jpg)
Associative Memories
• An associative memory is a content-addressable structure that maps a set of input patterns to a set of output patterns.
• Two types of associative memory: autoassociativeand heteroassociative.
• Auto-association– retrieves a previously stored pattern that most closely
resembles the current pattern. • Hetero-association
– the retrieved pattern is, in general, different from the input pattern not only in content but possibly also in type and format.
![Page 30: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/30.jpg)
Associative Memories
Auto-association
AA
Hetero-association
Niagara Waterfall
memory
memory
![Page 31: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/31.jpg)
Optimization Problems
• Associate costs with energy functions in Hopfield Networks– Need to be in quadratic form
• Hopfield Network finds local, satisfactory soluions, doesn’t choose solutions from a set.
• Local optimums, not global.
![Page 32: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/32.jpg)
The Discrete Hopfield NNs
1 2 3 n. . .
v1 v2 v3 vn
I1 I2 I3 In
w21 w31 wn1
w12 w32 wn2
w13 w23 wn3
w1n w2n w3n
![Page 33: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/33.jpg)
BAM Architecture
![Page 34: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/34.jpg)
BAM Energy landscape
![Page 35: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/35.jpg)
Auto associative bam architecture
![Page 36: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/36.jpg)
Single layer structure of bam
![Page 37: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/37.jpg)
Hopfield memory architecture
![Page 38: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/38.jpg)
Inhibitory connections between peS FOR TSP PROBLEM
![Page 39: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/39.jpg)
BAM DATASTRUCTURE
![Page 40: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/40.jpg)
Backpropagation Learning
• Architecture: – Feedforward network of at least one layer of non‐linear hidden
nodes, e.g., # of layers L ≥ 2 (not counting the input layer)– Node function is differentiable
most common: sigmoid function• Learning: supervised, error driven,
generalized delta rule• Call this type of nets BP nets• The weight update rule
(gradient descent approach)• Practical considerations• Variations of BP nets• Applications
![Page 41: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/41.jpg)
Review questions
• Discuss the role of generalized delta rule in neural networks.
• What is the importance of hamming distance in visualizing the data space.
• Discuss the pros and cons of BPN architecture • State the importance of BAM energy function.• How the Hopfield memory model is useful for optimizaion problems.
![Page 42: UNIT II BPN AND BAM - SRM Institute of Science and … -II.pdf · To make nonlinear partitions on the space we need to define each unit as a nonlinear. function ... – Adaline, Madaline](https://reader031.vdocuments.us/reader031/viewer/2022022602/5b504d9b7f8b9a2f6e8e322e/html5/thumbnails/42.jpg)
References
• J. A. Freeman and D. M. Skapura, Neural Networks‐ Algorithms, Applications and Programming Techniques, Pearson Education( singapore) Pvt. Ltd., 1991.
(Chapters 3 &4)
• cortex.snowcron.com
• pcrochat.online.fr/webus/tutorial/BPN_tutorial.html/neural_networks.htm