1 pattern recognition: statistical and neural lonnie c. ludeman lecture 21 oct 28, 2005 nanjing...
TRANSCRIPT
![Page 1: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/1.jpg)
1
Pattern Recognition:Statistical and Neural
Lonnie C. Ludeman
Lecture 21
Oct 28, 2005
Nanjing University of Science & Technology
![Page 2: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/2.jpg)
2
Lecture 21 Topics
1.Example – Analysis of simple Neural Network
2.Example - Synthesis of special forms of Artificial Neural Networks
3. General concepts of Training an Artificial Neural Network- Supervised and unsupervised,training sets
4. Neural Networks Nomenclature and Notation
5. Derivation and Description of the Backpropagation Algorithm for Feedforward Neural Networks
![Page 3: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/3.jpg)
3
Example: Analyze the following Neural Network
-1
1
-1
110
00
1
![Page 4: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/4.jpg)
4
Solution: Outputs of layer 1 ANEs
![Page 5: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/5.jpg)
5
Output of layer 2 ANE is
Thus from layer 1 we have
- 2 ≥ 0 < 0
![Page 6: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/6.jpg)
6
![Page 7: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/7.jpg)
7
Final Solution: Output Function for Given Neural Network
![Page 8: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/8.jpg)
8
Example: Synthesize a Neural Network
Given the following decision regions build a neural network to perform the classification process
Solution: Use Hyperplane-AND-OR structure
![Page 9: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/9.jpg)
9
Each gk(x) specifies a
hyperplane boundary
![Page 10: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/10.jpg)
10
Hyperplane Layer AND Layer OR Layer
all f(·) = μ(·)
Solution:
![Page 11: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/11.jpg)
11
Training a Neural Network
“With a teacher” “Without a teacher”
![Page 12: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/12.jpg)
12
![Page 13: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/13.jpg)
13
Training Set
xj are the training samples
dj is the class assigned to training sample xj
![Page 14: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/14.jpg)
14
Example of a training set:
( x1 = [ 0, 1 ,2 ]T , d1 = C1 ) ,
( x2 = [ 0, 1 ,0 ]T , d2 = C1 ) ,
( x3 = [ 0, 1 ,1 ]T , d3 = C1 ) ,
( x4 = [ 1, 0 ,2 ]T , d4 = C2 ) ,
( x5 = [ 1, 0 ,3 ]T , d5 = C2 ) ,
( x6 = [ 0, 0 ,1 ]T , d6 = C3 ) ,
( x7 = [ 0, 0 ,2 ]T , d7 = C3 )
( x8 = [ 0, 0 ,3 ]T d8 = C3 )
( x9 = [ 0, 0 ,3 ]T d9 = C3 )
( x10 = [ 1, 1 ,0 ]T d10 = C4 )
( x11 = [ 2, 2 ,0 ]T d11 = C4 )
( x12 = [ 2, 2 ,2 ]T d12 = C5 )
( x13 = [ 3, 2, 2 ]T d13 = C6 )
{
}
![Page 15: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/15.jpg)
15
General Weight Update Algorithm
x(k) is the training sample for the k th iteration
d(k) is the class assigned to training sample x(k) y(k) is the output vector for the k th training sample
![Page 16: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/16.jpg)
16
Training with a Teacher( Supervised)
1. Given a set of N ordered samples with their known class assignments.
2. Randomly select all weights in the neural network.
3. For each successive sample in the total set of samples, evaluate the output.
4. Use these outputs and the input sample to update the weights
5. Stop at some predetermined number of iterations or if given performance measure is satisfied. If not stopped go to step 3
![Page 17: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/17.jpg)
17
Training without a Teacher( Unsupervised)
1. Given a set of N ordered samples with unknown class assignments.
2. Randomly select all weights in the neural network.
3. For each successive sample in the total set of samples, evaluate the outputs.
4. Using these outputs and the inputs update the weights
5. If weights do not change significantly stop with that result. If weights change return to step 3
![Page 18: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/18.jpg)
18
Supervised Training of a Feedforward Neural Network
Nomenclature
![Page 19: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/19.jpg)
19
Output vector of layer m
Output vector of layer L
Node Number Layer m
Node Number Layer L
1
![Page 20: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/20.jpg)
20
Weight Matrix for layer m
Node 1 Node 2 Node Nm
N
Nm
![Page 21: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/21.jpg)
21
fix
Layers, Nets, Outputs, Nonlinearities
![Page 22: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/22.jpg)
22
Define the performance Ep for sample x(p) as
We wish to select weights so that Ep is
Minimized – Use Gradient Algorithm
![Page 23: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/23.jpg)
23
Gradient Algorithm for Updating the weights
p w(p)
px(p)
![Page 24: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/24.jpg)
24
Derivation of weight update equation for Last Layer (Rule #1) Backpropagation Algorihm
The partial of ym(L)
with respect to wkj(L) is
![Page 25: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/25.jpg)
25
General Rule #1 for Weight Update
Therefore
![Page 26: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/26.jpg)
26
Derivation of weight update equation for Next to Last Layer (L-1) Backpropagation Algorithm
![Page 27: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/27.jpg)
27
![Page 28: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/28.jpg)
28
General Rule #2 for Weight Update- Layer L-1 Backpropagation Algorithm
Therefore
and the weight correction is as follows
![Page 29: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/29.jpg)
29
where weight correction (general Rule #2) is
w
(L-1)
![Page 30: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/30.jpg)
30
Backpropagation Training Algorithm for Feedforward Neural networks
![Page 31: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/31.jpg)
31
Input pattern sample xk
![Page 32: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/32.jpg)
32
Calculate Outputs First Layer
![Page 33: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/33.jpg)
33
Calculate Outputs Second Layer
![Page 34: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/34.jpg)
34
Calculate Outputs Last Layer
![Page 35: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/35.jpg)
35
Check Performance
ETOTAL(p) ½ (d[x(p-i)] – f( wT(p-i)x(p-i) )2
i = 0
Ns - 1
ETOTAL(p+1) = ETOTAL(p) + Ep+1 (p+1) – Ep-Ns (p-Ns )
Single Sample Error
Over all Samples Error
Can be computed recursively
![Page 36: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/36.jpg)
36
Change Weights Last Layer using Rule #1
![Page 37: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/37.jpg)
37
Change Weights previous Layer using Rule #2
![Page 38: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/38.jpg)
38
Change Weights previous Layer using Modified Rule #2
![Page 39: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/39.jpg)
39
Input pattern sample xk+1
Continue Iterations Until
![Page 40: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/40.jpg)
40
Repeat process until performance is satisfied or maximum number of iterations are reached.
If performance not satisfied at maximum number of iterations the algorithm
stops and NO design is obtained.
If performance is satisfied then the current weights and structure provide the
required design.
![Page 41: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/41.jpg)
41
Freeze Weights to get Acceptable Neural Net Design
![Page 42: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/42.jpg)
42
Backpropagation Algorithm for Training Feedforward Artificial Neural Networks
![Page 43: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/43.jpg)
43
Summary Lecture 21
1.Example – Analysis of simple Neural Network
2.Example - Synthesis of special forms of Artificial Neural Networks
3. General concepts of Training an Artificial Neural Network- Supervised and unsupervised,and description of training sets
4. Neural Networks Nomenclature and Notation
5. Derivation and Description of the Backpropagation Algorithm for Feedforward Neural Networks
![Page 44: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 21 Oct 28, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.us/reader030/viewer/2022032607/56649ec65503460f94bd16b2/html5/thumbnails/44.jpg)
44
End of Lecture 21