neural networks in electrical engineering prof. howard silver school of computer sciences and...
Post on 21-Dec-2015
216 views
TRANSCRIPT
![Page 1: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/1.jpg)
Neural Networks in Electrical Engineering
Prof. Howard SilverSchool of Computer Sciences and Engineering
Fairleigh Dickinson UniversityTeaneck, New Jersey
![Page 2: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/2.jpg)
Axon fromAnother Neuron
Axon fromAnother Neuron
Synaptic Gap
Synaptic Gap
Soma
Axon
Dendrite
Dendrite ofAnother Neuron
Dendrite ofAnother Neuron
Biological Neuron
![Page 3: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/3.jpg)
![Page 4: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/4.jpg)
![Page 5: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/5.jpg)
![Page 6: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/6.jpg)
Steps in applying Perceptron:
! Initialize the weights and bias to 0.! Set the learning rate alpha (0 < α <= 1) and threshold θ.! For each input pattern,!__Compute for each output ! yin = b + x1 * w1 + x2 * w2 + x3 * w3 + ...... ! and set !__y = ‑1 for yin < ‑θ !__y = 0 for -θ<= yin <= θ!__y = 1 for yin > θ!__If the jth output yj is not equal to tj, set!____wij(new) = wij(old) + α * xi * tj
!____bj(new) = bj(old) + α * tj
!__(else no change in wij and bj)
Example of Supervised Learning Algorithm - Perceptron
![Page 7: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/7.jpg)
Perceptron Applied to Character Recognition
Character Inputs ("A") (Binary)..#.. 00100.#.#. 01010#...# 10001##### 11111#...# 10001
'0010001010100011111110001'
'10000000000000000000000000'
Neural net inputs x1 to x25
Binary target output string (t1 to t26)
![Page 8: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/8.jpg)
EDU» a_zlearnCharacter training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####
Enter number of training epochs (m) 5Enter learning rate (alpha) 1Enter threshold value (theta) 0.1
Character training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####
![Page 9: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/9.jpg)
Final outputs after training:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000 A01000000000000000000100000 B00000000000000000000000000 C00010000000000000000000000 D00001000000000000000000000 E00100100000000000000000000 F000000?0000000100000000000 G00000001000000000000000000 H00000000100000000000000000 I00000000010000000000000000 J00000000001000000000000000 K00000010000100000000000000 L00000000000010000000000000 M00000000000001000000000000 N00?00000000000000000000000 O00000000000000010000000000 P00000000000000001000000000 Q00000000000000000100000000 R00000000000000000010000000 S00000000000000000001000000 T00000000000000000000000000 U00000000000000000000010000 V00000000000000100000001000 W00000000000000000000000100 X00000000000000000000000010 Y00000000000000?00000000001 Z
![Page 10: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/10.jpg)
Enter a test pattern (1s and 0s in quotes)'1010001010100011111110001'Character entered:#.#...#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000Sorted outputs before activation (strongest first):AWRODVYUSLKIHJGFECZXTMNPQB
![Page 11: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/11.jpg)
Enter a test pattern (1s and 0s in quotes)'1010101010100011111110001'Character entered:#.#.#.#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ?0000000000000000000000000Sorted outputs before activation (strongest first):AWRVSODYUKGIHXJECMLFZTNQBP
![Page 12: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/12.jpg)
Enter a test pattern (1s and 0s in quotes)'0010001010111111000110001'Character entered:..#...#.#.######...##...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ00000?0100101?010010000010Sorted outputs before activation (strongest first):YHSPMKNFEWUARODVLJZXBGTQIC
![Page 13: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/13.jpg)
SN = 5
Signal Classification with Perceptron
![Page 14: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/14.jpg)
SN = 5
![Page 15: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/15.jpg)
SN=5
![Page 16: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/16.jpg)
NNET11H.M
EDU» nnet11hEnter number of training epochs (m) 20Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100011000:EDU» nnet11hEnter number of training epochs (m) 30Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:101010000:
![Page 17: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/17.jpg)
EDU» nnet11hEnter number of training epochs (m) 40Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100010001Enter signal to noise ratio 100Classification of signals embedded in noise100010001:Enter signal to noise ratio 10Classification of signals embedded in noise100010001:
![Page 18: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/18.jpg)
Enter signal to noise ratio 5Classification of signals embedded in noise100010000:Enter signal to noise ratio 2Classification of signals embedded in noise100010000:Enter signal to noise ratio 1Classification of signals embedded in noise100010010:Enter signal to noise ratio 1Classification of signals embedded in noise100011001
![Page 19: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/19.jpg)
>> nnet11iEnter frequency separation in pct. (del) 100Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 1 Classification of signals embedded in noise 100010001
Classification of Three Sinusoids of Different Frequency
Signals Noisy Signals
![Page 20: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/20.jpg)
>> nnet11iEnter frequency separation in pct. (del) 10Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 100010001
Signals Noisy Signals
![Page 21: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/21.jpg)
Signals Noisy Signals
Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 0?0010001
>> nnet11iEnter frequency separation in pct. (del) 5Number of samples per cycle (xtot) 64Enter number of training epochs (m) 500 Final outputs after training: 100010001
![Page 22: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/22.jpg)
Signals Noisy Signals
>> nnet11iEnter frequency separation in pct. (del) 1Number of samples per cycle (xtot) 1000Enter number of training epochs (m) 10000 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 100 Classification of signals embedded in noise 1000?0001
![Page 23: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/23.jpg)
K o ho n e n L e arn in g a nd Se lf O rg a n iz in g M a psL ine a r a rra y : # (R=0) * # * (R=1) * * # * * (R=2)* * * # * * * (R=3)R e c ta n g u lar g rid : # (R=0)
* * * * # * (R=1) * * *
* * * * * * * * * * * * # * * (R=2) * * * * * * * * * *
Unsupervised Learning
![Page 24: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/24.jpg)
Initialize the weights (e.g. random values).
Set the neighborhood radius (R) and a learning rate (α).
Repeat the steps below until convergence or a maximum number of epochs is reached.
For each input pattern X = [x1 x2 x3 ......]
Compute a "distance"
D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2 + (w3j ‑ x3)2 + ...... for each cluster (i.e. all j), and find jmin, the value of j corresponding to the minimum D(j).
If j is "in the neighborhood of" jmin, wij(new) = wij(old) + α [xi ‑ wij(old)] for all i.
Decrease α (linearly or geometrically) and reduce R (at a specified rate) if R > 0.
Kohonen Learning Steps
![Page 25: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/25.jpg)
S e lf O rg a n iz ing M ap s fo r A lp ha b e tic C ha rac te r S e t
E xa m p le 4.5 in Fa u se tt
C ha ra c te r tra in in g se t :..##... ######. ..####. #####.. ####### ...#### ###..## ...#... .#....# .#....# .#...#. .#....# .....#. .#..#.. ...#... .#....# #...... .#....# .#..... .....#. .#.#... ..#.#.. .#....# #...... .#....# .#.#... .....#. .##.... ..#.#.. .#####. #...... .#....# .###... .....#. .##.... .#####. .#....# #...... .#....# .#.#... .....#. .#.#... .#...#. .#....# #...... .#....# .#..... .#...#. .#..#.. .#...#. .#....# .#....# .#...#. .#....# .#...#. .#...#. ###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1
...#... ######. ..###.. #####.. ####### .....#. #....#.
...#... #.....# .#...#. #....#. #...... .....#. #...#..
...#... #.....# #.....# #.....# #...... .....#. #..#...
..#.#.. #.....# #...... #.....# #...... .....#. #.#....
..#.#.. ######. #...... #.....# #####.. .....#. ##.....
.#...#. #.....# #...... #.....# #...... .....#. #.#....
.#####. #.....# #.....# #.....# #...... .#...#. #..#...
.#...#. #.....# .#...#. #....#. #...... .#...#. #...#..
.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2
...#... ######. ..###.# #####.. ####### ....### ###..##
...#... .#....# .#...## .#...#. .#....# .....#. .#...#.
..#.#.. .#....# #.....# .#....# .#..#.. .....#. .#..#..
..#.#.. .#####. #...... .#....# .####.. .....#. .#.#...
.#...#. .#....# #...... .#....# .#..#.. .....#. .##....
.#####. .#....# #...... .#....# .#..... .....#. .#.#... #.....# .#....# #.....# .#....# .#..... .....#. .#..#.. #.....# .#....# .#...#. .#...#. .#....# .#...#. .#...#. ##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3
###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1
.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2
##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3
NNET19.M
![Page 26: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/26.jpg)
T he u n its a ssoc ia te d w ith e a c h p a tte rn
(a s sho w n in Fa u se tt fo r E x a m p les 4 .5 a n d 4 .6 ).(F o r in it ia l R = 0 )Unit Patterns2 B1, B3, D1, D3, E1, E3, K1, K311 A1, A2, A314 C1, C2, C3, J1, J2, J325 B2, D2, E2, K2(F o r in it ia l R = 1 )Unit Patterns2 C14 C2, C36 J1, J2, J38 D1, D39 B1, B310 E111 E312 K1, K314 K216 D217 B2, E219 A120 A221 A3
![Page 27: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/27.jpg)
N N E T 1 9 A .M
S a m e c h a rac te r p a tte rn s are o rga n iz e d into a tw o -d im e ns io na l re c ta ng u la r a rra y o f c lus t e r u n its .
A sa m p le o utp u t o f t h is pro g ra m a fte r 1 0 0 e p oc h sR o w : 3 3 4 1 5 1 5 5 5 1 5 1 1 5 1 3 3 3 1 4 1C o lum n : 2 2 1 4 3 4 1 5 5 5 4 5 3 2 3 4 4 5 2 2 2
Column 1 2 3 4 5 1 K1,K3 E1,E3 B1,B3 D1,D3 2Row 3 A1,A2 J1,J2 J3 4 A3 K2 5 C1 E2 B2 D2 C2,C3
![Page 28: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/28.jpg)
Two input neurons ‑ one each for the x and y coordinate numbers for the cities
The Traveling Salesman Problem
Distance function:
D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2
Example - 4 cities on the vertices of a square
![Page 29: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/29.jpg)
! C ity A : -1 , -1! C ity B : -1 , 1! C ity C : 1 , -1! C ity D : 1 , 1
N N E T 1 9C .M
The coordinates of the cities:
Weight matrices from three different runs:
W = [ 1 1 ‑1 ‑1 1 ‑1 ‑1 1]
W = [ 1 ‑1 ‑1 1 1 1 ‑1 ‑1]
W = [‑1 1 1 ‑1 ‑1 ‑1 1 1]
![Page 30: Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and Engineering Fairleigh Dickinson University Teaneck, New Jersey](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d5d5503460f94a3c520/html5/thumbnails/30.jpg)
100 “Randomly” Located Cities
0.1 < α < 0.25, 100 epochs, Initial R = 2