neural networks in electrical engineering prof. howard silver school of computer sciences and...

Neural Networks in Electrical Engineering

Prof. Howard SilverSchool of Computer Sciences and Engineering

Fairleigh Dickinson UniversityTeaneck, New Jersey

Axon fromAnother Neuron

Axon fromAnother Neuron

Synaptic Gap

Synaptic Gap

Soma

Axon

Dendrite

Dendrite ofAnother Neuron

Dendrite ofAnother Neuron

Biological Neuron

Steps in applying Perceptron:

! Initialize the weights and bias to 0.! Set the learning rate alpha (0 < α <= 1) and threshold θ.! For each input pattern,!__Compute for each output ! yin = b + x1 * w1 + x2 * w2 + x3 * w3 + ...... ! and set !__y = ‑1 for yin < ‑θ !__y = 0 for -θ<= yin <= θ!__y = 1 for yin > θ!__If the jth output yj is not equal to tj, set!____wij(new) = wij(old) + α * xi * tj

!____bj(new) = bj(old) + α * tj

!__(else no change in wij and bj)

Example of Supervised Learning Algorithm - Perceptron

Perceptron Applied to Character Recognition

Character Inputs ("A") (Binary)..#.. 00100.#.#. 01010#...# 10001##### 11111#...# 10001

'0010001010100011111110001'

'10000000000000000000000000'

Neural net inputs x1 to x25

Binary target output string (t1 to t26)

EDU» a_zlearnCharacter training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####

Enter number of training epochs (m) 5Enter learning rate (alpha) 1Enter threshold value (theta) 0.1

Character training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####

Final outputs after training:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000 A01000000000000000000100000 B00000000000000000000000000 C00010000000000000000000000 D00001000000000000000000000 E00100100000000000000000000 F000000?0000000100000000000 G00000001000000000000000000 H00000000100000000000000000 I00000000010000000000000000 J00000000001000000000000000 K00000010000100000000000000 L00000000000010000000000000 M00000000000001000000000000 N00?00000000000000000000000 O00000000000000010000000000 P00000000000000001000000000 Q00000000000000000100000000 R00000000000000000010000000 S00000000000000000001000000 T00000000000000000000000000 U00000000000000000000010000 V00000000000000100000001000 W00000000000000000000000100 X00000000000000000000000010 Y00000000000000?00000000001 Z

Enter a test pattern (1s and 0s in quotes)'1010001010100011111110001'Character entered:#.#...#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000Sorted outputs before activation (strongest first):AWRODVYUSLKIHJGFECZXTMNPQB

Enter a test pattern (1s and 0s in quotes)'1010101010100011111110001'Character entered:#.#.#.#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ?0000000000000000000000000Sorted outputs before activation (strongest first):AWRVSODYUKGIHXJECMLFZTNQBP

Enter a test pattern (1s and 0s in quotes)'0010001010111111000110001'Character entered:..#...#.#.######...##...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ00000?0100101?010010000010Sorted outputs before activation (strongest first):YHSPMKNFEWUARODVLJZXBGTQIC

SN = 5

Signal Classification with Perceptron

SN = 5

SN=5

NNET11H.M

EDU» nnet11hEnter number of training epochs (m) 20Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100011000:EDU» nnet11hEnter number of training epochs (m) 30Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:101010000:

EDU» nnet11hEnter number of training epochs (m) 40Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100010001Enter signal to noise ratio 100Classification of signals embedded in noise100010001:Enter signal to noise ratio 10Classification of signals embedded in noise100010001:

Enter signal to noise ratio 5Classification of signals embedded in noise100010000:Enter signal to noise ratio 2Classification of signals embedded in noise100010000:Enter signal to noise ratio 1Classification of signals embedded in noise100010010:Enter signal to noise ratio 1Classification of signals embedded in noise100011001

>> nnet11iEnter frequency separation in pct. (del) 100Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001

Enter signal to noise ratio (SN) - zero to exit 1 Classification of signals embedded in noise 100010001

Classification of Three Sinusoids of Different Frequency

Signals Noisy Signals

>> nnet11iEnter frequency separation in pct. (del) 10Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001

Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 100010001

Signals Noisy Signals

Signals Noisy Signals

Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 0?0010001

>> nnet11iEnter frequency separation in pct. (del) 5Number of samples per cycle (xtot) 64Enter number of training epochs (m) 500 Final outputs after training: 100010001

Signals Noisy Signals

>> nnet11iEnter frequency separation in pct. (del) 1Number of samples per cycle (xtot) 1000Enter number of training epochs (m) 10000 Final outputs after training: 100010001

Enter signal to noise ratio (SN) - zero to exit 100 Classification of signals embedded in noise 1000?0001

K o ho n e n L e arn in g a nd Se lf O rg a n iz in g M a psL ine a r a rra y : # (R=0) * # * (R=1) * * # * * (R=2)* * * # * * * (R=3)R e c ta n g u lar g rid : # (R=0)

* * * * # * (R=1) * * *

* * * * * * * * * * * * # * * (R=2) * * * * * * * * * *

Unsupervised Learning

Initialize the weights (e.g. random values).

Set the neighborhood radius (R) and a learning rate (α).

Repeat the steps below until convergence or a maximum number of epochs is reached.

For each input pattern X = [x1 x2 x3 ......]

Compute a "distance"

D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2 + (w3j ‑ x3)2 + ...... for each cluster (i.e. all j), and find jmin, the value of j corresponding to the minimum D(j).

If j is "in the neighborhood of" jmin, wij(new) = wij(old) + α [xi ‑ wij(old)] for all i.

Decrease α (linearly or geometrically) and reduce R (at a specified rate) if R > 0.

Kohonen Learning Steps

S e lf O rg a n iz ing M ap s fo r A lp ha b e tic C ha rac te r S e t

E xa m p le 4.5 in Fa u se tt

C ha ra c te r tra in in g se t :..##... ######. ..####. #####.. ####### ...#### ###..## ...#... .#....# .#....# .#...#. .#....# .....#. .#..#.. ...#... .#....# #...... .#....# .#..... .....#. .#.#... ..#.#.. .#....# #...... .#....# .#.#... .....#. .##.... ..#.#.. .#####. #...... .#....# .###... .....#. .##.... .#####. .#....# #...... .#....# .#.#... .....#. .#.#... .#...#. .#....# #...... .#....# .#..... .#...#. .#..#.. .#...#. .#....# .#....# .#...#. .#....# .#...#. .#...#. ###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1

...#... ######. ..###.. #####.. ####### .....#. #....#.

...#... #.....# .#...#. #....#. #...... .....#. #...#..

...#... #.....# #.....# #.....# #...... .....#. #..#...

..#.#.. #.....# #...... #.....# #...... .....#. #.#....

..#.#.. ######. #...... #.....# #####.. .....#. ##.....

.#...#. #.....# #...... #.....# #...... .....#. #.#....

.#####. #.....# #.....# #.....# #...... .#...#. #..#...

.#...#. #.....# .#...#. #....#. #...... .#...#. #...#..

.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2

...#... ######. ..###.# #####.. ####### ....### ###..##

...#... .#....# .#...## .#...#. .#....# .....#. .#...#.

..#.#.. .#....# #.....# .#....# .#..#.. .....#. .#..#..

..#.#.. .#####. #...... .#....# .####.. .....#. .#.#...

.#...#. .#....# #...... .#....# .#..#.. .....#. .##....

.#####. .#....# #...... .#....# .#..... .....#. .#.#... #.....# .#....# #.....# .#....# .#..... .....#. .#..#.. #.....# .#....# .#...#. .#...#. .#....# .#...#. .#...#. ##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3

###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1

.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2

##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3

NNET19.M

T he u n its a ssoc ia te d w ith e a c h p a tte rn

(a s sho w n in Fa u se tt fo r E x a m p les 4 .5 a n d 4 .6 ).(F o r in it ia l R = 0 )Unit Patterns2 B1, B3, D1, D3, E1, E3, K1, K311 A1, A2, A314 C1, C2, C3, J1, J2, J325 B2, D2, E2, K2(F o r in it ia l R = 1 )Unit Patterns2 C14 C2, C36 J1, J2, J38 D1, D39 B1, B310 E111 E312 K1, K314 K216 D217 B2, E219 A120 A221 A3

N N E T 1 9 A .M

S a m e c h a rac te r p a tte rn s are o rga n iz e d into a tw o -d im e ns io na l re c ta ng u la r a rra y o f c lus t e r u n its .

A sa m p le o utp u t o f t h is pro g ra m a fte r 1 0 0 e p oc h sR o w : 3 3 4 1 5 1 5 5 5 1 5 1 1 5 1 3 3 3 1 4 1C o lum n : 2 2 1 4 3 4 1 5 5 5 4 5 3 2 3 4 4 5 2 2 2

Column 1 2 3 4 5 1 K1,K3 E1,E3 B1,B3 D1,D3 2Row 3 A1,A2 J1,J2 J3 4 A3 K2 5 C1 E2 B2 D2 C2,C3

Two input neurons ‑ one each for the x and y coordinate numbers for the cities

The Traveling Salesman Problem

Distance function:

D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2

Example - 4 cities on the vertices of a square

! C ity A : -1 , -1! C ity B : -1 , 1! C ity C : 1 , -1! C ity D : 1 , 1

N N E T 1 9C .M

The coordinates of the cities:

Weight matrices from three different runs:

W = [ 1 1 ‑1 ‑1 1 ‑1 ‑1 1]

W = [ 1 ‑1 ‑1 1 1 1 ‑1 ‑1]

W = [‑1 1 1 ‑1 ‑1 ‑1 1 1]

100 “Randomly” Located Cities

0.1 < α < 0.25, 100 epochs, Initial R = 2