381 adaptive resonance theory unsupervised learning

38 1

Adaptive Resonance Theory

Unsupervised Learning

22

Learning Objectives

1.1. IntroductionIntroduction

2.2. ART architectureART architecture

3.3. ART implementationART implementation

4.4. ART2ART2

5.5. ConclusionConclusion

33

Introduction

1. Developed by Carpenter and Grossberg

2. ART1-accept binary data

3. ART2-accept continuous value data

4. ART3-improved ART

5. Fuzzy ART- accept fuzzy data

44

ART Architecture (1/20)1.1. Brief description:Brief description:

a.a. Accept an input vector and Accept an input vector and classify it into one of categories.classify it into one of categories.

b.b. The category it belongs to The category it belongs to depend on which one it most depend on which one it most resembles.resembles.

c.c. If the input vector does not If the input vector does not match any of the stored match any of the stored pattern, a new category is created.pattern, a new category is created.

55

ART Architecture (2/20)d.d. If the input vector matched (within If the input vector matched (within

vigilance level) any of stored vigilance level) any of stored pattern, the pattern is adjusted pattern, the pattern is adjusted

(trained) to make it more (trained) to make it more like the like the input vector.input vector.

2.2. Simplified ART ArchitectureSimplified ART Architecturea.a. two layers: two layers: comparison(F1), comparison(F1),

recognition(F2)recognition(F2)b.b. 3 control function: 3 control function: Gain 1, Gain 2Gain 1, Gain 2, ,

and and ResetReset. .

66

ART Architecture (3/20)

77


3.3. Functions of each models:Functions of each models:

a.a. Comparison layer:Comparison layer:

-accept binary input -accept binary input XX

-initially pass -initially pass XX to to CC . So . So CC = = XX

-binary vector -binary vector RR is produced from is produced from recognition layer to modify recognition layer to modify

CC

88


99

ART Architecture (6/20)-each neuron in the comparison layer -each neuron in the comparison layer

has 3 has 3 inputs:inputs:XX: input vector: input vector

PPjj: : weighted sum of recognition layer weighted sum of recognition layer outputoutput

Gain 1Gain 1: : same signal to all neuronssame signal to all neurons-use “two-third” rule-use “two-third” rule

=> at least => at least two of a neuron’s three two of a neuron’s three inputs must be one, otherwise, inputs must be one, otherwise,

the the output is zero.output is zero.-initially, -initially, Gain 1Gain 1 is set to one and is set to one and RR are are

set set to 0.to 0.

1010


b.b. Recognition layer:Recognition layer:

-compute -compute dot productdot product of of BB and and CC

--tthe neuron with largest output winshe neuron with largest output wins

-the winning neuron is set to one -the winning neuron is set to one others are set to zero.others are set to zero.

c.c. Gain 2Gain 2

-OR -OR of components ofof components of XX

1111


1212


d.d. Gain 1Gain 1

OR of OR of XX

components G2 OR of components G2 OR of RR G1G1

--------------------------------------------------------------------------------------------------------

00 0 0 00 0 0

11 1 1 00 1 1

11 1 1 1 1 0 0

00 0 0 1 1 0 0

1313


e.e. ResetReset

-measures the similarity between -measures the similarity between XX and and CC

1414


4.4. OperationsOperationsa. recognition phasea. recognition phase

i) when no input vector i) when no input vector XX , then , then G2 = 0G2 = 0

ii) G2 = 0 disables all neurons in ii) G2 = 0 disables all neurons in recognition layerrecognition layer=> => RR=0=0

iii) This makes sure all neurons in iii) This makes sure all neurons in this layer start out at same state.this layer start out at same state.

1515

ART Architecture (13/20) iv) Then, vector iv) Then, vector XX is applied, this is applied, this XX

must have at least one must have at least one component is “1”. (component is “1”. (∵∵OR of OR of

XX is is “1” => G2=1; “1” => G2=1; ∵∵OR of OR of RR is is “0” “0” => G1=1). => G1=1).

v) So G2, G1 =1, and v) So G2, G1 =1, and CC = = XX..

vi) dot product vi) dot product BBjj.C.C. find neuron . find neuron jj that that has the largest result, fire has the largest result, fire “one”, “one”, others fire “zero”.others fire “zero”.

vii) This vii) This jjthth neuron has output neuron has output rrjj of of RR equal to one, and all others equal to one, and all others

equal equal to zero.to zero.

1616


1717


b. Comparison phaseb. Comparison phase

i) i) rrjj going through weight going through weight ttjiji (binary) (binary)

to each neuron in comparison to each neuron in comparison layer providing an input signal layer providing an input signal PPii..

ii) Now ii) Now RR is not zero anymore, G1 = is not zero anymore, G1 = 0 and by 2/3 rule, the 0 and by 2/3 rule, the

components of components of CC will be one will be one only when only when XX and and P P are ones. are ones.

1818


iii) If there is a substantial iii) If there is a substantial mismatch between mismatch between XX

and and PP, , CC contains many contains many zeros, zeros, while while XX contains ones. This contains ones. This will will trigger Reset function to trigger Reset function to inhabit the output of firing inhabit the output of firing

neuron in the recognition layer to neuron in the recognition layer to zero and disable it for the zero and disable it for the

duration of the current duration of the current classification. classification.

1919


2020


c. Search phasec. Search phase i) if no Reset signal generated, the i) if no Reset signal generated, the

match is in tolerate level, and match is in tolerate level, and the classification is finished. the classification is finished. Otherwise, search other nodes.Otherwise, search other nodes.

ii) to search other node, let ii) to search other node, let rrjj = 0 = 0 => => RR = 0 => G1 =1 => = 0 => G1 =1 => CC = = XX

again.again.

iii) different neuron wins => different iii) different neuron wins => different pattern pattern PP is fed back to is fed back to

comparison layer.comparison layer.

2121


1)1) AA stored pattern is found then the stored pattern is found then the network enters training cycle.network enters training cycle.

2)2) All stored pattern are tried and no All stored pattern are tried and no match , assign new neuron set match , assign new neuron set BBjj

and and TTjj..

2222


d.d. Performance issues:Performance issues:

i)i) sequential searchsequential search

ii)ii) stabilizationstabilization

2323

ART Implementation(1/3)1.1. InitializationInitialization

TTjj, , BBjj, Vigilance level , Vigilance level BBjj::

0< 0< bbijij < < L/(L-1+m)L/(L-1+m) for all for all ii, , jj

mm: # of components in the input vector: # of components in the input vectorLL: as constant > 1 (: as constant > 1 (LL=2, typically)=2, typically)

all all bbij ij are the same valueare the same value

TTjj: : ttijij=1 for all =1 for all ii, , jj

ρρ: 0<: 0<ρρ<1 coarse distinction at first, fine <1 coarse distinction at first, fine distinction at last.distinction at last.

2424

ART Implementation(2/3)2.2. RecognitionRecognition

NETNETj j = (= (BBjjCC))

OUTOUTjj=1=1 NETNETjj>>TT

00 otherwiseotherwise

3.3. ComparisonComparison

Xinof

CinofS

1 #

1 #

Xinof

CinofS

1 #

1 #

4.4. SearchSearchSearching process is going until a pattern is Searching process is going until a pattern is matched or no pattern is matched.matched or no pattern is matched.

2525

ART Implementation(3/3)5.5. TrainingTraining

If the input If the input XX is matched, is matched,

2

neuron firing :

ofcomponent the:

)1/()(

L

j

Cic

cLcLb

thi

kkiij

2

neuron firing :

ofcomponent the:

)1/()(

L

j

Cic

cLcLb

thi

kkiij

ict iji allfor ict iji allfor

for newly stored for newly stored TTjj : :

2626

ART Algorithm(1/4)1.1. InitializationInitialization

L > 1, 0<ρ<1L > 1, 0<ρ<1

0< 0< bbijij < < L/(L-1+m)L/(L-1+m) for all for all ii, , jj

ttijij=1=1 for all for all ii, , jj2.2. While stopping condition is false, do 3-While stopping condition is false, do 3-

12:12:3.3. For each training pattern(For each training pattern(XX),),

do 4-11:do 4-11:4.4. Set Set R R = 0,= 0, C C = = XX

5.5. For each node in For each node in recognition layer that is recognition layer that is not inhibited (rnot inhibited (r-1):-1):

2727

ART Algorithm(2/4)6.6. If If rrjj -1, r -1, rjj = = ((BBjjCC))

7.7. While Reset is trueWhile Reset is true8.8. Find winning Find winning

neuron neuron j, j, If If rrjj - -1 1 then this then this pattern pattern cannot cannot be be clusteredclustered..

9.9. Let Let PP ==TTJJ,,

compute compute CC

2828

ART Algorithm(3/4)

10.10.# 1

, 1# 1

# 1 , proceed to 11.

# 1

J

of in Cif S r

of in X

of in Cif S

of in X

# 1 , 1

# 1

# 1 , proceed to 11.

# 1

J

of in Cif S r

of in X

of in Cif S

of in X

11.11. Update Update BBJJ, , TTJJ==CC

12.12. Test for stopping Test for stopping conditionsconditions

a.a. no weight changeno weight change

b.b. maximum number of maximum number of epochs epochs

reached.reached.

2929

ART Algorithm(4/4)

**** Stopping conditions:Stopping conditions:

a.a. no weight changeno weight change

b.b. maximum number of epochs maximum number of epochs reached.reached.

3030

ART2 (1/9)1.1. Several sublayers replace F1 layerSeveral sublayers replace F1 layer2.2. Update functions:Update functions:

)()( , ,

, ,

iiijiiiiii

ii

ii

ii

qbfxfvdtupausw

Pe

pq

We

wx

Ve

vu

)()( , ,

, ,

iiijiiiiii

ii

ii

ii

qbfxfvdtupausw

Pe

pq

We

wx

Ve

vu

3.3. Parameters:Parameters:mm:: dimension of input vectordimension of input vectornn:: number of cluster units number of cluster units

3131

ART2 (2/9)

3232

ART2 (3/9)3.3. Parameters(continue):Parameters(continue):

a, ba, b:: fixed weight in F1, cannot be zero.fixed weight in F1, cannot be zero.cc:: fixed weight used in testing for fixed weight used in testing for

Reset, 0<c<1.Reset, 0<c<1.dd:: output activation of F2,output activation of F2,e:e: a small parameter to prevent a small parameter to prevent

division by zero when norm of a division by zero when norm of a vector is zero.vector is zero.

:: noise suppression parameternoise suppression parameter:: learning ratelearning rate:: vigilance levelvigilance level

11

d

cd1

1

d

cd

3333

ART2 (4/9)3.3. Algorithm:Algorithm:

aa.. Initialization: Initialization:

a, b, c, d, e, a, b, c, d, e, , , , , b.b. perform # of epochs of trainingperform # of epochs of training

c.c. For each input pattern, For each input pattern, SS,,

do d. to m. do d. to m.

d.d. update F1update F1

)( ,0 ,0

, , ,0

iiii

iii

ii

xfvpq

swSe

sxu

)( ,0 ,0

, , ,0

iiii

iii

ii

xfvpq

swSe

sxu

3434

ART2 (5/9)d.d. update F1 againupdate F1 again

)()( ,

, ,

, ,

iiiii

iiii

i

ii

ii

qbfxfvup

auswPe

pq

We

wx

Ve

vu

)()( ,

, ,

, ,

iiiii

iiii

i

ii

ii

qbfxfvup

auswPe

pq

We

wx

Ve

vu

e.e. compute signal to F2compute signal to F2

j ij ii

y b pj ij ii

y b p

3535

ART2 (6/9)f.f. While Reset is true, do g. to h.While Reset is true, do g. to h.g.g. Find winning neuron Find winning neuron JJ in in F2 F2 h.h. Check for Reset:Check for Reset:

PcUe

cpur

dtupVe

vu

iii

Jiiii

i

, ,

PcUe

cpur

dtupVe

vu

iii

Jiiii

i

, ,

3636

ART2 (7/9)i) i) If ||If ||RR|| < || < - e - e, ,

then then yyJ J = -1 (inhibit = -1 (inhibit

JJ) ) Reset is true; Reset is true; repeat repeat f. f.

ii) If ||ii) If ||RR|| >= || >= - e - e, ,

thenthen

).()(,

, ,

iiii

i

iiiii

qbfxfvPe

pq

We

wxausw

).()(,

, ,

iiii

i

iiiii

qbfxfvPe

pq

We

wxausw

3737

ART2 (8/9)i.i. Do j. to Do j. to ll. for # of learning . for # of learning

iterations.iterations.

j.j. update weight for update weight for neuron neuron JJ

iJiiJ

JiiJi

bdddub

tdddut

)1(1

)1(1

iJiiJ

JiiJi

bdddub

tdddut

)1(1

)1(1

k.k. Update F1Update F1

)()( ,

,,

, ,

iiijiii

iiii

i

ii

ii

qbfxfvdtup

auswPe

pq

We

wx

Ve

vu

)()( ,

,,

, ,

iiijiii

iiii

i

ii

ii

qbfxfvdtup

auswPe

pq

We

wx

Ve

vu

3838

ART2 (9/9)

l.l. Test stopping condition Test stopping condition for weight updates.for weight updates.

m.m. Test stopping condition for # Test stopping condition for # of epochs.of epochs.

381 adaptive resonance theory unsupervised learning

Documents