11

OPTIMIZED INTERCONNECTIONS IN OPTIMIZED INTERCONNECTIONS IN PROBABILISTIC SELF-ORGANIZING LEARNINGPROBABILISTIC SELF-ORGANIZING LEARNING

Janusz Starzyk, Mingwei Ding, Haibo He Janusz Starzyk, Mingwei Ding, Haibo He

School of EECSSchool of EECSOhio University, Athens, OHOhio University, Athens, OH

February 14-16, 2005February 14-16, 2005Innsbruck, AustriaInnsbruck, Austria

22

OUTLINEOUTLINE

• IntroductionIntroduction• Self-organizing neural network structure Self-organizing neural network structure • Optimal and fixed input weightsOptimal and fixed input weights

Optimal weights Optimal weights Binary weightsBinary weights

• Simulation resultsSimulation results Financial data analysisFinancial data analysis Power quality classification Power quality classification

• Hardware platform developmentHardware platform development• ConclusionConclusion

33

Self-Organizing Learning Self-Organizing Learning Array (SOLAR )Array (SOLAR )

SOLAR CHARACTERISTICSSOLAR CHARACTERISTICS

Entropy based self-Entropy based self-organizationorganization

Dynamical Dynamical reconfigurationreconfiguration

Local and sparse Local and sparse iinterconnectionsnterconnections

Online inputs selectionOnline inputs selectionFeature neurons and Feature neurons and

merging neuronsmerging neurons

44

SOLAR Hardware StructureSOLAR Hardware Structure

55

Neuron StructureNeuron Structure

66

Self-organizationSelf-organization

Each neuron has the ability to self-Each neuron has the ability to self-organize according to received organize according to received informationinformation• Functionality – chose internal arithmetic Functionality – chose internal arithmetic

and logic functionsand logic functions• Input selection – chose input Input selection – chose input

connectionsconnections

77

Input selectionInput selection

Merging neurons Merging neurons receive inputs from receive inputs from previous layersprevious layers

Variable probability of Variable probability of received information received information correctnesscorrectness

Two input selection Two input selection strategiesstrategies• Random selectionRandom selection• Greedy selectionGreedy selection

88

Input Weighting Input Weighting Weighted signal merging:Weighted signal merging:

Signal energy

Noise energy

N

Sn , n0

S2 , n0

S1, n0 W1

W2

Wn

22211

2 )...(ˆ nn swswsws

)...(ˆ 222

21

20

2nwwwnn

99

Input Weighting (cont’d)Input Weighting (cont’d) Objective functionObjective function

• Maximize the energy/noise ratioMaximize the energy/noise ratio

Set gradient of the objective function to 0

*

)()ˆ/ˆ(max 22i

wwFns

i

0iW

F for i=1, 2, … n

constAs

w

i

i

1010

Optimum Input weightingOptimum Input weighting

2-class classification problem2-class classification problem• Each neuron receives recognition rate Each neuron receives recognition rate pp from from

the previous layerthe previous layer

]1,0[p When p=0.5, least information, it can be either classWhen p=0 or 1, most information, knows which class

data belong to for sure

Define

Signal/noise ratio

5.0pp

)(2

11

5.0ˆ

psignp

p

p

noise

pp

1111

Optimum Input weighting (cont’d)Optimum Input weighting (cont’d)

Using the optimization resultUsing the optimization result

Optimum weight:

Weighted output:

Solve and

represents our belief that result belong to class 1

n

kk

ii

p

pw

1

ˆ

ˆ

n

ii

n

iii

n

ii

n

iii

out

p

pp

w

pw

n

sp

1

2

1

1

2

1

ˆ

ˆˆˆ

ˆ

)ˆ(ˆ1

2/1

outout

out

psignp

p

5.0 outout pp

outp

constAs

w

i

i

1212

Optimum Input weighting (cont’d)Optimum Input weighting (cont’d)

Example:

Consider 3 inputs to a neuron with correct classification probabilities equal to pi

Estimated output probability pout for various input probabilities is as follows

1313

Binary weightingBinary weighting Simplified selection algorithm is desired for hardware Simplified selection algorithm is desired for hardware

implementationimplementation Choose 0 or 1 as the weights for all the connected inputsChoose 0 or 1 as the weights for all the connected inputs

This equation can be used to study the effect of adding This equation can be used to study the effect of adding or removing connections of different signal strengthor removing connections of different signal strength

n

P

n

Pn

n

S ij

ij

ij Cii

Ci

Cii

2

2

2

2

2

ˆ

1

ˆ1

ˆ

ˆ

n

P

n

S i

ˆ

ˆ

ˆ

1414

Binary weighting (cont’d)Binary weighting (cont’d) A stronger connection PA stronger connection Pmaxmax

a weaker connection Pa weaker connection Pmixmix Criteria for adding weaker Criteria for adding weaker

connectionconnection Pmix

Pmax 0.5

0.5

Pcomb

Gain of information for different Pmax and Pmix Threshold for adding a new connection

1515

Binary weighting (cont’d)Binary weighting (cont’d)

From previous results, selection criteria for From previous results, selection criteria for binary weighting can be established.binary weighting can be established.

Pmax=0.69 0.5

0.5Pcomb

Pmix>0.60

Pmix<0.60

0.5

Threshold for adding a weaker connection

Pmax=0.69

1616

Simulation resultsSimulation results

Case I: Prediction of financial performanceCase I: Prediction of financial performance• Based on S&P Research Insight DatabaseBased on S&P Research Insight Database• More than 10,000 companies includedMore than 10,000 companies included• Training and testing on 3-year periodTraining and testing on 3-year period• 192 features extracted192 features extracted• Kernel PCA used to reduce 192 features to 13~15Kernel PCA used to reduce 192 features to 13~15

Fig. from http://goldennumber.net/stocks.htm

1717

Simulation results (cont’d)Simulation results (cont’d)

Test year

2001 2002 2003

Performance 0.5846 0.5962 0.5577

Training and testing data structure:

Test result:

1818

Simulation results (cont’d)Simulation results (cont’d) Case II: Power quality disturbance classification problemCase II: Power quality disturbance classification problem

People spill out onto Madison Avenue in New People spill out onto Madison Avenue in New York after blackout hit. York after blackout hit.

(4:00 pm, 14, August, 2003, CNN Report)(4:00 pm, 14, August, 2003, CNN Report)

Cars stopped about three-quarters of the way up Cars stopped about three-quarters of the way up the first hill of the Magnum XL200 ride at Cedar the first hill of the Magnum XL200 ride at Cedar

Point Amusement Park in Sandusky, Ohio. Point Amusement Park in Sandusky, Ohio. (15, August, 2003, CNN Report)(15, August, 2003, CNN Report)

  

  

THE COST:THE COST:According to the North According to the North

American Electric Reliability American Electric Reliability Council (NERC)Council (NERC)

1919

Formulation of the problem:Formulation of the problem:• Wavelet Multiresolution Analysis (MRA) is used Wavelet Multiresolution Analysis (MRA) is used

for feature vector constructionfor feature vector construction• 7 classes classification problem:7 classes classification problem:

Undisturbed sinusoid (normal); swell; sag; Undisturbed sinusoid (normal); swell; sag; harmonics; harmonics;

outage; sag with harmonic; swell with harmonicoutage; sag with harmonic; swell with harmonic

• Two hundred cases of each class were Two hundred cases of each class were generated for training and another 200 cases generated for training and another 200 cases were generated for testing. were generated for testing.

Simulation results (cont’d)Simulation results (cont’d)

2020

Simulation results (cont’d)Simulation results (cont’d)

Reference [16]: T. K. A. Galil et. al, “Power Quality Disturbance Classification Using the Inductive Inference Approach, ” IEEE Transactions On Power Delivery, Vol.19, No.4, October, 2004

2121

XILINX

XILINX

VIRTEX XCV 1000

VIRTEX XCV 1000

Hardware DevelopmentHardware Development

2222

Conclusion Conclusion

Input selection strategyInput selection strategy Optimum weighting scheme theoryOptimum weighting scheme theory Simple binary weighting for Simple binary weighting for

practical usepractical use Searching criteria for useful Searching criteria for useful

connectionsconnections Application studyApplication study Hardware platform design Hardware platform design

2323

Questions?