polynomial discrete time cellular neural networks eduardo gomez-ramirez † giovanni egidio...

Polynomial Discrete Time Cellular Neural Networks

Eduardo Gomez-Ramirez † Giovanni Egidio Pazienza‡

† LIDETEA, POSGRADO E INVESTIGACION Universidad La Salle – México, D.F. ‡ Department d’Electronica, EALS

Universitat “Ramon Llull” – Barcelona, Spain

Outline Cellular Neural Networks (CNN)

Introduction and Objective Genetic Algorithms (GA) Polynomial Discrete Time CNNs

(PDTCNNs) XOR Problem Game of Life

Learning vs Design Conclusions and future workIntro CNN & GA Polyn. CNN XOR GoL Conclusions

CNN: Introduction CNN for complex task (linearly

nonseparable data)

Multilayer CNNs Include more degrees of freedom for the

output state of each layer Search in a finite set of templates

Single layer: Polynomial CNNs

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Improve the representation power of a single layer CNN including a simple nonlinear term to solve problems with linearly nonseparable data (XOR)

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Objective

CNN: mathematical model

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

The simplified mathematical model is:

where xc is the state of the cell, uc the input and yc the output

iubtyatxdt

dx dcd

dcd

cc

)()(

)1)(1)((2

1 txtxy ccc

CNN: Activation Function

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

CNN: Block Diagram

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

CNN: Discrete Model

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Computing x(∞), the model can be represented as

iubnyanx dcd

dcd

c )()(

0)(,1

0)(,1)(

nx

nxny

c

c

c

using the following activation function

Steps: Crossover C(Fg) Mutation M(*) Adding random parent Ag()

GA: main steps proposed

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

dc

bc

da

ba

FC

dcMbaMM

MF

g

g 212

1 ,,

GA: Crossover

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

125.0125.

000

000

A

125.125.0

000

000

B I=0

125.00

000

000

A

125.125.125.

000

000

B I=0

2121

2211

2222

2121

2211

1111

1

cccc

bbbb

aaaa

cccc

bbbb

aaaa

P

Individual 1

Individual 2

GA: Crossover

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

a1

b1

c1

a2

b2

c2

GA: Mutation

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

mij

mijmij PrF

PrFPFM

)(

)(),(

where rU(0,1) is a random variable with uniform distribution defined on a probability space (,,P),

GA: Mutation (resolution)

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

'2

'1

'2

'1

'2

'2

'1

'1

'2

'2

'2

'2

'2

'1

'2

'1

'2

'2

'1

'1

'1

'1

'1

'1

'1

cccc

bbbb

aaaa

cccc

bbbb

aaaa

P

125.0125.

000

000

A

125.125.0

000

000

B I=0

125.00

000

000

A

125.125.125.

000

000

B I=0

Individual 1

Individual 2

a1

b1

c1

a2

b2

c2

Population=sons + sons mutated

g

g

g

C FA

M C F

ggn

pgg AOnAS pmin,

GA: Selecting Parents

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

)(1 wF

FF

g

gg

GA: Adding Random Parent

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

THEOREM 1: (Weierstrass’s Approximation Theorem)

Let g be a continuous real valued function defined on a closed interval [a,b]. Then, given any positive, there exists a polynomial y (which may depend on ) with real coefficients such that:

For every x [a,b].

Polynomial Discrete Time Cellular Neural Network

)()( xyxg

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

THEOREM 2 *: Any Boolean Function of n-variables can be realized using a Polynomial Threshold gates of order sn.The quadratic threshold gate can be defined:

And s is the number of inputs and T is the threshold constant.

Polynomial Discrete Time Cellular Neural Network

otherwise

Txxwxwify

n

i

n

ijiijii

0

11 1

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

* N. J. Nilsson. The Mathematical Foundations of Learning Machines. McGraw Hill, New York, 1990.

PDTCNN: the model (I)

cdd

cNd

dcd

cNd

dcd

c iyuguBkyAkxrr

),()()()()(

0)1(1

0)1(1))1(()(

kxf

kxifkxfky

c

ccc

)0(

)0(

)0(

),(

)(

)(

)(

cd

cNd

cd

cd

cNd

cd

dd

cNd

cd

dd

yuP

uyP

yuP

yug

r

r

r

)(

)(

)(

),(

)(

)(

)(

kyuP

ukyP

kyuP

yug

cd

cNd

cd

cd

cNd

cd

dd

cNd

cd

dd

r

r

r

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

PDTCNN: the model (II)

)()(),()()(

kuQkyPyug d

cNd

cd

d

cNd

cd

dd

rr

cdd

cNd

dcd

cNd

dcd

c iyuguBkyAkxrr

),()()()()(

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

PDTCNN: Solving XOR problemSome papers: Z. Yang, Y. Nishio, A. Ushida,

Templates and algorithms for two-layer cellular neural networks. IJCNN’02, 2002.

F. Chen, G. He, G. Chen & X. Xu,

Implementation of Arbitrary Boolean Functions via CNN. CNNA’06, 2006.

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

PDTCNN:Solving XOR problem M. Balsi, Generalized CNN: Potentials of

a CNN with Non-Uniform Weights. CNNA-92, 2002 .

E. Bilgili, I. C. Göknar and O. N. Ucan, Cellular neural network with trapezoidal activation function. Int. J. Circ. Theor. Appl., 2005

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Learning parameters Initialpop=20000 Number of fathers=7 Maximum number of random

parents to be add = 3 Kpro=0.8 Increment=1 Mutation Probability=0.15

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

PDTCNN:First Scheme U:uij=xijxij+1

1401

150140

iP

BA

0300

010030

iP

BA

0100

010010

iP

BA

)()()( 1 kyuukykx ccccc

)(

)(

)(

),(

)(

)(

)(

kyuP

ukyP

kyuP

yug

cd

cNd

cd

cd

cNd

cd

dd

cNd

cd

dd

r

r

r

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

PDTCNN:Second Scheme U:uij=xijyij

2281

000000

iP

BA

b)

1151

000001

iP

BAc)

0 1 0 0 0 0

0 1 0 0

A B

P i

)()()( kyukykx cccc

)0(

)0(

)0(

),(

)(

)(

)(

cd

cNd

cd

cd

cNd

cd

dd

cNd

cd

dd

yuP

uyP

yuP

yug

r

r

r

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

The Game of Life (I) The Game of Life (GoL) is a totalistic cellular

automaton consisting in a two-dimensional grid cells, that may be either alive (black) or dead (white).

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

The Game of Life (II) The state of each cell varies according to the

following rules: Birth: a cell that is dead at time t becomes

alive at time t + 1 only if exactly 3 of its neighbors were alive at time t;

Survival: a cell that was living at time t will remain alive at t + 1 if and only if it had exactly 2 or 3 alive neighbors at time t.

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

The Game of Life (III) Every sufficient well-stated mathematical problem can be reduced to a question about Life;

It is possible to make a life computer (logic gates, storage etc.);

Life is universal: it can be programmed to perform any desired calculation;

Given a large enough Life space and enough time, self-reproducing animals will emerge...

The whole universe is a CA! (E.Fredkin, MIT).Intro CNN & GA Polyn. CNN XOR GoL Conclusions

The Game of Life – NOT gate

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

A

CNN & GoL Multilayer CNN (Chua, Roska) – 1990 Activation function (Chua, Roska) –

1990 CNN-UM (Roska,Chua) -1990 CNN Universal Cells (Dogaru, Chua) –

1999

Simplicity vs. Computational power

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Polynomial CNN (I)

),()()( ddede

ede

d yugiubnyanx

0)(,1

0)(,1)(

nx

nxny

d

d

d

What’s g(ud,yd)?

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Polynomial CNN (II) In the simplest case g(ud, yd) is a second

degree polynomial, whose general form is

2

0

2 ))()()()((),(i

iedei

iedei

dd yqupyug

200 )()()1()( ed

eed

e yqp

)()()()( 11ed

eed

e yqup

)1()()()( 22

2ed

eed

e qup

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Polynomial CNN (III)

Thanks to some considerations we find that

)()()()()()()( 12

0ded

eed

eed

eed

ed yupyqiubnyanx

000

00

000

caA

ppp

pcp

ppp

bbb

bbb

bbb

B

000

00

000

0 0cqQ

ppp

pcp

ppp

ppp

ppp

ppp

P

111

111

111

1

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Polynomial CNN (IV)

pcppccccd uypbuypbnx )()()(

iyqya cccc 2

uc and appear in the state equation

direct link with totalistic Cellular Automata

pu

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

GoL: Rules (I)

GoL: Rules (II)

Rule 1: a cell will be alive at least 3 of the 9 cells in its 3 × 3 neigh. are alive

Black pixel= +1

White pixel= -1pixel centr. = 1 (black)

Σ neigh. = -2 (5 w, 2 b)

next state = -1 (white)

GoL: Rules (III)

Rule 2: a cell will be alive if at most 3 of its 8 neighbors are alive

Black pixel= +1

White pixel= -1pixel centr. = 1 (black)

Σ neigh. = -2 (5 w, 2 b)

next state = 1 (black)

Design algorithm (I) First iteration: we try to perform the first rule (a

cell will be alive at least 3 of the 9 cells in its 3 × 3 neigh. are alive)

If Y(0)=0

bc=1 bp=1 i=3

pcppccccd uypbuypbnx )()()(

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Design algorithm (II) Second iteration: we try to accomplish the

second rule (a cell will be alive if at most 3 of its 8 neighbors are alive)

pcpcccd uypuypnx )1()1()(

32 cccc yqya

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Design algorithm (III) Hyp: pc=0

Templates found using learning

Coming soon...

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Conclusions (I) In general:

In some cases it is possible to reduce a multilayer DTCNN to a single layer PDTCNN

Thanks to the GoL we can explore the capacity of PDTCNNs for Universal Machine

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Conclusions (II) About learning:

The resolution used reduces the search space

The step “Add random parent” improves the behavior to avoid local minimas

About design We give a simple algorithm to design

templates for the Polynomial CNN

Future Work Implementations of mathematical

morphology functions with PDTCNNs

Intro CNN & GA Polyn. CNN XOR GoL Conclusions

Polynomial Discrete Time Cellular Neural Networks

Eduardo Gomez-Ramirez Giovanni Egidio Pazienza

egr@ci.ulsa.mxgpazienza@salle.url.edu