lecture 39 hopfield network

Intro. ANN & Fuzzy Systems

Lecture 39 Hopfield Network

(C) 2001-2003 by Yu Hen Hu 2


Outline

• Fundamentals of Hopfield Net• Analog Implementation• Associate Retrieval• Solving Optimization Problem

(C) 2001-2003 by Yu Hen Hu 3


Fundamentals of Hopfield Net

• Proposed by J.J. Hopfield. A fully Connected, feed-back, fixed weight network.

• Each neuron accepts its input from the outputs of all other neurons and the its own input:

Net function

Output:

iii

jiijiji

RuI

Ivwu

/

.01

01

i

ii u

uv

+

+

+

V1

V2

V3

I1

I2

I3

–T1

–T2

–T3

(C) 2001-2003 by Yu Hen Hu 4


Discrete Time Formulation

• Define V = [V1, V2, • • •, Vn]T, T = [T1, T2, • • •, Tn]T, I = [I1, I2, • • •, In]T, and

Then V(t+1) = sgn{ WV(t) + I(t) – T(t)}

0

0

0

0

321

33231

22321

11312

nnn

n

n

n

www

www

www

www

W

(C) 2001-2003 by Yu Hen Hu 5


Example

Let

Then

0;

0111

1011

1101

1110

;

1

1

1

1

)0(

TIWv

1

1

1

1

}

1

3

1

1

sgn{)]0(sgn[)1( vWv

(C) 2001-2003 by Yu Hen Hu 6


Example (continued)

[1 1 1 –1]T and [–1 –1 –1 1]T are the two stable attractors. Note that

)1(

1

1

1

1

}

3

3

3

3

sgn{)]1(sgn[)2( vvWv

0111

1011

1101

1110

1

1

1

1

1111

1

1

1

1

W

(C) 2001-2003 by Yu Hen Hu 7


Observations

• Let v* = [ 1 1 1 1]T. For any v(0) such that vT(0)v* 0,

Otherwise, v(t) will oscillate between ±v(0).• Exercise: try v(0) = [ 1 1 1 1]T or [ 1 1 1 1]T. • Discussion:

– Synchronous update: All neurons are updated together. Suitable for digital implementation

– Asynchronous update: Some neurons are updated faster than others. Not all neurons are updated simultaneously. Most natural for analog implementation.

*)(lim vtvt

(C) 2001-2003 by Yu Hen Hu 8


Lyapunov function for Stability

Consider a scalar function E(V) satisfying:

(i) E(V*) = 0

(ii) E(V) > 0 for V V*

(iii) dE/dV = 0 at V = V*, and dE/dV < 0 for V V*

If such an E(V) can be found, it is called a Lyapunov function, and the system is asymptotically stable (i.e. V V* as t ).

(C) 2001-2003 by Yu Hen Hu 9


Hopfield Net Energy Function

• Hence, Hopfield net dynamic equation is to minimize E(v) along descending gradient direction.

• Stability of Hopfield Net – If wij = wji & wii = 0, the output will converge to a local minimum (instead of oscillating).

RuIWvvE

dzzfR

vIvvwvE

v

i

v

iii

iiji ji

jiij

i

/)(

)(1

2

1)(

0

1

,

(C) 2001-2003 by Yu Hen Hu 10


Associative Retrieval

• Want to store a set of binary input vector {bm; 1 m

M} such that when a perturbed b'm is presented as I

(input), the binary output V= bm.

• Weight Matrix: Assume binary values ±1

T

M

m

Tmm

Tmm

U

bbdiagbbW

]000[

1

(C) 2001-2003 by Yu Hen Hu 11


Example

b1 = [ 1 1 1 –1]T, b2 = [1 1 –1 –1]T

Let I = V(0) = [ –1 1 –1 –1]T, then

0

0

0

;

0022

0000

2002

2020

UW

1

1

1

1

4

0

0

0

)2(,

1

1

1

1

0

0

0

4

)()1( fVfITfV

(C) 2001-2003 by Yu Hen Hu 12


Hopfield Net Solution to TSP

• (Hopfield and Tank) Use an n by n matrix to represent a tour. Vij – i-th city as the j-th stop. Each

entry is a neuron!

A 0 1 0 0 0 5

B 0 0 0 1 0 4

C 0 0 0 0 1 3

D 0 0 1 0 0 2

E 1 0 0 0 0 1City/tour 1 2 3 4 5

(C) 2001-2003 by Yu Hen Hu 13


Energy Function

First three terms makes V a permutation matrix. Last term minimizes the tour distance

Validity of the solution – e.g. the A, B, C, D coefficients in the TSP problem. Quality of the solution – the initial condition will affect the

x yx iiyiyixyx

x iix

i x xyiyix

x i ijjxix

vvvdD

NvC

vvB

vvA

E

1,1,,,

2

,

,,,,

22

22

otherwise. 0; if 1 where

)1()1( 1,1,,

ji

DCBAW

ij

ijijxyxyijijxyyjxi

lecture 39 hopfield network

Documents