7/28/2019 Neural Basic

1/6

Dynamic Neural Networlks: An OverviewN.K. Sinha*, M.M. Gupta** and D. H. Rao***

*McMaster University, Hamill on, Canada**University of Saskatchewan, Saskatoon, Canada***Gogte Institute of Technology. Belgaum, India

A B S T R A C TOver the last decade several advance< havcbeen made in the pddigm of artificial neurdnetworks with specific emphasis on architecturesadlearning algorithms. However, most of the work isfixu.stxi on static ( f d o n m d ) neural networks.

These neural networks respond instantaneously to theinputs, for they do not posses any time delay units.The use of time delays in neural networks isneurobiologically motivated, since it is well k m w nthat signal delays arc omnipresent in the brain aalplay an important role in neurobiological informationprocessing. This conccpl has led to the developmentof dynamic neural networks. It is envisaged thatdynamic neural networks, in addition to betterreprescntation of biological neural systems, offerbetter L omputationdl capabilities compared to theirstatic counterparts. The objective of this paptx is togive an overview of dynamic neural structures.

1. IntroductionMost of the neural network structures USXIprcsently for engineering applications are static(fimifiward) neural networks. These neural

networks, having a number of neurons, respondinstantaneously to the inputs. 'This form of staticinput-output mapping is well suited for patternrecognition applications wherc both the input vectorx and the output vector y represent spatial patternsthat are independent of time. The absence of fixxihackin static mural nctwtwks ensur&\ that networks an:conditionally stahlc:. 1 lowever, these networks svfierfrom the following Imitations 11 1: (i) In Wfiwdnetworks the information flows form Neuron A to H,to C, to D, a d never comes hack to A . (ii) Thestructure of an artificial neuron is not dynamic innature and performs a simple summation operation,and (iii) The static neuron model does not take intoaccount the time delays that affect the dynamics of thesystem. Time delays afe the inherent chracmtics ofbiological neurons.

Unlike a static neural network, a dynamicneural network employs extensive fAiiiick be twenthe neurons of a layer, andor between the layers of

the network. This feedback implies that the networkhas local memory characteristics. The node equationsin dynrmic networks are descri bedby difhentd ordiffkwnce equation. Because of feedbackpaths fiomtheir outputs to the inputs, the response of dynamicneural networks is recursive. That is, the weightsare adjusted, the output is then recalculated, and theprwem is repeated. For a stable network, successiveiterations prtxluce smaller and smaller output changesuntil eventually the outputs become constant.

Neural networks with feedback areparticularly appropriate for system modeling,identification, control and filtering applications.These networks are important h u s e many of thesystems that we wish to model in the real wr ld arenon-linear dynamical systems. This is true, forexample, in controls area in which we wish to modelthe fimvard or inverse dynamics of systems such asairplanes, rockets, sqmceLratland robots 12 - 51.

The dynamic neural structures, in general,can be classified into two categories. The firstcategory encompasses the dynamic neural stnrcturesdcvelopd based on the concept of single neurondynamics as an extension of static neural networks.The " T e n t neural network developed by Hopfielld161, Brain-State-in-a-Box developed by Anderson et id171, the-delay neural network &vdoped by Lang,Waibel and Hinton 181, dynamic neural unit (DNII)developed by Gupta and Rao 191 belong to thiiscategory. The .wmd category encompasses dynamxcneural :itructures which are developed based on theinteractionof excitatoryand inhibitory o r antagonisticneural !;ubp)pulations. Neural structures developedby Wikon and Cowan [IO],Gupta and Knopf [l l] ,and Rao and Gupta 1121belong to this category.

The objective of this paper is to describe theabove nientioned dynamic structures in brief. In viewof this, the paper is organizedas follows. The firstcategory of dyllarmc neural nehmks is described inSection 2. The second category, namely; dynamicnetworls based on neural subpopulations, is desQibedin Section 3, followed by the conclusions in the lastsection.

0-7803-5812-0/00/$10.0092000IEEE 491

7/28/2019 Neural Basic

2/6

2. Single Neuron Dynamics2.1 Recurrent Neural Networks

The feedback, also known as recurrent,neural networks, intnduced by Hopfield, is one of thefirst dynamic neural network models. This modelconsists of a single layer Network N1, included in afeedback configuration with a time delay as shown inFig. 1. this figure, y(k) and y(k+l) represent thestates of the neural network at instants k and k+ 1, xorepresents the initial value, W(k) &notes the vectorof the neural weights, Yr .] is the nonlinear activationfunction, amiz-l represents the unit delay operator.This feedback network represents a discxete-timedynamical system and can be &bed by thefollowing equation [R

whereN1 .I =Y W XI Given an nitial value xo,the dynamical system evolves to an equilibrium stateif NI is suitably chosen. The set of initial conditionsin the neighborhood of ~0 which converge to thesame equilibrium state is then identified with thatstate. The term "associative memory" is used todescri besuch systems.

Hopfield neural model has been widely usedin many applications such as system identificationam control, robotics, machme vision, patternrecognition, associative memories, combinatorialoptimization problems. For mathematica!tractability, it is aSSumed that Hopfield nemalstructure contains only single layer of neurons.Recently, however, multi-layered recurrent neuralnetworks have been developed aml used for manyapplications. Furthermore, inputs, when exist, arcassumed to be constant 151. In spite o f the interestingapplications that recurrent neural network.. have beenused for, the basic architecture of the neuron is static;that is, the neuron simply provides a w igh t4integration of the synaptic inputs over a period oftime. In other words, there are no dynamicalelements within the structureof the neuron.2 .2 Brain-State-in-a-Box (BSB) NeuralModel

The Brain-State-in-a-Box (BSB) neuralmodel, devdopwl by Anderson etal 171, is a positivefeedback system with amplitude limitation. Itconsists of a highly interconnectedset of neurons that

feed back upon themselves. The BSB operates byusing the built-in positive ftuxlback to amplify aninput pattern, until all neurons in the structure mdriven into saturation. From the dynamics p i n t ofview, the BSB model can be viewed as a discretelinear system in a saturated mode. The maindifference between the BSB and a usual disaetemsystem is that the linear system is defined on % ,while the BSB mtdel is &find on the closed n-dimensional hypercube 131. The BSB model is shownin Fig.2, and is defined by the following set ofequations.

whereW demtes a symmetric weight matrix, x(k) isthe state vectorof the model at discrete time k, V1.1 isa piecewise linear function and [3 is a small positii econstant called the fedback factor. The BSR modelfunctions as follows [31: An activation pattern x(0) isinput into the BSB model as the initial state vector,and Eqn. 2(a) is used to compute y(0). Equation 2(b)is then used to truncate y(O), obtaining the updatedstate vector x ( 1). This pmcedureis repeated until theBSB mtdel withes a stable state represented by aparticular comer of th e unit hypercube. Positivefixxiback on the HS R mtxlel causes th e initial statevector x(0) to increase in Euclidean length with anincreasing number of iterations until it hits a wall ofthe box (unit hypercube), then sliding along the walland eventually ending up in a stable corner o f thebox.

A natural application for the BSI3 nitdel ISclustering. This follows from the fict that thcstable corners of the unit hypercuh ac t as pointattractors with well-behwed basins of attraction.Consequently, the BSR mtxicl niay he u s d as ailunsupewised clustering algorithm, with cach stakk:corner the unit hypercube represcnting a c.lu\ter qdrelatecl data.2 .3 l'ime-Delay Neural Networks ('I'DNN)

It is possible to use a static network toprocess timc: series data by simply converting thetcrnporal !qucncc into a staic: pattern by utth~ldingthe sequence ovcr time. From a practical point 01view, it is possible to unfold lhe sequence over afinite period of time 121 '[his can be accomplishedby feeding the input sequence into a tapped delay lineinto a static neural network architecture. Anarchitecture like this is often ref& to as a 'rime-.Delay Neural Network ('I'DNN) 171. It should benoted, however, that the 'I'DNN is a j&&+hnwtd

492

7/28/2019 Neural Basic

3/6

network it attains dynamic behavior by virtue of thefactthat each synapseof the n e m k s designed asanFIR filter. For more details with regard tomathematical development and learning algorithm,r e z a d e n s a r e ~ t o3].The T D N neural strucutt:has been used inmany applications. For example, Sejnowski and

conversion, while Waibel et a1 [141 employed thisstructure for phoneme recognition. Narendta andpartbasarthy [5] have applied this type of neuralstructure for system identification and control ofnodinea~ynamical systems.

R0S-g [13] used TDNN for t ex t - tO-Fh

2.4 Dynamic Neural Unit (DNU)The dynap ic m m l unit (DNU), proposedby Gupta and Rao (61, is a dynamic model of thebiological neuron. It consists of a seconl-orderdynamics whose output constitutes the argument to atime-varying nonlinearactivation function. Thus, the

DNU performs two distinct operations: (i) thesynapt ic operarion and (ii) the somatic opemtion.The first operation cOrreSpOnds to adaptation ofi k d h m d and feedback synaptic weights, while thesecond COrreSpOMiS o adaptation of gain (slope) of thenonlinear activation function. The DNU consists ofMay elements, feed- and f d h k pathsweighted by the synaptic weights aff and hfirespectively representing a xxond-txder structurefollowed by a nonlinear activation function as shownin Fig. 3. In DNU, vl(k) is the output of' thedynamic structure, u(k) is the neural output, and aff=[ao, a,, 51 and bfb =Ibl b21 are the vectors ofadaptable tbdf "d and fedback weightsrespectively.The nonlinear mapping operation onvl(k) yields a neural output U@) given by:

where Y [.] is some nonlinear activation function,usually the sigmoidal function and gs is the somaticgain which controls slope of the activation function.Details of this neural structure, the development oflearning algorithm and its application to the controlof unknownnonlinear dynamic systems are discussedin [9].3. Dynamic Neural Structures Based o nNeural Subpopulations

The neural network structures kscr ibwi inthe earlier section consider the behavior of a single

neuron as the basic computing unit for dexribingneural i n f o " processing operations. Eachcomputing unit in the network is based on anidaalivd neuron. An ideal neuron is assummi torespond optimally to the applied inputs. However,experimental studies in neuro-physiology show thatthe response of a biological neuron appear random[7J, a d nly by averaging many observations is itpossible to obtain predictable results. However,mathematical analysis has shown that these randomcells can transmit reliable information if they aresufficiently redundant in numbers.

The total neural activity ge mdtd within atissue layer is a result of spatially I t x a l i z x lassembliesof densely interconnected nerve cells Calildneumi' population, or neural mus.\. The neuralpopulation is comprised of neurons, and its propertieshave a generic resemblance to those of Wvic'lualneurons. But it is not identical to them, and itsproperties can not be predicted from measurements onsingle neurons. This is due to the fact that theproperties of neural population depend on variousparanems of individual neuronsa d lso dependuponthe interconnections hetween neurons 1101. The studyof neural networks based o n single-neuron analysi\precludes the above two facets of biological neuralstruclurw. The conceptual gap between the functionsof single neurons and those of numbers of neurons, aneural mass, is still very wide.

Fach neural population inay be hitherdivided into several coexisting suhpopulatiotis. Asubpopulation contains a large class of similarnt:urons that lic in close spatial proximity. The mostcoininon neural mass is the mixturc o f excitaloty(posi tivc) and h1hihifor-ynegative) subpopulations ofneurc:)ns. 'ne xcitatory neural subpopulationincreases the electro-chemical potential of the post-synaptic neuron, while the inhihitory suhpipulationd u c ~he electctro-chemical potential. The mini rnunitopoloby o f such a neural mass contain! cxc i t i i t t n y(positive), inhihitory (negative), excitatoryinhibitory (synaptic connection from excitattny toinhihitory), and inhibitory - excitattny (synapticconrmtion from inhibitory to excitatory) l idbackloops. B d n this hypothesis, Gupb and K.nopfprqmed a neural model n a d P-N neural procmsorI I 1 for machine vision applications, and Ra o anlGupta proposed dyn arnic neur al processor (DNP) fixrobotics and control applications 1151. A briefdmxiption of these two dynamic neural structures isgiven below.3.1 P-N Neural Processor

The computational role performed by theprocessor,named the Positive-Negative (PN) r i e d

493

7/28/2019 Neural Basic

4/6

processor, emulates the spatio-temporal informationprocessing capabilities of certain neural activity fieldsfound along the human visual pathway. The state-space model of this visual information prmessorcorreqwnds to a bi-layered two-dimensional array ofdensely interconnectednonlinear processing ekmenrs(PES). An individual PE represents the neuralactivity exhibited by a spatially localizedsubpopulation of excitatory or inhibitory nerve cells.A ~ for the statespace model of the PN m a lprocessor is shown in Fig. 4.

Two types of inputs are received by theconstituent PES. One type arises from the externalinputs originating in the signal space S(k), and thesecond type originates from the lateral and recurrentconnections between the PES within a commonnetwork X(k). The spatial transformatir:n of theexternal input signal is given by the matrix A of thefe@brwud subnet, and the strength of lateral adrecurrent connections between individual PES isgiven by the matrix W of thefedbucksubnet.

A variety of information processingoperations asstxiated with the early stages of machinevisior can be realized by the basic PN new&processor architecture. These diverse operations a~achieved by selectively programming the systemmatrices A and W, and the parameters of thenonlincar mapping opmtor @[ - I . The coefficients ofA in the fkedfiwd subnet are made to act as a linearspatial filter that either smooth or enhance featuresembedded within the external input S(k) . Thecwfficient- wlcctd for W ili the feedback subnet, inconjunction with properly chosen parameters for @ I - 1,will determine the computational role of the processorin terms of spatio-temporal filtering (STF), motiondetection, short-term visual mcmory(STVM),patio-temporal stabilization (STS) and pulse fi.txluencymodulation (PFM).3.2 Dynamic Neural Processor (DNP)

The dynamic neural processor (DNP) isdeveloped based on physiological evidence thatnervous activity of any complexity depends upon theinteraction of the excitatory and inhibitory neuralsubpopulations. The DNP hus functionally mimicsthe aggregate dynamic properties of a neuralpopulation or neural mass. The DNP comprises oftwo basic nodes called the dynumic nczrrul units(briefly desctihed in the preceing subsection) and arr:coupled in excitatory and inhibitory modes. TheDNU embodies delay eloments, tixxlforward a dfixxlkick weights tblloweci by a time-varyingnonlinear activation function; thus different from theconventionally assumed structure of neurons. ThemorpholoLy of DNP is shown in Fig. 6.

The functional dynamics exhibited by aneural computing unit, the DNU, is defined by asecond-orderdi fk mc e equation. The state variablesuE(k+l) and U (k+l ), generated at time (k+l ) by theIexcitatoryand id&&axy n e d un ts of the proposedneural processor, are modeled by the nonlinearfunctional relationships:

where v (k) and v{k) represent the proption ofneurons in the neural unit that m i v e nputs gmterthan an intrinsic threshold, andE and I represent thenonlinear excitatory and inhibitory actions of theneurons. The neurons that w i v e nputs greater thana threshold value is given by a nonlinearfunction ofvh(k), Y [vh(k)] . The total inputs incident on theexcitatoryand inhibitory neural units arerespectively

E

where w and w are the weights associated with theexcitatmy and inhibitory neural inputs respectively,w and w represent the self-synaptic connectionstrengths, w and w represent the inter-neuronsynaptic strengths, and 8 and 8 repsent thethresholds of excitattny and inhibitory mummrespectively.

E IEE n

1E E1E I

The DNP has been used for computationalpurposes, such as functional approximation,computation of inverse kinematic transformations ofrobots and control of unknown nonlinear dynamicsystems [151.4. Conclusions

Static neuron model used in artificialInetworks ignores many of the characteristics of itsbiological counterpart, and is a grossly simplifiedversion. As the name indicates, static (feedfhmd)

494

7/28/2019 Neural Basic

5/6

neural network? do not incorporate feedback On theother hand, dynamic neural networks employ extensivef d b k etween the neurons of a layer, and/or beheenthe layers of the network. This feedback implies thatdynamic neural networks have local memorycharacteristics. The node equations in dynamicnetworks are descr i bed by differential or differenceequations. Neural networks with feedback areparticularly appropriate for system modeling,identification, control and filtering applications.Recently, many researchers are focusing their efforts indeveloping dynamic neural structures and dyraarmclearning algorithms. In thispw we have presented anoverview of dynamic neural structures.

ReferencesJ. J. Hopfield, "Artificial Neural Networksare Coming", Z E E E w e n , An Intewiewby W. Myers, pp. 3-6, April 1990.D.R. Hush and B.G. Home, "Progress in

Supervised Neural Networks", ZEEE SignalProcessing Magazine, January, Volume 10,NO. 1, pp. 8-39, 1993.S . Haykin, Neural Networks: AComprehensive Foundation, Z Z EPress. 1994.MM. Gupta and D.H. Rao, "Neuro-ConlrolSystems: A Tutorial", in NeuroControlSystems: Theory and Applications, Eds.,IEEE Press, March 1994.M.M. Gupta and D.H. b o , pp. 1 - 43,

K.S. N a " and K. Parthamthy,"Identification and Control of Dynam~calSystems Using Neural Networks", ZEEETrans. on Neural Networks, pp. 4-27, Vol.1, No. 1, March 1990.JJ. Hopfield, "Neurons with (;lradedResponse Have Collective ComputationalProperties Like of Those Two-StateNeurons", Proceedings of the NarionalAcademy of Sciences, Vol. 81, pp. 3088-.3092,1984.J.A. Andason, .W. Silverstein, S.A. Ritzand R.S. Jones, "Distinctive Features,Categorical Perception, and ProbabilityLaming: Some Applications of a NeuralModel", Psychological Review, vol. 84, pp.413-451, 1977.

KJ. Lang, A.H. Waibel and G.E. Hinton," A Time-Delay Neural Network Architecturefrom Isolated Word Recognition", Neu dNetworks, Vol. 3, No. 1, pp. 234,1990.M.M. Gupta and D.H. b o , " m cNeural Units With Applications to theControl of Unknown Nonlinear Systems",Ihe Joumal of Intelligent and FuzzySy,stems, Vol. 1, No. 1, pp. 73-92, Jan.1993.H.R. Wilson and J.D. Cowan, "Excitatoryami Inhibitory htemctions in LocakedPopulations of Model Neurons", BiophysimlJ a ~ t n a l ,Vol. 12, pp. 1-24,1972.M M. G q t a andG.K. Knopf, " A MultitaskVisual Infurmation Processor with aBilologically Motivated Design", J m d ofVimal Communication and ImageRepresentarion, 3 , No. 3, pp. 230-246,Sept. 1992.D.H. Rao and MM. Gupta, "FunctionalApproximation Using Dynamic NeuralPr~xesm",n t e " d Joltmal of hhmlMhs-Pamllel Computing and I n f o r m a t hsystems, Vol. 5 , pp. 573-592,1994.T. Sejnowski and C.R. Rosenberg,"NETtalk:A Neural NetworkThat Learns toRelad Aloud", Tech. Report JHWEECS-86/01, John Hopkins University, 1986.A. Waibel, T. Hanazawa, G. Hinton, K.Shikano and KJ. Lang, "PhonemeRecognition Using Time-Delay NeuralNetworks", IEEE l).ans. on Acoustics,Speech and signa l Processing, Vol. 37, No.3, pp. 328-339, March 1989.M.M. Gupta and D.H. bo, "DynamicNwral Processor and Its Applications toRobotics and Control", in IntelligentControl, ZEEE Z - k s , Eds., M.M. Gupta andN.K. Sinha, pp. 515-545,19%.

495

7/28/2019 Neural Basic

6/6

Static nard n c l w r k

Il k )Fig. 2 Block diagramoflhc 0SB m od c l

Fig.4: Thc Y N ncunl pnrcssor.

Fig. 5 : 'Ilic dyriamic ricuml prwcssor (DN1').

496