computational models of the basal ganglia

Review

Computational Models of the Basal Ganglia

*Andrew Gillies, PhD, and †Gordon Arbuthnott, PhD

*Institute for Adaptive and Neural Computation, and †Preclinical Veterinary Sciences, The University of Edinburgh,Edinburgh, U.K.

Summary: Computer simulation studies and mathematicalanalysis of models of the basal ganglia are being used increas-ingly to explore theories of basal ganglia function. We reviewthe implications of these new models for a general understand-ing of basal ganglia function in normal as well as in diseasedbrains. The focus is on their functional similarities rather thanon the details of mathematical methodologies and simulationtechniques. Most of the models suggest a vital role for the basalganglia in learning. Although this interest in learning is partlydriven by experimental results associating the acute firing ofdopamine cells with reward prediction in monkeys, some of the

models have preceded the electrophysiological results. An-other common theme of the models is selection. In this case, thestriatum is seen as detecting and selecting cortical contexts foraccess to basal ganglia output. Although the behavioral conse-quences of this selection are hard to define, the models provideframeworks within which to explore these ideas empirically.This provides a means of refining our understanding of basalganglia function and to consider dysfunction within the newlogical frameworks.Key Words: Basal ganglia—Computermodel—Parkinson’s disease—Reinforcement learning.

Over the last decade a large number of models of thebasal ganglia have been developed. A great proportion ofthese are demonstrated and explored through computersimulation. Many of the models not only address facetsof basal ganglia dysfunction, such as Parkinson’s dis-ease, but also attempt to place dysfunction into the con-text of normal basal ganglia operation. We describe threeprominent classes of models, each of which can bealigned with different aspects of basal ganglia functionand consequently views basal ganglia dysfunction fromdifferent perspectives. Shared among these classes is acommon theme, the ability for the basal ganglia to detectcontexts of cortical activity and respond to these contextsin terms of choosing sets of cortical activity, possibly foraction by the motor system. The majority of modelsbring together the detection of cortical contexts with theselection or enhancement of cortical activity, which maybe the selection of motor or cognitive actions or the

modulation of activity in the generation of sequences ofpatterns. A broad selection of models will be outlined,demonstrating the diverse approaches in describing thefunctional properties of basal ganglia anatomy andphysiology.

The models chosen for inclusion in this review de-scribe basal ganglia function at the level of networks ofnerve cells. This allows us to make comparisons at asimilar level of functional description. All the modelsreviewed are based on or associated with basal gangliapathways as shown in Figure 1. This diagram illustratesmany of the predominant pathways within the basal gan-glia and introduces the abbreviations used. Primary neu-rotransmitter types associated with each projection arealso shown.

MODELS OF REINFORCEMENT LEARNING

The phenomenon of learning behaviors from rewardswhich act as reinforcement has been extensively mod-eled.1,2 Almost all models that ascribe a learning role tothe basal ganglia are based on some form of reinforce-ment learning (a class of methods specifying how a se-quence of actions can be learned through reward signals

Received January 7, 2000. Accepted March 24, 2000.Address correspondence and reprint requests to Andrew Gillies, PhD,

Institute for Adaptive and Neural Computation, Division of Informat-ics, 5 Forrest Hill, University of Edinburgh, Edinburgh EH1 2QL,Scotland.

Movement DisordersVol. 15, No. 5, 2000, pp. 762–770© 2000 Movement Disorder Society

762

provided at the conclusion of the sequence). Recent ex-perimental evidence suggests that the basal ganglia maymediate aspects of reinforcement learning.3 In a classicconditioning task, after a number of trials, dopamineneurons lose their response to the primary rewardingstimulus and at the same time gain a response to theconditioned stimulus.4 This observation has led to theadoption of a reinforcement learning algorithm of anengineering origin, called the Temporal Difference (TD)learning algorithm, as a description of basal ganglialearning.

Barto1 describes an application of the TD algorithm inthe Actor–Critic model, which has been fitted to certainaspects of basal ganglia anatomy, physiology, and cyto-chemistry.5 The Actor–Critic model is split into twocomponents, the actor and the critic (Fig. 2). The criticprovides an “effective reinforcement” signal to the actor,which then learns behavioral responses to the environ-ment using this signal. The objective of the actor is tolearn to generate or select actions that maximize primaryreward. However, the actor learns from instantaneousfeedback. If the actor receives primary reinforcement di-rectly from the environment and the arrival of reinforce-ment is delayed after the action that caused it (which isusually the case), the actor would then fail to associatethe underlying action with the reward. This problem is

called the temporal difference problem. The role of thecritic is to learn to predict primary reward and produce asignal (called the “effective reinforcement”) that can beused as reinforcement by the actor so that the actor re-ceives an effective reward to associate with the actionjust initiated. Thus, there are two learning systems here:the actor that learns to perform actions that maximizereward, and the critic that learns to predict rewards fromthe actions being performed and the current state of theenvironment.

It has been proposed that neural implementation of theactor is associated with striatal matrix compartments andthat of the critic with striatal patch compartments anddopamine neurons of the substantia nigra pars compacta(SNc). The dopamine signal originating from the SNcplays the role of effective or predicted reinforcement.5

Properties of such a role have been identified in dopa-mine neurons.3 In the classic conditioning task men-tioned above, dopamine responses move to the earliestpredictor of reward4 (for example, the conditionedstimulus) providing a “critic-like” prediction of primaryreward. Although the mechanisms shown in Figure 2work though the direct pathway, the indirect pathwayinvolving the globus pallidus external segment (GPe)and subthalamic nucleus (STN) is proposed to mediatethe transfer of dopamine response from primary rein-forcement to predictor.5

This model of reinforcement learning has been ex-plored in various ways in the context of basal gangliaarchitecture. A number of similar models examine thetypes of SNc dopamine neuron responses expected if

FIG. 1. Major pathways through the basal ganglia. The indirect path-way between the striatum and substantia nigra/external globus pallidusis noted. The dot–dash line represents a dopaminergic projection. CTX,cortex; STR, striatum; STN; subthalamic nucleus; GPe, globus pallidusexternal segment; GPi, globus pallidus internal segment: THAL, thala-mus; SNr, substantia nigra pars reticulata; SNc, substantia nigra parscompacta. Solid pathways indicate primarily GABA (g-aminobutyricacid)-mediated inhibitory connections; dotted pathways indicate pri-marily glutamate-mediated excitatory connections.

FIG. 2. The Actor–Critic model with possible neurobiologic imple-mentation correlates shown. Corticostriatal connections can have eitherexcitatory or inhibitory influences. The actor generates actions throughthe output units of the GPi/SNr. The critic predicts the primary rein-forcement these actions should generate. The predicted reinforcementis used as a reward signal in strengthening or weakening the cortico-striatal connections.

COMPUTATIONAL MODELS OF THE BASAL GANGLIA 763

Movement Disorders, Vol. 15, No. 5, 2000

they are indeed producing the effective reinforcementsignal.6,7 Properties predicted for dopamine neurons in-clude a negative response when a reward that was pre-dicted fails to occur at the time it was expected.6 Simi-larly, once dopamine response has moved to the earliestof a number of sensory cues that each predict reward,removal of one of the intermediate cues should result ina depression in dopamine response at the expected timeof the cue.7 Both of these predictions have been observedexperimentally.8 These models also make a number ofassumptions. The majority of models of reinforcementlearning assume a temporal representation of sensorycues. Within the cortical representation of the cue, thereis a temporal component allowing the explicit represen-tation of time from cue onset to reward. Such a repre-sentation of cue inputs to the striatum is by no meanscertain. Furthermore, the models require an input of pri-mary reward signal to the nigral dopamine neurons. Theorigin of such a signal is unclear.

Suri and Schultz9 demonstrate the ability of an Actor–Critic model to learnsequencesof pattern associations.For example, stimulus A requires action Q, after whichstimulus B is presented requiring action R, and so on.Reward is only received when the final required action inthe sequence is performed and all preceding action as-sociations in the sequence were correct. The model il-lustrates how the temporal difference algorithm yieldslearning of long patterns of associations. They furtherdemonstrate that replacing the effective reward signal,that is, the prediction of primary reward, with primaryreward itself leads to a failure in learning long sequencesof associations. This illustrates the value of a predictedreward signal in long sequence learning.

Nigral dopamine neurons respond not only to rewards,but also to novel or unexpected stimuli.8 Consequently,within the context of these models, novel stimuli areconsidered to be rewarding events. Unexpected visualstimuli lead to early dopamine responses and saccadiceye movements.10 Redgrave et al.11 argue that becausethe dopamine response occurs before completion of thesaccade to the visual location of the stimulus, such anevent should not be considered a reward before its visualidentification. However, because dopamine neuron re-sponses do not discriminate among different reward mo-dalities, Schultz8 suggests the dopamine response mayrepresent a reward alert signal rather than the rewarditself. It seems likely that the “critic” would need to bebiased in this way to facilitate the associations of newstimuli and the prediction of reward.

There are a small number of reinforcement learningmodels of the basal ganglia that are not based on theTemporal Difference algorithm.12–14These models focuson the basal ganglia circuitry and use a simpler form ofreinforcement learning. Dominey et al.13,14 demonstratethis in a model of the interaction among cortex, basalganglia, and thalamus. The model is considered withrespect to visuospatial tasks requiring saccadic actions.Figure 3 illustrates the circuitry simulated in this model.The system consists of various cortical representations ofthe current visual field. In the model, saccades may becontrolled through a direct projection from the frontaleye fields (FEF) to the superior colliculus (SC). Thebasal ganglia modulates this effect in response to learnedcortical contextual activity. For example, in a conditionalvisual discrimination task in which the model must learnto generate saccades to one of two cues (only one ofwhich is rewarded), the basal ganglia learns through re-inforcement learning to disinhibit only the correct re-sponse represented in the superior colliculus.14 In learn-ing sensorimotor sequences, the model prefrontal cortexarchitecture is designed so that each point in the se-quence holds a unique pattern of activity. This providesa novel context for the striatal neurons to differentiateone item in the sequence from the next.

Reinforcement learning has become a major compo-nent of many contemporary models of the basal ganglia.However, these same models highlight many issues yetto be addressed. For example, there is no clear consensusfor the role of the indirect pathway. The simplicity andelegance of the reinforcement learning paradigm is moreeasily associated with the direct pathways as sites forimplementation. The indirect pathway may be better de-scribed as an indirect network15 within which there is thelikelihood of complex dynamics,16 and its role in rein-forcement learning models remains unclear.

FIG. 3. Schematic diagram indicating the connectivity in the sequencelearning model of Dominey et al.14 Solid lines indicate inhibitoryconnections; dotted lines indicate excitatory connections. The dot–dashline indicates a dopaminergic projection. PP, posterior parietal cortex;FEF, frontal eye fields; PFC, prefrontal cortex; SC, superior colliculus.

A. GILLIES AND G. ARBUTHNOTT764


At first sight, models that emphasize a role for learn-ing in the actions of dopamine in basal ganglia functionargue against the clinical data. What has learning to dowith the obvious neurologic problems of Parkinson’s dis-ease? Surely, no one thinks that not being able to get upfrom an armchair or freezing in the middle of the road isa problem of learning. One solution to this dilemma hasbeen to assign dopamine two roles in the striatum. Onerole is phasic, requiring a burst of activity in nigral cells,perhaps with actions at D1 receptors, being the one in-volved in learning. The other role is tonic, in whichlow-level dopamine is the result of the resting firing levelof dopamine cells.L-Dopa, probably acting more on D2

receptors, replicates this action. The absence of this tonicactivity is likely to be the cause of the neuropathology.This dual functional role of dopamine is implementedexplicitly in the models of Dominey et al.

A more elegant solution is to assume that the presenceof low dopamine levels is a valid reinforcement signal.For example, in the absence of reward presentation, yetwhen the reward is expected (that is, predicted), there isa suppression of dopamine activity. Under the TD inter-pretation, this is an error signal that indicates that lessreward than expected has occurred. Consequently, theactions that led to this will be less prone to be chosen inthe future. Under parkinsonian conditions, such an inter-pretation would label every selection of a corticostriatalset as generating less reward than expected. All actionsbecome less likely to be selected, and eventually thecorrect response is no movement. An error of this kind ismore potent than a simple loss of learning and is con-sistent with TD models.

MODELS OF SERIAL PROCESSING

One of the characteristic anatomic features of the basalganglia is the observation of loops. Cortical-basal ganglia-thalamic-cortical loops have raised the idea of a func-tional role in serial processing. Early neural networkmodels of loops have been used to learn and performserial tasks,17 and they have given rise to some earlymodels of the basal ganglia.18 The possible involvementof the basal ganglia in movement control and planninghas long provided a motivation for predicting serial pro-cessing capabilities. Planned sequences of motor actionsnot only require sequential processing abilities, but alsorepresentations of sequential events. A role in motor op-erations and the nature of basal ganglia loops taken to-gether have provided a powerful motivation for basalganglia models of serial processing.

Like we have seen above, the roles of serial processingand reinforcement learning in the basal ganglia are com-patible.9,14 In the Suri and Schultz9 model, the environ-

ment provided the cue to the next item in the sequence,whereas in the models of Dominey et al.14 an internalrepresentation of a sequence is generated across the pre-frontal cortex. This internal representation can then beused to reproduce the sequence. Beiser and Houk19 pre-sent a model of the direct pathway being used in thegeneration of representations of such temporally serialevents. They propose that the basal ganglia acts as anencoder, processing serial events into a spatial represen-tation of prefrontal cortex activity. Medium spiny neu-rons of the caudate play the role of context detectors.Because each receives up to 10,000 different corticostria-tal afferents,20 this places these striatal neurons in anoptimal position for such a role. The model prefrontalcortex receives (preprocessed) stimuli from the environ-ment resulting in an initial pattern of activity. Lateralinhibition among medium spiny cells in the caudate al-lows only a limited number of medium spiny neurons torespond to this activity pattern. Those that do responddisinhibit thalamic neurons, engaging a sustained recur-rent activity between the thalamus and cortex. The tha-lamic input, together with new stimuli to the prefrontalcortex, modifies its pattern of activity. This new prefron-tal pattern of activity in turn augments the caudate re-sponses. Beiser and Houk demonstrate how this contin-ued looping process builds unique representations of theserial stimuli in the prefrontal cortex.

An alternative to the cortical location for short-termmemory in the production and encoding of sequences isillustrated in a model by Berns and Sejnowski.21 In thismodel, the subthalamic–pallidal loop is proposed as asite for such short-term memory. During learning, thestriatum repeatedly imposes a temporal sequence patternon the GPe (through direct inhibition). As the input to theSTN arises from the GPe and is given significant con-nection delays, the activity pattern across the STN is atleast one time step behind that being imposed on theGPe. Thus, learning associates the previous step in atemporal sequence across the STN with the current stepin the GPe. As the GPe feeds back to the STN, thisallows the learning and generation of sequences. Oncelearned, the activity of the STN (the previous step in thesequence) generates the activity pattern of the currentstep over the GPe and the output units of the globuspallidus internal segment (GPi). The GPe then feeds thiscurrent step onto the STN, thus becoming the previousstep, and the cycle continues. Learning is implementedthrough a reinforcement algorithm allowing the effectsof dopamine to be assessed in a simple manner. Loss ofdopamine is modeled through a global reduction in learn-ing rate, leading to deficits in the sequence learning.



The capacity to generate sequences of activity patternshas also been proposed solely within the architecture ofthe striatum. Wickens and Arbuthnott22 demonstrate thatlateral connectivity within the striatum can be used togenerate spatiotemporal patterns. In their model, inhibi-tion by one medium spiny neuron of another occursthrough axon collaterals among medium spiny cells suf-ficiently close to one another (termed the “domain ofinhibition”). The model demonstrates that with uniformrandom input, usually only one medium spiny neuron isactive within a domain as a result of the mutual inhibi-tion. Furthermore, as a consequence of a slowly increas-ing after-hyperpolarizing conductance, the active neuronmay eventually become inactive, allowing an alternativeneuron within the domain to begin firing. Although thegeneration of these spatiotemporal patterns is stronglydependent on external input, this model demonstrates theability for a lateral inhibitory network to generate localspatiotemporal patterns. The nature of the patterns de-pends on the strength of the after-hyperpolarizing con-ductance. Inclusion of inhibitory interneurons increasesthe stability of the spatiotemporal patterns. This idea hasbeen extended to use the up and down state properties ofmedium spiny neurons23 in a model of basal gangliacontrol of cortical sequential activity.24

Connolly and Burns25–27describe a model that repre-sents matrix compartments within the striatum as a net-work of linked resistors. Anatomically this would as-sume electrotonic coupling between the medium spinyneurons. There is evidence for a weak electronic cou-pling as a result of dye spread to several neurons from

intracellular electrodes.28 However, gap junctions haveprimarily been seen in the electron microscope betweenparvalbumin-positive interneurons.29 In this model, thecorticostriatal input sets up goal and boundary conditionsover the resistance network corresponding to, for ex-ample, target and obstacle positions, respectively, in thereal world. The goal and boundaries are represented byclamped trough and peak voltages, respectively, settingup a voltage landscape. Under the assumption of a directrepresentation of limb position in the striatum, this cre-ates a trajectory of activity following a gradient andavoiding the boundaries (voltage peaks). It is proposedthat the temporal activity sequence generated provides anoptimal movement trajectory within the context of theimposed boundaries. This model depends critically on adirect representation of limb position in the striatum.

Contreras-Vidal and Stelmach30 present a model thatplaces the basal ganglia circuitry directly as a “GO” sig-nal for movement (Fig. 4). Although in this model thebasal ganglia does not generate sequential activity, itplays the role of providing a measured gating signal to atheoretical model that computes the differences betweena target position vector and a present position vector of alimb.31 The output of the model basal ganglia circuitrygates the rate of the integration between target and pres-ent position vectors. As the system captures the influenceof basal ganglia circuitry on the vectors representingchanging limb position, direct comparisons can be madeto experimentally measurable phenomena such as reac-tion time and movement time. This model includes bothdirect and indirect pathways and consequently, the ef-fects that parkinsonism-type conditions have on theseexperimentally measurable parameters can be gauged.The parkinsonian conditions simulated follow that de-scribed by Albin et al.,32 in which striatal activity alongthe direct pathway (that is, the pathway associated withsubstance P) is diminished and striatal activity along theindirect pathway (that is, associated with enkephalin) isenhanced. Under these conditions, the model replicatescertain parkinsonism phenomena. For example, as dopa-mine levels decrease, both task reaction time and move-ment time increase. For low levels of dopamine, themodel fails to gate all movements and akinesia-type be-havior results. The model thus demonstrates a singlemechanism for both bradykinesia and akinesia.

Because the mechanisms underlying these variousmodels of serial processing are different, so too are theirdescriptions of how basal ganglia disorders influenceprocessing. For example, Wickens et al.33 explore theeffects of dopamine on the activity within domains ofinhibition in the striatum as described above. By model-ing particular channel conductances within striatal me-

FIG. 4. Schematic diagram indicating the connectivity in the model ofContreras-Vidal and Stelmach.30 The thalamic unit plays the role of agate in a “vector integration model” which determines the speed ofchange from limb present position to target position. Solid lines indi-cate an excitatory connection; dotted lines indicate an inhibitoryconnection.



dium spiny neurons and incorporating the influence ofthe large aspiny cholinergic interneurons, Wickens et al.demonstrate a dramatic change in the functioning of thedomains of inhibition under conditions of low dopamine.As described above, a single neuron within such a do-main is likely to be active as a result of competition fromthe inhibitory lateral interactions. Under conditions ofreduced levels of dopamine, the model demonstrates co-activation of neurons within a domain. It is proposed thatadjacent neural populations may represent antagonisticmuscles and consequently, coactivation may be associ-ated with symptoms such as rigidity.

MODELS OF ACTION SELECTION

The major output nuclei of the basal ganglia (GPi,substantia nigra pars reticulata [SNr]) exhibit a tonic ac-tivity that exerts an inhibitory influence over their tar-gets. Because activity along the direct pathway leads todisinhibition of these targets, it has been proposed thatthis allows the selection of specific motor programs.34,35

A focused disinhibition surrounded by inhibition on tar-gets would allow this process to be finely tuned. In thistheoretical model, the tonic inhibition can be maintainedor enhanced through the STN by way of the indirectpathway providing widespread excitation to the outputnuclei.35 Such an elegant center-surround model of basalganglia processing has provided motivation for the roleof the basal ganglia in selection of actions or motor pro-grams. This idea has been extended by Redgrave etal.11,36,37so that the basal ganglia may be considered ageneralresource selection mechanism.

The arguments for such a centralized resource selec-tion system are posited from an engineering efficiencyperspective. A central system saves the need for combi-natorially massive connections between each competingresource. It simply requires each resource area to connectto the central mechanism. The projections from wide-spread cortical areas to the striatum certainly encouragesuch a view. Figure 5 illustrates the central selectionmodel presented by Gurney et al.36 within the context ofthe basal ganglia circuitry. Cortical input to the striatumis described in terms of “channels,” where each channelrefers to a competing resource or action, and originatesfrom the (cortical) neural substrate responsible for pro-cessing it. Channels are assumed to carry salience infor-mation so that at the striatal level, competition amongmedium spiny neurons is determined by salience. Themodel is split into selection and control mechanisms. Thedirect pathway in cooperation with signals from the STNmediates selection. Local competition within the modelstriatum solveslocal resource conflicts leaving a singlechannel active. For example, under a somatotopic repre-

sentation this may resolve conflicts for a common motorresource. The activity of the local winning active channelrepresents that channel’s salience. Consequently, outputfrom the striatum can be used inglobal resource com-petition. The direct projection from the striatum to theoutput nuclei (GPi, SNr) inhibits the channel zones inthese nuclei in proportion to the saliences of the com-peting channels. The STN provides a diffuse and uniformexcitation to channel zones over the output nuclei. Thisdiffuse excitation from the STN combined with inhibi-tion from the GPe scales the activity of the output nucleiso that only the channels with the highest saliences re-main sufficiently depressed to disinhibit the targets andthus perform selection. Within the control pathway, feed-back from the GPe to the STN scales STN activity mak-ing it independent of the number of competingchannels.

The effects of dopamine are included in the modelthrough the division of striatum into medium spiny neu-rons with D1 or D2 receptors. In agreement with Albin etal.,32 dopamine inhibits the D2-mediated pathway andfacilitates the D1-mediated pathway. Thus, a reduction ofdopamine enhances the control mechanisms and sup-presses the selection mechanisms. As the level of dopa-mine is reduced in the simulations, the possibility ofmultiple channel selection also reduces. If the selectionof multiple channels corresponds to the selection andoperation of two or more simultaneous behaviors, thenthe effects of reduced dopamine in the model is consis-tent with the phenomenon of “loss of associated move-ments” in early Parkinson’s disease. Low levels of do-

FIG. 5. Schematic diagram indicating the connectivity in the model ofGurney et al.36 Solid lines indicate an inhibitory connection; dottedlines indicate an excitatory connection.



pamine in the model prevent selection of any action orresource. This has been proposed as being consistentwith akinesia. As dopamine levels increase, so do thepossibilities for several anomalies. In addition to the cor-rect selection of the highest salient channels, in themodel with intermediate dopamine levels there is a pos-sibility that two highly salient channels cause each otherto fail to be selected. This results in neither channel beingchosen. In the conditions of high levels of dopamine,many competing channels with high salience may beselected. These behaviors can be undesirable in a selec-tion system. Although these anomalies may be looselyassociated with certain behavioral conditions (for ex-ample, possibly schizophrenia), their interpretation asexperimentally predictable behavior is still unclear.

DISCUSSION

The models reviewed range not only in their func-tional descriptions of the basal ganglia, but also in theirmethod of description. For example, in the models ofDominey et al.,13,14 a single neuron is described as acontinuous time differential equation, whereas in the Suriand Schultz9 model, a single neuron is a summation unitand time is discrete. Modeling a single neuron as asimple differential equation or as a logic function im-poses limitations on how the resulting network modelsshould be interpreted. For example, as mentioned above,depletion of dopamine is modeled simplistically. Withone exception, none of the models capture the effects ofdopamine on striatal cells. Consequently, these modelshave limited informational value at this level. It is in-creasingly possible to model neurons at more detailedbiochemical levels; however, this is only in the earlystages of being extended into networks of cells. As an-other example, all of the models include pathways be-tween structures without recourse to the known anatomicvariations within these pathways such as preserved to-pology.38 Thus, the brush strokes with which the modelscurrently describe basal ganglia function are broad.Nonetheless, they provide a unique framework on whichto build.

The three themes of models presented here share muchin common. Collectively, there is an emphasis on theimportance of learning, in particular, reinforcementlearning, to the function of the basal ganglia. Reinforce-ment learning is not only posited as a function in itself,but is also consistent with models of the serial processingand action selection themes. Learning through rewardplays a key role in the development of context detectionor selection of cortical activity. Cortical activity relatedto a future increase in reward is hypothesized to evoke astronger striatal response on subsequent occurrences.

This stronger response may influence its chance of be-havioral selection or enhance the original cortical activ-ity in the production of learned cortical sequences. How-ever, the mechanisms of learning, such as the represen-tation of error in reward prediction in dopamineresponses, are still debated.

At a functional level, sequencing and action selectionmodels are compatible. It is possible that the selection ofcortical activity would also involve its modification, con-sequently influencing its future selection. This may sta-bilize selections as suggested by Redgrave et al.37 oralternatively allow selection of the appropriate next be-havioral set in a sequence.

These extensions of ideas extracted from the modelshave the same limitations as the models themselves. Notonly are the models pitched at a high level as describedabove, but they also make implicit or explicit assump-tions yet to be experimentally verified. The majority ofmodels rely primarily on the direct pathway. Thecomplexities of the indirect pathway and the currentshifting of our understanding of its anatomy and physi-ology make it a difficult system to model. However,it is clear that the indirect pathway plays a crucial rolein basal ganglia function, and it is important that thisbe increasingly addressed in basal ganglia models.Another common feature of many of the models is theassumption of lateral inhibition within the striatum. Allthe current models rely to a greater or lesser extent on theinterconnectivity of the striatal neurons. Although ana-tomic data supports this,39 there is no physiological con-firmation of mutually inhibitory connections betweencells in the neostriatum.40 Other aspects of the anatomicpathways underlying many of the models (see Fig. 1) aresimilarly subject to criticism. For example, there is in-creasing evidence for multiple corticostriatal path-ways20,41,42 in which functional significance is not yetaddressed in any of the models. In addition, the interpre-tation of the loss of dopamine in Parkinson’s disease, asdescribed by Albin et al.,32 is also still under active de-bate.

However abstract, all the models reviewed have im-portant ideas to convey. Although the physical modelshave many operational inconsistencies with each other,they provide a framework within which basal gangliafunctions are beginning to be structured. The develop-ment of models needs to be extended to lower levels. Forexample, only the models of Wickens et al.33 capturedetailed anatomy within a single nuclei. Recent modelsof the STN and GPe16,43have demonstrated that the ana-tomic arrangements within a nucleus can be crucial tohow the nucleus operates within the basal ganglia as awhole. Models of lateral connectivity within the STN



demonstrate that this can functionally lead to widespreadswitch-like behavior across the nucleus. This has impor-tant consequences for how other basal ganglia nucleimay operate in this context. For example, it has furtherbeen shown that under parkinsonian conditions, as de-scribed by Albin et al., this switch-like behavior in theSTN can lead to continuous oscillations of bursting-likeactivity in both the STN and GPe.16 These models dem-onstrate not only functional constraints on STN opera-tion, but also the importance of considering multiple lev-els in model development. It is an exciting prospect thatmodeling will continue to provide increasingly detailedframeworks within which the full extent of basal gangliafunction may eventually emerge.

REFERENCES

1. Barto A. Adaptive critics and the basal ganglia. In: Houk J, DavisJ, Beiser D, eds.Models of Information Processing in the BasalGanglia.Cambridge, MA: MIT Press, 1995:215–232.

2. Sutton R, Barto A.Reinforcement Learning: An Introduction.Cambridge, MA: MIT Press, 1998.

3. Hollerman J, Schultz W. Dopamine neurons report an error in thetemporal prediction of reward during learning.Nature Neurosci-ence1998;1:304–309.

4. Mirenowicz J, Schultz W. Importance of unpredictability for re-ward responses in primate dopamine neurons.J Neurophysiol1994;72:1024–1027.

5. Houk J, Adams J, Barto A. A model of how the basal gangliagenerate and use neural signals that predict reinforcement. In:Houk J, Davis J, Beiser D, eds.Models of Information Processingin the Basal Ganglia.Cambridge, MA: MIT Press, 1995:249–270.

6. Montague PR, Dayan D, Sejnowski T. A framework for mesence-phalic dopamine systems based on predictive hebbian learning.JNeurosci1996;16:1936–1947.

7. Schultz W, Dayan P, Montague R. A neural substrate of predictionand reward.Science1997;275:1593–1599.

8. Schultz W. Predictive reward signal of dopamine neurons.J Neu-rophysiol1998;80:1–27.

9. Suri R, Schultz W. Learning of sequential movements by neuralnetwork model with dopamine-like reinforcement signal.ExpBrain Res1998;121:350–354.

10. Ljungberg T, Apicella P, Schultz W. Responses of monkey dopa-mine neurons during learning of behavioral reactions.J Neuro-physiol1992;67:145–163.

11. Redgrave P, Prescott T, Gurney K. Is the short-latency dopamineresponse too short to signal reward error?Trends Neurosci1999b;122:145–151.

12. Arbib M, Dominey P. Modeling the roles of basal ganglia in timingand sequencing saccadic eye movements. In: Houk J, Davis J,Beiser D, eds.Models of Information Processing in the BasalGanglia.Cambridge, MA: MIT Press, 1995:149–162.

13. Dominey P. Complex sensory-motor sequence learning based onrecurrent state representation and reinforcement learning.Biol Cy-bern 1995;73:265–274.

14. Dominey P, Arbib M, Joseph J. A model of corticostriatal plastic-ity for learning oculomotor associations and sequences.J Cogni-tive Neuroscience1995;7:311–336.

15. Parent A, Cicchetti F. The current model of basal ganglia organi-sation under scrutiny.Mov Disord1998;13:199–202.

16. Gillies A, Willshaw D. A model of the interaction between globuspallidus and subthalamic nucleus.Eur J Neurosci1998b;10(suppl10):301.

17. Jordon M. Attractor dynamics and parallelism in a connectionist

sequential machine. In:Proceedings of the 8th Annual Conferenceof the Cognitive Science Society.Hillsdale, NJ: Erlbaum, 1986:531–546.

18. Brotchie P, Iansek R, Horne M. A neural network model of neuralactivity in the monkey globus pallidus.Neurosci Lett1991;131:33–36.

19. Beiser D, Houk J. Model of cortical-basal ganglia ganglionic pro-cessing: encoding the serial order of sensory events.J Neuro-physiol1998;79:3168–3188.

20. Kindaid A, Zheng T, Wilson C. Connectivity and convergence ofsingle corticostriatal axons.J Neurosci1998;18:4722–4731.

21. Berns G, Sejnowski T. A computational model of how the basalganglia produce sequences.J Cognitive Neuroscience1998;10:108–121.

22. Wickens J, Arbuthnott G. The corticostriatal system on computersimulation: an intermediate mechanism for sequencing of actions.In: Arbuthnott G, Emson P, eds.Progress in Brain Research,vol99. Oxford: Elsevier Science, 1993.

23. Wickens J, Wilson C. Regulation of action-potential firing in spinyneurons of the rat neostriatum in vivo.J Neurophysiol1998;79:2358–2364.

24. Plenz DJW, Kitai S. Basal ganglia control of sequential activity inthe cerebral cortex: a model. In: Bower J, ed.Computational Neu-roscience.New York, NY: Academic Press, 1996:397–402.

25. Connolly C, Burns J. A model for the functioning of the striatum.Biol Cybern1993a;68:535–544.

26. Connolly C, Burns J. A new model and its relationship to basalganglia diseases.Neurosci Res1993b;16:271–274.

27. Connolly C, Burns J. Reward-related signals carried by dopamineneurons. In: Houk J, Davis J, Beiser D, eds.Models of InformationProcessing in the Basal Ganglia.Cambridge, MA: MIT Press,1995:163–183.

28. Onn S, Grace A. Alterations in electrophysiological activity anddye coupling of striatal spiny and aspiny neurons in dopamine-denervated rat striatum recorded in vivo.Synapse1999;33:1–15.

29. Kita H, Kosaka T, Heizmann C. Parvalbumin-immunoreactiveneurons in the rat neostriatum—a light and electron-microscopicstudy.Brain Res1990;536:1–15.

30. Contreras-Vidal J, Stelmach G. A neural model of basal ganglia-thalamocortical relations in normal and parkinsonian movement.Biol Cybern1995;73:467–476.

31. Bullock D, Grossberg S. Neural dynamics of planned arm move-ments: emergent invariants and speed accuracy properties duringtrajectory formation.Psychol Rev1988;95:49–90.

32. Albin R, Young A, Penny J. Functional anatomy of basal gangliadisorders.Trends Neurosci1989;12:366–375.

33. Wickens J, Alexander M, Miller R. Two dynamic modes of striatalfunction under dopaminergic/cholinergic control: simulation andanalysis of a model.Synapse1991;8:1–12.

34. Chevalier G, Deniau J. Disinhibition as a basic process in theexpression of striatal functions.Trends Neurosci1990;13:277–280.

35. Mink J, Thach W. Basal ganglia intrinsic circuits and their role inbehavior.Curr Opin Neurobiol1993;3:950–957.

36. Gurney K, Prescott T, Redgrave P. A computational model ofaction selection in the basal ganglia: a new functional anatomy.1999. In preparation.

37. Redgrave P, Prescott T, Gurney K. The basal ganglia: a vertebratesolution to the selection problem?Neuroscience1999a;89:1009–1023.

38. Joel D, Weiner I. The connections of the primate subthalamicnucleus: indirect pathways and the open-interconnected scheme of



basal ganglia-thalamocortical circuity.Brain Res Rev1997;23:62–78.

39. Yung K, Smith A, Levey A, Bolam J. Synaptic connections be-tween spiny neurons of the direct and indirect pathways in theneostriatum of rat: evidence from dopamine receptor and immu-nostaining.Eur J Neurosci1996;8:861–869.

40. Jaeger D, Kita H, Wilson C. Surround inhibition among projectionneurons is weak or nonexistent in the rat neostriatum.J Neuro-physiol1994;72:2555–2558.

41. Kindaid A, Wilson C. Corticostriatal innervation of the patch andmatrix in the rat neostriatum.J Comp Neurol1996;374:578–592.

42. Wright A, Norrie L, Ingham C, Hutton E, Arbuthnott G. Doubleanterograde tracing of outputs from adjacent ‘barrel columns’ ofrat somatosensory cortex, neostriatal projection patterns and ter-minal ultrastructure.Neuroscience1999;88:119–133.

43. Gillies A, Willshaw D. A massively connected subthalamicnucleus leads to the generation of widespread pulses.Proc R SocLond B1998a;265:2101–2109.



computational models of the basal ganglia

Documents