banishing the homunculus: making working memory work

B

T

DB

Asbrcotaoombtmacomalbaoeo

Kfm

Tlpwrtaadn

*EAAdsmeacnW

Neuroscience 139 (2006) 105–118

0d

ANISHING THE HOMUNCULUS: MAKING WORKING MEMORY WORK

ntwws

bturc(mmadwitswtPp

B

Onbhntbwft

1

2

. E. HAZY, M. J. FRANK AND R. C. O’REILLY*

epartment of Psychology, University of Colorado Boulder, 345 UCB,oulder, CO 80309, USA

bstract—The prefrontal cortex has long been thought toubserve both working memory and “executive” function,ut the mechanistic basis of their integrated function hasemained poorly understood, often amounting to a homun-ulus. This paper reviews the progress in our laboratory andthers pursuing a long-term research agenda to deconstructhis homunculus by elucidating the precise computationalnd neural mechanisms underlying these phenomena. Weutline six key functional demands underlying working mem-ry, and then describe the current state of our computationalodel of the prefrontal cortex and associated systems in theasal ganglia (BG). The model, called PBWM (prefrontal cor-ex, basal ganglia working memory model), relies on activelyaintained representations in the prefrontal cortex, which

re dynamically updated/gated by the basal ganglia. It isapable of developing human-like performance largely on itswn by taking advantage of powerful reinforcement learningechanisms, based on the midbrain dopaminergic system

nd its activation via the basal ganglia and amygdala. Theseearning mechanisms enable the model to learn to controloth itself and other brain areas in a strategic, task-appropri-te manner. The model can learn challenging working mem-ry tasks, and has been corroborated by several importantmpirical studies. © 2005 Published by Elsevier Ltd on behalff IBRO.

ey words: basal ganglia, prefrontal cortex, dopamine, rein-orcement learning, Pavlovian conditioning, computationalodeling.

his article reviews an ongoing research agenda in ouraboratory and others that is attempting to elucidate therecise computational and neural mechanisms underlyingorking memory and “executive” function. Our approach

epresents an attempt to understand these phenomena inerms of a set of biologically based, computational mech-nisms. This approach has resulted in the identification ofcore set of six functional demands that collectively help

efine the fundamental nature of working memory from aeuro-mechanistic perspective. Our proposed mecha-

Corresponding author.-mail address: [email protected] (R. C. O’Reilly).bbreviations: ADHD, attention deficit hyperactivity disorder; AX-CPT,X version of continuous performance task; BG, basal ganglia; DA,opamine; GPe, globus pallidus, external segment; HC, hippocampalystem; ID/ED, intradimensional/extradimensional task; MT, multi-taskodel; PBWM, prefrontal cortex, basal ganglia working memory mod-l; PC, posterior cortex; PFC, prefrontal cortex; PVLV, primary valuend learned value; SIR, store/ignore/recall; SNc, substantia nigra parsompacta; SNr, substantia nigra pars reticulata; STN, subthalamic
ucleus; TD, temporal difference; VTA, ventral tegmental area; WCST,isconsin card sort task.
306-4522/06$30.00�0.00 © 2005 Published by Elsevier Ltd on behalf of IBRO.oi:10.1016/j.neuroscience.2005.04.067

105

isms for dealing with these functional demands collec-ively explain many of the same phenomena as traditionalorking memory constructs, but in a manner that contrastsith them in important ways and does so in a comprehen-ive, integrated way.

The overall format for the article is as follows. After arief introduction of our approach to working memory inerms of developing a biologically-based architecture fornderstanding human cognition, we then describe the cur-ent version of our computational model of the prefrontalortex (PFC) and basal ganglia (BG) in working memoryPBWM, prefrontal cortex, basal ganglia working memoryodel), with special emphasis on six key functional de-ands underlying working memory. To try to make thingss transparent as possible, we describe both the functionalemands and the model itself in the context of a concreteorking memory task. We then review some of the empir-

cal data that have tested predictions of our model, andhen outline our research trajectory that is attempting toimulate many of the most important task paradigms oforking memory and executive function in a single instan-

iation of a comprehensive model built around the coreBWM mechanisms. Finally, we discuss some overall im-lications for future work.

iologically-based cognitive architecture

ur PBWM working memory model is motivated by aumber of considerations derived from an overarchingiologically-based cognitive architecture for understandinguman cognition (Atallah et al., 2004; O’Reilly and Mu-akata, 2000). This tripartite architecture is composed ofhree functionally complementary brain systems that cane understood in terms of a set of computational tradeoffs,hich provide a more precise and often subtle set of

unctional properties for these areas (Fig. 1). These sys-ems are as follows:

. The posterior cortex (PC) system that performs thevast majority of the “automatic” sensory and motorprocessing in the brain. This system exhibits slow,integrative learning that extracts the long-term statisti-cal structure of the environment, thereby efficientlyrepresenting accumulated knowledge and skills. In asense, the PC system provides the “substrate” uponwhich the other two (“higher level”) systems operate toproduce working memory phenomena.

. The hippocampal system (HC) system that is special-ized for rapid (e.g. one trial) learning that binds to-gether arbitrary information, which can be subse-quently recalled in the service of controlled processing.The neural specializations required for this rapid learn-
ing without interference are incompatible with the inte-

3

pmwnmTripa2tNamli

Ott2i

wti((apmtmpmpebeipr

dt(aepmntepthdntatft

tortottata

FcFbicHimFtpt

T. E. Hazy et al. / Neuroscience 139 (2006) 105–118106

grative statistical learning of the posterior cortical sys-tem, thus motivating the need for two separate neuralsystems.

. The PFC/BG system that is specialized for active main-tenance of internal contextual information (PFC), whichcan be dynamically updated by the BG. This systemcan bias (control) ongoing processing throughout thecortex according to actively maintained information inthe PFC (e.g. goals, instructions, partial products). Itincludes a dopamine (DA)-based learning system thatuses a version of reinforcement learning mechanismsthat are widely thought to be supported by the BG andrelated mid-brain structures.

Because it represents the canonical form of corticalrocessing, our model of the PC lies at the foundation ofuch of our work. It is based upon a computational frame-ork called Leabra (O’Reilly, 1998; O’Reilly and Mu-akata, 2000; O’Reilly, 2001) that contains all of the basicechanisms and properties required by our models.hese mechanisms have mostly been developed sepa-ately by other researchers over many years. Our special-zed model of the HC has been developed over numerousublications (O’Reilly and McClelland, 1994; McClelland etl., 1995; O’Reilly and Rudy, 2001; Norman and O’Reilley,003; O’Reilly and Norman, 2002), and also includes manyhemes going back to earlier work (e.g., Marr, 1971; Mc-aughton and Morris, 1987; Rolls, 1989) that are widelydopted in the literature. Our hippocampal model (imple-ented in the Leabra framework) has been used to simu-

ate a wide range of learning and memory phenomena,

ig. 1. Tripartite cognitive architecture defined in terms of differentomputational tradeoffs associated with PC, Hippocampus (HC) androntal Cortex/BG (FC/BG) (with motor frontal cortex constituting alend between FC and PC specializations). Large overlapping circles

n PC represent overlapping distributed representations used to en-ode semantic and perceptual information. Small separated circles inC represent sparse, pattern-separated representations used to rap-

dly encode (“bind”) entire patterns of information across cortex whileinimizing interference. Isolated, self-connected representations inC represent isolated stripes (columns) of neurons capable of sus-

ained firing (i.e. active maintenance or working memory). The BG alsolay a critical role in the FC system by modulating (“gating”) activationshere based on learned reinforcement history.

ncluding human recognition memory (Norman and b

’Reilly, 2003), and animal learning paradigms (e.g. con-extual fear conditioning, nonlinear discrimination learning,ransitive inference) (O’Reilly and Rudy, 2001; Frank et al.,003). Our models of the PFC/BG system are elaborated

n the remainder of this paper.In the context of this overall architecture, we can define

orking memory as an emergent property of the interac-ions between these three specialized brain areas, involv-ng both active maintenance of task-relevant informationPFC/BG), and rapid learning of arbitrary associationsHC). These mechanisms support basic memory functionsssociated with working memory (i.e. memory for partialroducts of ongoing processing, task goals, etc.), but alsoore complex controlled processing functions that are

ypically ascribed to a “central executive” in other workingemory frameworks (e.g., Baddeley, 1986). Controlledrocessing emerges from the biasing influence of activelyaintained and updated PFC representations on otherarts of the system (e.g., Cohen et al., 1990, 1996; O’Reillyt al., 1999; Miller and Cohen, 2001). This “top-down”iasing (which can also be supported by the HC to somextent) supports the performance of task-relevant process-

ng in the face of competition from habitual or more well-racticed forms of processing that may not be taskelevant.

This contrast between theoretical models that explicitlyistinguish between working memory and executive func-ion, typically designating them as separate “modules”e.g., Baddeley, 1986), and our more emergent, interactivepproach, is critical. Our view that working memory andxecutive function are really two sides of the same coinrovides a more parsimonious model, that we believe isore consistent with extant biological data on the nature ofeural specializations in the PFC/BG system. These con-rasting approaches may be due in part to traditional mod-ls relying on a computer-like mental architecture, whererocessing is centralized and long-term memory is essen-ially a passive store. For those models, it makes sense toave a separate set of cache-like working memory buffersedicated to the temporary storage of items that areeeded during processing by a completely separate cen-ral executive module (Baddeley, 1986). However, theserchitectures do not correspond to the known micro-archi-ecture of the brain, in which processing and memoryunctions are typically distributed within and performed byhe very same neural substrates (Rumelhart et al., 1986).

Thus, instead of thinking about the moving of informa-ion from long-term memory into and out of working mem-ry buffers, we think that information is distributed in aelatively stable configuration throughout the cortex, andhat working memory amounts to the controlled activationf these representations. This view shares some similari-ies with the view of working memory offered by produc-ion-system accounts (e.g. ACT, Anderson, 1983; Lovett etl., 1999). However, it does not include the structural dis-inction between declarative and procedural knowledgessumed by such accounts.

To summarize then, our contention is that activation-
ased working memory is best thought of as the primary

mtdbkat

frtmtwToaatm

T

BmttTsTs(t

h1opstecwrsstvsi11

2tdssTa

wm

Sm

Wt

1

2

3

4

5

Fstlw(mlt

T. E. Hazy et al. / Neuroscience 139 (2006) 105–118 107

echanism behind the PFC/BG system, and that con-rolled processing is a resulting emergent function thatepends critically on this mechanism. Furthermore, weelieve it is likely that working memory may represent aind of phylogenetic extension of the same kinds of mech-nisms that underlie all forms of complex motor coordina-ion and planning.

This conceptualization has important implications fororming theories and building models. While it certainlyemains perfectly valid (as well as heuristically productive)o formulate theories to explain a narrow subset of workingemory phenomena, in doing so one runs the distinct risk

hat those theories will be inherently incapable of dealingith other, equally important phenomena in the domain.hus, any given theoretic model might best be judged notnly by its fit to the data it was formulated for, but also by itsbility to account for other working memory phenomena—ort least its potential to do so. In other words, we believe

hat a truly robust theory of working memory is best for-ulated with the big picture in mind.

he PBWM model of working memory

ased on our cumulative work on a wide variety of workingemory tasks, we have identified a core set of six func-

ional demands, enumerated below, that are required byasks involving working memory and executive function.aken together, these functional demands provide a basicet of constraints for our biologically-based PBWM model.he 1-2-AX task, which is an extension of the widely-tudied AX version of the continuous performance taskAX-CPT), provides a nice demonstration for these informa-ion-processing demands on the working memory system.

The AX-CPT is a standard working memory task thatas been extensively studied in humans (Cohen et al.,997; Braver and Cohen, 2000; Braver et al., unpublishedbservations; Braver et al., 1999; Frank and O’Reilly, un-ublished observations). The subject is presented withequential letter stimuli (A, X, B, Y), and is asked to detecthe specific sequence of an A followed on the very nextvent by an X, by pushing the target (right) button. All otherombinations (A–Y, B–X, B–Y) should be responded toith a non-target (left) button push. This task requires a

elatively simple form of working memory, where the priortimulus must be maintained over a delay until the nexttimulus appears, so that the subject can discriminate thearget from non-target sequences. This is the kind of acti-ation-based working memory that has often been ob-erved for example in electrophysiological studies of work-ng memory in monkeys (e.g. Fuster and Alexander,971; Kubota and Niki, 1971; Miyashita and Chang,988; Funahashi et al., 1989; Miller et al., 1996).

In the 1-2 extension of the AX-CPT task (1-2-AX; Fig.; Frank et al., 2001; Kroger et al., unpublished observa-ions), the target sequence varies depending on prior taskemand stimuli (a 1 or 2). Specifically, if the subject lastaw a 1, then the target sequence is A–X. However, if theubject last saw a 2, then the target sequence is B–Y.hus, the task demand stimuli define an outer loop of
ctive maintenance (maintenance of task demands) within
hich there can be any number of inner loops of activeaintenance for the A–X level sequences.

ix key functional demands underlying workingemory

ith the 1-2-AX task as a concrete example, the six func-ional demands upon the working memory system are:

. Rapid updating: The working memory system shouldbe able to rapidly encode and maintain new informa-tion as it occurs. In the 1-2-AX task, as each relevantstimulus is presented, it must be rapidly encoded inworking memory.

. Robust maintenance: Information that remains rele-vant should be maintained in the face of the interfer-ence from ongoing processing or other stimulus inputs.In the 1-2-AX task, the task demand stimuli (1 or 2) inthe outer loop must be maintained in the face of theinterference from ongoing processing of inner loopstimuli and irrelevant distractors. Also, a specific A or Bmust also be maintained for the duration of each innerloop.

. Multiple, separate working memory representations:To maintain the outer loop stimuli (1 or 2) while updat-ing the inner loop stimuli (A or B), these two sets ofrepresentations must be distinct within the PFC (i.e.they must not be in direct mutual competition with oneanother, such that only one such representation couldbe active at a time).

. Selective updating: Only some elements of workingmemory should be updated at any given time, whileothers are maintained. For example, in the inner loop,A or B should be updated while the task demandstimulus (1 or 2) is maintained.

. Top-down biasing of processing: For working memoryrepresentations to achieve controlled processing, theymust be able to bias (control) processing elsewhere inthe brain. For example, whichever outer loop stimulus(1 or 2) is active at a given time must bias processing

ig. 2. The 1-2-AX task. Stimuli are presented one at a time in aequence. The participant responds by pressing the right key (R) tohe target sequence, otherwise a left key (L) is pressed. If the subjectast saw a 1, then the target sequence is an A followed by an X. If a 2as last seen, then the target is a B followed by a Y. Distractor stimuli

e.g. 3, C, Z) may be presented at any point and are to be ignored. Theaintenance of the task stimuli (1 or 2) constitutes a temporal outer-

oop around multiple inner-loop memory updates required to detect thearget sequence.

in the PFC/BG system itself, to condition responses

6

topsv

nPa2bsh

D

IuwpftC2ucaf

pmBordtrrt

FiAit

FwiwT


and working memory updates as a function of thecurrent target sequence.

. Learning what and when to gate: Underlying all suc-cessful working memory task performance is the needto learn when to gate appropriately. This is a challeng-ing problem because the benefits of having gatedsomething in are only available later in time (e.g. en-coding the one task demand stimulus only affects overtbehavior and error-feedback later when confrontedwith an A–X sequence).

Earlier computational work by our group of collabora-ors has instantiated and validated a number of aspects ofur overall theory, including the graded nature of controlledrocessing (Cohen et al., 1990); the ability of PFC repre-entations to bias subsequent processing (Cohen and Ser-an-Schreiber, 1992); the role of PFC in active mainte-

ig. 3. Illustration of active gating. When the gate is open, sensorynput can rapidly update working memory (e.g. encoding the cue item

in the 1-2-AX task), but when it is closed, it cannot, thereby prevent-ng other distracting information (e.g. distractor C) from interfering withhe maintenance of previously stored information.

ig. 4. The BG are interconnected with frontal cortex through a series ohich is bidirectionally excitatory with frontal cortex, the SNr is tonically

n dorsal striatum fire, they inhibit the SNr, and thus disinhibit frontal c
orking memory representations in PFC. The indirect pathway “NoGo” neuronshe STN provides an additional dynamic background of inhibition (NoGo) by e
ance (Braver et al., 1997); the ability of the BG to updateFC working memory representations (Frank et al., 2001);nd the role of the HC in rapid learning (O’Reilly and Rudy,001; Norman and O’Reilly, 2003). Most recently, we haveeen focused on the role of the PFC/BG system, and mostpecifically, how, mechanistically, it can learn to do what itas to do to support working memory.

ynamic updating via BG gating

t is important to note that the first two demands (rapidpdating versus robust maintenance) are in direct conflictith each other when viewed in terms of standard neuralrocessing mechanisms. This motivates the obvious needor a dynamic gating mechanism to switch between thesewo modes of operation (Cohen et al., 1996; Braver andohen, 2000; O’Reilly et al., 1999; O’Reilly and Munakata,000). When the gate is open, working memory can getpdated by incoming stimulus information; when it islosed, currently active working memory representationsre robustly maintained even in the face of potential inter-erence as from intervening distractor stimuli (Fig. 3).

A central feature of our PBWM model is that the BGrovide this requisite dynamic gating mechanism for infor-ation maintained via sustained activation in the PFC. TheG are interconnected with frontal cortex through a seriesf parallel loops (Fig. 4). When direct pathway “Go” neu-ons in dorsal striatum fire, they inhibit the SNr, and thusisinhibit frontal cortex, producing a gating-like modulationhat we argue triggers the update of working memoryepresentations in PFC. The indirect pathway “NoGo” neu-ons of dorsal striatum counteract this effect by inhibitinghe inhibitory GPe (globus pallidus, external segment). The

l loops, each of the form shown. Working backward from the thalamus,nd inhibiting this excitatory circuit. When direct pathway “Go” neuronsducing a gating-like modulation that we argue triggers the update of

f paralleactive a

ortex, pro
of dorsal striatum counteract this effect by inhibiting the inhibitory GPe.xciting the SNr.

SbrammW1111raaru

1

2

3

4

5

6

m(tsp(vmWsmr

L

O

FsosnPrab


TN (subthalamic nucleus) provides an additional dynamicackground of inhibition (NoGo) by exciting the SNr. Aseviewed in Frank et al., 2001, this idea is consistent with

wide range of empirical data and other computationalodels that have been developed largely in the domain ofotor control, but also for working memory as well (e.g.ickens, 1993; Houk and Wise, 1995; Wickens et al.,

995; Dominey et al., 1995; Berns and Sejnowski, 1995,998; Jackson and Houghton, 1995; Beiser and Houk,998; Nakahara and Doya, 1998; Kropotov and Etlinger,999; Amos, 2000; Nakahara et al., 2001). Our ideasegarding just how the PFC and BG might work together toccomplish this complex coordination are outlined below,long with a brief description of the specific biologically-ealistic computational mechanisms that our PBWM modelses to instantiate them.

. Rapid updating occurs when direct pathway spiny neu-rons in the dorsal striatum fire (Go units). Go firingdirectly inhibits the substantia nigra pars reticulata(SNr), and releases its tonic inhibition of the thalamus.This thalamic disinhibition enables, but does not di-rectly cause (i.e. gates), a loop of excitation into thecorresponding PFC “stripe” (see Multiple, separateworking memory representations below). The effect ofthis net excitation is to toggle the state of bistablecurrents in the PFC neurons. Striatal Go neurons in thedirect pathway are in competition (downstream in theSNr, if not actually in the striatum; Mink, 1996; Wick-ens, 1993) with a corresponding NoGo (indirect) path-way that promotes greater inhibition of thalamic neu-rons, thereby working to block gating.

. Robust maintenance occurs via two intrinsic PFCmechanisms: 1) recurrent excitatory connectivity(O’Reilly et al., 1999; O’Reilly and Munakata, 2000),and; 2) bistability (Fellous et al., 1998; Wang, 1999;Durstewitz et al., 1999; Durstewitz et al., 2000), thelatter of which is toggled between a maintenance stateand a non-maintenance state by the Go gating signalfrom the BG.

. Multiple, separate working representations are possi-ble because of the “striped” micro-anatomy of the PFC,which is characterized by small, relatively isolatedgroups of interconnected neurons, thereby preventingundo interference between representations in different(even nearby) stripes (Levitt et al, 1993; Pucak et al.,1996). We estimate there may be as many as 20,000such stripes in human PFC (Frank et al., 2001).

. Selective updating occurs because of the existence ofindependently updatable parallel loops of connectivitythrough different areas of the BG and frontal cortex(Alexander et al., 1986; Graybiel and Kimura, 1995;Middleton and Strick, 2000). We hypothesize thatthese loops are selective to the relatively fine-grainedlevel of the anatomical stripes in PFC. This stripe-based gating architecture has an important advantageover the global nature of a purely DA-based gatingsignal (Braver and Cohen, 2000; Rougier and O’Reilly,
2002; Tanaka, 2002), which appears computationally t
inadequate for supporting a selective updating functionby itself.

. Top-down biasing of processing occurs via projectionsfrom actively-maintained representations in PFC to rel-evant areas throughout the brain, most typically thePC, but also the HC and the PFC/BG itself (Cohen andServan-Schreiber, 1992; Fuster, 1989).

. Learning what and when to gate is accomplished by aDA-based reinforcement-learning mechanism that iscapable of providing temporally-appropriate learningsignals to train gating update activity in the striatal Goand NoGo synapses. Thus, each spiny neuron devel-ops its own unique pattern of connection weights en-abling separate Go vs. NoGo decisions in each stripe.

Fig. 5 shows how the BG-mediated selective gatingechanism can enable performance of the 1-2-AX task

see Frank et al., 2001 for a working simulation). When aask demand stimulus is presented (e.g. 1), a BG gatingignal (i.e. a Go signal) must be activated to enable aarticular PFC stripe to gate in and retain this informationpanel a), and no stripe (or NoGo firing) should be acti-ated for a distractor such as C (Panel b). A different stripeust be gated for the subsequent cue stimulus A (panel c).hen the X stimulus is presented, the combination of this

timulus representation plus the maintained PFC workingemory representations is sufficient to trigger a target

esponse R (panel d).

earning when to gate in the BG

f all the aspects of our model that purport to deconstruct

ig. 5. Illustration of how the BG gating of different PFC stripes canolve the 1-2-AX task (light color�active; dark�not-active). (a) Thene task is gated into an anterior PFC stripe because a correspondingtriatal stripe fired Go. (b) The distractor C fails to fire striatial Goeurons, so it will not be maintained; however, it does elicit transientFC activity. Note that the one persists because of gating-induced

obust maintenance. (c) The A is gated in. (d) A right key-press motorction is activated (using same BG-mediated disinhibition mechanism)ased on X input plus maintained PFC context.

he homunculus, learning when to gate is clearly the most

cebials

uiscmitcflatTaa2eecfaessa

tsumttttrtwp

ba(pl

eiehc

k(

cimcidfpcta

tSeps1fusfiwcaPBlstnt

me1ateua(otutsdmOvdt


entral. For any working memory model, either the knowl-dge of when to update working memory must be built iny the model’s designer, or, somehow, a model must learn

t on its own, relying only on its built-in constraints inter-cting with its training experience. That is, without such a

earning mechanism, our model would have to resort toome kind of intelligent homunculus to control gating.

Our approach for simulating how the BG learns topdate task-relevant versus irrelevant working memory

nformation builds on our prior work for how the sameystem modulates the selection of motor responses. Spe-ifically, the BG are thought to facilitate the selection of theost appropriate response, while suppressing all compet-

ng responses (Mink, 1996). In our models, the BG learnhe distinction between good and bad responses viahanges in DA firing during positive and negative rein-orcement (for details, see Frank, 2005). In brief, our modeleverages the observation that DA D1 and D2 receptorsre relatively segregated in Go and NoGo cells, respec-ively (Gerfen, 1992; Wise et al., 1996; Aubert et al., 2000).his is of interest because DA effects on neuronal excit-bility and synaptic plasticity are dependent on whether itcts via D1 or D2 receptors (Hernandez-Lopez et al., 1997,000; Nishi et al., 1997; Centonze et al., 2001). The netffect is that increases in DA during positive reinforcementnhance BG Go firing and learning via simulated D1 re-eptors, whereas decreases in DA during negative rein-orcement have the opposite effect, enhancing NoGo firingnd learning via simulated D2 receptors. This functionalitynables the BG system to learn to discriminate betweenubtly different reinforcement values of alternative re-ponses, and is consistent with several lines of biologicalnd behavioral evidence (Frank, 2005).

For the PBWM model, we have extended these ideaso include BG modulation of PFC working memory repre-entations. Thus, increases in DA reinforce BG Go firing topdate information that is adaptive to store in workingemory, while decreases in DA allow the model to learn

hat its current working memory state is maladaptive. Inhis manner, the BG eventually come to update informationhat is task-relevant, because maintenance of this informa-ion over time leads to adaptive behavior and reinforcedesponses. Conversely, the system learns to ignore dis-racting information, because its maintenance will interfereith that of task-relevant information and therefore lead tooor performance.

As Fig. 5 illustrates, the learning problem in the BGoils down to learning when to fire a Go vs. NoGo signal ingiven stripe based on the combination of current sensory

PC) input and maintained PFC activations. From a com-utational perspective, there are two fundamental prob-

ems that must be solved by the learning mechanism.

Temporal credit assignment. The benefits of havingncoded a given piece of information into prefrontal work-

ng memory are typically only available later in time (e.g.ncoding the one task demand stimulus can only reallyelp later (in terms of getting an actual reward) when
onfronted with an A–X sequence). Thus, the problem is to (
now which prior events were critical for subsequent goodor bad) performance.

Structural credit assignment. The network must de-ide which stripes should encode which different pieces of

nformation at a given time, and when successful perfor-ance occurs, it must reinforce those stripes that actually

ontributed to this success. This form of credit assignments what neural network models are typically very good atoing (e.g. the backpropagation algorithm), but clearly thisorm of structural credit assignment interacts with the tem-oral credit assignment problem and with the unique mi-ro-anatomical structure of the stripe-loop architecture ofhe PFC and BG, making the technical problem consider-bly more complex.

The firing patterns of midbrain DA neurons (ventralegmental area, VTA; substantia nigra pars compacta,Nc; both strongly innervated by the BG) exhibit the prop-rties necessary to solve the temporal credit assignmentroblem, because they learn to fire for stimuli that predictubsequent rewards (e.g. Schultz et al., 1993; Schultz,998). This property is illustrated in schematic form in Fig. 6

or a simple Pavlovian conditioning paradigm, where a stim-lus (e.g. a tone) predicts a subsequent reward. Fig. 5bhows how this predictive DA firing can reinforce BG Goring to gate in and subsequently maintain a stimulus,hen such maintenance leads to subsequent reward. Spe-ifically, the DA firing can move discretely from the time ofreward to the onset of a stimulus that, if maintained in theFC, leads to the subsequent delivery of this reward.ecause this DA firing occurs at the time when the stimu-

us comes on, it is well timed to facilitate the storage of thistimulus in PFC. In our model, this occurs by reinforcinghe connections between the stimulus and the Go gatingeurons in the striatum, which then cause updating of PFCo maintain the stimulus.

The apparently predictive nature of the DA firing hasost often been explained in terms of the temporal differ-nces (TD) reinforcement learning mechanism (Sutton,988; Sutton and Barto, 1998; Schultz et al., 1995; Houk etl., 1995; Montague et al., 1996; Suri et al., 2001; Con-reras-Vidal and Schultz, 1999; Joel et al., 2002). However,xtensive exploration and analysis of these models has leds to develop a non-TD-based account, which movesway from the prediction framework upon which it is basedO’Reilly and Frank, in press; O’Reilly et al., unpublishedbservations). In brief, TD depends on chaining of predic-ions from one time step to the next, and any weak link (i.e.npredictable event) can break this chain. In many of theasks faced by our models (e.g. the 1-2-AX task), theequence of stimulus states is almost completely unpre-ictable, and this significantly disrupts the TD chainingechanism (O’Reilly et al., unpublished observations).ur alternative learning mechanism, called PVLV (primary

alue and learned value) involves two separable but inter-ependent learning mechanisms, each of which is essen-ially a simple delta-rule or Rescorla-Wagner mechanism
Rescorla and Wagner, 1972; Widrow and Hoff, 1960).

Tw

mbppvt

E

IePfedd

P

Tmm1Ptni2m(lPi

B

Rsmdoc2tfdwldGnaPtacts

mciitto

rdtTtBrDWmpddmtcovptipt

s“letCcwCswtoputawmisOawle


his PVLV mechanism shares several features in commonith the model of Brown et al., 1999.

The further details of these PBWM and PVLV learningechanisms are beyond the scope of this paper, but theasic results are that the resulting model can learn com-lex working memory tasks such as the 1-2-AX task basedurely on trial-and-error experience with the task. We pro-ide more examples of the learning and performance ofhis model below.

mpirical tests of the model

n addition to incorporating a large amount of existingmpirical data, the overall PBWM model of the role of theFC and BG in working memory makes a number of

urther predictions, several of which have been testedmpirically. We review some of these below, and theniscuss ongoing applications of the PBWM model to un-erstand working memory behavior.

FC organization

he PBWM model strongly predicts that information thatust be updated at different points in time during a workingemory task (e.g. the outer loop vs. the inner loop of the-2-AX task) should be represented in different parts of theFC. This prediction has been tested directly in the 1-2-AX

ask (Kroger et al., unpublished observations), and in aumber of other tasks that also share this same kind of

nner/outer loop structure (e.g. Braver and Bongiolatti,002; Koechlin et al., 2000, 2003). In all of these cases,ore anterior areas of PFC, specifically the frontal pole

Broadman’s area 10) was selectively activated for outer-oop information, while more posterior areas (dorso-lateralFC, Broadman’s areas 46/9) were active for inner-loop

nformation.

G/DA mechanisms

ecently, we have tested various aspects of the hypothe-ized roles of the BG/DA system across both reinforce-ent learning and working memory processes. First, weemonstrated striking support for a central prediction ofur model regarding DA involvement in “Go” and “NoGo”ognitive reinforcement learning (Frank et al., 2004; Frank,005). We tested Parkinson patients on and off medica-ion, along with healthy senior control participants matchedor age, education and a measure of verbal IQ. We pre-icted that decreased levels of DA in Parkinson’s diseaseould lead to spared NoGo learning, but impaired Go

earning (which depends on DA bursts). We further pre-icted that dopaminergic medication should alleviate theo learning deficit, but would block the effects of DA dipseeded to support NoGo learning, as was simulated toccount for other medication-induced cognitive deficits inarkinson’s disease (Frank, 2005). Results were consis-

ent with these predictions. In a probabilistic learning task,ll patients and aged-matched controls learned to makehoices that were more likely to result in positive ratherhan negative reinforcement. The difference was in their
trategy: patients taking their regular dose of dopaminergic s
edication implicitly learned more about the positive out-omes of their decisions (i.e. they were better at Go learn-

ng), whereas those who had abstained from taking med-cation implicitly learned to avoid negative outcomes (bet-er NoGo learning). Age-matched controls did not differ inheir tendency to learn more from the positive/negativeutcomes of their decisions.

We have also tested predictions for a more a generalole for BG/DA in cognitive function by administering lowoses of DA agonists/antagonists to young, healthy par-icipants (Frank and O’Reilly, unpublished observations).he drugs used (cabergoline and haloperidol) were selec-

ive for D2 receptors, which are by far most prevalent in theG. By acting on presynaptic D2 receptors, cabergoline

educes, while haloperidol enhances, the amount of phasicA that is released during dopaminergic cell bursting (e.g.,u et al., 2002). Again, results were consistent with ourodel: Increases in DA during learning caused partici-ants to learn more about the positive outcomes of theirecisions (as in medicated Parkinson’s patients), whereasecreases in DA caused the same participants to learnore about negative outcomes (as in non-medicated pa-

ients). Notably, these same effects were borne out in theontext of a modified version of the AX-CPT working mem-ry task. In our version, a variable number of task-irrele-ant distractor stimuli were presented during the delayeriod, and participants were told to ignore these distrac-ors for the purpose of target (A–X) detection. We alsoncluded an attentional set-shifting condition, in which thereviously task-relevant letters (A, X, B, Y) became dis-ractors, while previous distractors were now task-relevant.

Interestingly, increases in DA by haloperidol enhancedelective working memory updating of task-relevant (i.e.positively-valenced”), but not distracting (“negatively-va-enced”) information. By our model’s account, DA releasevoked during the presentation of task-relevant informa-ion reinforces BG Go firing to update this information.onsistent with this analysis, increased DA release alsoaused difficulty not updating (i.e. ignoring) this informationhen it subsequently became distracting in the set-shift.onversely, under cabergoline (decreased DA release)et-shifting deficits were observed that were consistentith impaired, rather than enhanced, Go learning. In par-

icular, whereas set-shifting deficits under haloperidol werenly observed in trials for which participants had to ignorereviously task-relevant distractors, deficits were observednder cabergoline when they only had to update the newask-relevant set (i.e. in trials without distractors). Finally,nd perhaps most suggestive for a role of BG DA inorking memory, participants with low baseline workingemory span were most subject to the effects of increases

n DA by haloperidol, while those with high span were mostubject to decreases in DA by cabergoline (Frank and’Reilly, unpublished observations). These latter resultsre consistent with the notion that individual differences inorking memory span are partially characterized by under-

ying differences in DA levels (Kimberg et al., 1997), butxtend this hypothesis in a more mechanistic fashion con-
istent with our modeling.

ttFenncApPmwsboitt

Ss

TvpapsctrTwcTa

gm(

itiM1a(a

FesrfiPslPsu

T

T

S

A

1W

I

E

A

SSN


Taken together, these results provide strong supporthat BG signals, under modulation by DA, are critical forhe updating of PFC working memory representations.urther, the model’s success in capturing subtle cognitiveffects in both Parkinson’s disease and controlled DA ma-ipulation suggests that it can also be applied to mecha-istically understand cognitive deficits in those with moreomplex disorders involving BG/DA dysfunction, such asDHD (attention deficit hyperactivity disorder) and schizo-hrenia. However, it is also the case that, although theBWM model has been specifically designed to includeany biological aspects, it obviously also goes beyondhat is currently known. For example, the model ascribespecific roles to subsets of neurons in the nucleus accum-ens, which provides testable hypotheses about the biol-gy and function of these systems in the brain. It will be

nteresting to see how some of these ideas implemented inhe PBWM model stand up to further biological investiga-ions.

imulating multiple working memory tasks in aingle integrated model

he PBWM model is quite complex. Although it is moti-ated by a wide range of empirical data, and some of itsredictions have been tested and confirmed as describedbove, it is nevertheless important to constrain the com-lexity of the model further by subjecting it to increasinglytringent tests. One strategy that we have employed suc-essfully in the past with both our hippocampal and pos-erior cortical models is to apply these models to as wide aange of cognitive neuroscience phenomena as possible.o the extent that the same basic model can account for aide range of data, it provides confidence that the model isapturing some critical core elements of cognitive function.he virtues of this general approach have been forcefullyrgued by Newell, 1990.

For these reasons, one important current researchoal is to attempt to simulate a wide range of workingemory tasks with one instantiation of the PBWM model

Table 1). This research builds upon earlier work simulat-

able 1. Brief descriptions of the tasks simulated in the multitask (MT

ask Brief description

troop Color words printed in same or different ink colors, reword/color in color naming condition, csp with PFC

X-CPT Letters appear one at a time on the screen. Target s(impaired with PFC damage).

-2-AX Like AX-CPT, except target depends on prior 1 or 2CST Wisconsin Card Sorting Task: multidimensional stimu

“rule” learned by trial-and-error, and changes unexD/ED Intradimensional/Extradimensional task, which is like

changes within a given dimension, or across differriksen A central, to-be-named stimulus is flanked by consist

flankers.BCA/ABBA A sequence of stimuli is presented; a response is req

ABBA condition.erial recall A sequence of verbal items must be recalled in ordeternberg Digits are presented in an array, then removed. One
-Back Repetitions separated by N in a continuous stimulus sequenc
ng many of the paradigmatic tasks thought to be charac-eristic of working memory and executive function, includ-ng: the Stroop effect (Cohen et al., 1990; O’Reilly and

unakata, 2000); the AX-CPT (Braver et al., 1995); the-2-AX (O’Reilly and Frank, in press); the WCST (Rougiernd O’Reilly, 2002); the ID/ED dynamic categorization taskO’Reilly et al., 2002); and the Eriksen flanker task (Eriksennd Eriksen, 1974; Cohen et al., 1992; Yeung et al., 2004;

ig. 6. (a) Schematic of DA neural firing for an input stimulus (Input,.g. a tone) that reliably predicts a subsequent reward (unconditionedtimulus US/r). Initially, DA fires at the point of reward, but then overepeated trials learns to fire at the onset of the stimulus. (b) This DAring pattern can solve the temporal credit assignment problem forFC active maintenance. Here, the PFC maintains the transient inputtimulus (initially by chance), leading to reward. As the DA systemearns, it can predict subsequent reward at stimulus onset, by virtue ofFC “bridging the gap” (in place of a sustained input). DA firing attimulus onset reinforces the firing of BG Go neurons, which drivepdating in PFC.

and the evidence implicating the PFC in them

is either color or word. Differential slowing is observed for conflicting.of A followed by X must be detected, requiring maintenance of A

1�AX, 2�BY target sequence.shape, number) are sorted according to one dimension; SortingIncreased perseveration on initial rule observed with PFC damage.

xcept multiple stimuli are presented simultaneously, allowing rulesions. PFC-related perseveration also observed.onsistent stimuli. PFC damage causes more intrusion from

en the 1st stimulus repeats. PFC damage causes false alarms in

is probed and measured capacity is often greater than serial recall.

) model,

sponsedamage

equence

stimulus;li (color,pectedly.WCST eent dimenent or inc

uired wh

r.location

e must be detected. PFC damage impairs performance.

BAl(1

tegbmmopamitnia

T

FlcB

bat

(sseosticwttb

FBcoBsS(s irect and

F


ogacz and Cohen, 2004). We also plan to simulate theBCA/ABBA task (Miller et al., 1996), serial recall (phono-

ogical loop) (Burgess and Hitch, 1999), Sternberg taskSternberg, 1996), and the N-Back task (Braver et al.,997).

In addition to the basic goal of simulating all of theseasks within a single model, this model will allow us toxplore fundamental questions about the nature and ori-ins of cognitive control, and intelligence more generally,y varying the training experiences that we give to theodel prior to testing on the full set of standard experi-ental tasks. Specifically, we plan to build up a repertoiref complex cognitive skills by initial training on much sim-ler tasks, and to explore the extent to which performancecross a range of tasks can build efficiently upon a com-on set of shared task elements. Before discussing these

ssues, we first describe our comprehensive MT (multi-ask) model (designed around the basic PBWM mecha-isms) that we are developing for this project, and our

nitial efforts demonstrating basic competency for the par-digmatic working memory tasks listed in Table 1.

he full MT model

ig. 7 shows the MT (multitask) model, with input/outputayers appearing at the top of the network, posterior corti-al “Hidden” layers and PFC layer in the middle, and

ig. 7. Overall structure of the MT model. Similar to our other PBWMG layers are at the bottom-right. The PVLV learning algorithm layersomponent also includes the BG (Matrix, SNrThal) and PFC on the righr multiple featured stimuli are presented in one or more “slots” in the Sased on these inputs, plus context provided by PFC input, the Hiddehown model has four “stripes” reflected in the four subgroups of theNrThal layers. LVe, learned value, excitatory (anatomically associate

same anatomical locus); PVe, primary value (PV) excitatory, externtriosomes of ventral striatum); SNrThal, abstracted layer reflecting d

G/midbrain areas for learning and gating of PFC at theda

ottom. The input/output representations were designed toccommodate the vagaries of each individual task in a wayhat achieves a high level of surface validity.

The perceptual input representations in the MT modelFig. 8) assume a high level of perceptual preprocessing,uch that different stimulus items (“objects”) are repre-ented with consistent and unique activity patterns. Wencode three separate (orthogonal) stimulus dimensions:bject identity, color, and size, and we also provide threepatial locations in which a given object may appear. Theask instruction layer tells the network what to do with thenput stimuli, including the overall task and any more spe-ific pieces of information that might be required (e.g.hether to do word reading or color naming in the Stroop

ask). We are exploring different ways of presenting theseask instructions, including presenting them only at theeginning of a block of trials on a given task, at the begin-

he input/output layers are at the top-left of the diagram. The PFC andlower left hand corner (highlighted in darker gray), and the full PBWMe (highlighted in lighter gray). Depending on the particular task, singlelayer, along with task instructions in the Task_Instruct and SIR layers.etermines the correct output in verbal or nonverbal form, or both. TheMatrix (striatal matrisomes) layers, and the four units of the SNc ande central nucleus of the amygdala); LVi, learned value (LV) inhibitory; PVi, primary value inhibitory (anatomically associated with patch/indirect pathways via substantia nigra, pars reticulata and thalamus.

ig. 8. Perceptual input features, organized along three separate

models, tare in thet hand sidtimuli_Inn layer dPFC andd with thal reward

imensions. Three separate locations of these features are provideds input to the network.

neatitoitpeacol

rntSfptatBcgfuftme

tibtilwasitn2

oef(c

vbtH

mtmumts

C

TidnmmtdtyoTsr

pfme“pfnhhbmatstm

Metcwrcebhmqspbw


ing of a trial but not during it, and constantly throughoutach trial. Furthermore, we will explore the presentation ofgiven task in a more surface valid manner, where after

raining on a set of basic cognitive operations, a given tasks then “described” to the model in terms of the combina-ion of these operations, instead of having it simply mem-rize the meaning of a distinctive task label. We have also

ncluded a sub-category of instruction inputs in the form ofhe store/ignore/recall (SIR) layer, which can be used torovide explicit working memory update signals that arencoded in a variety of different ways in different tasks,nd may also be present via implicit timing signals via theerebellum (Mauk and Buonomano, 2004; Ivry, 1996). Theutputs include both verbal and non-verbal responses, the

atter including button presses and pointing to locations.The task instruction and SIR input each have a sepa-

ate hidden layer associated with them, which enables theetwork to develop a more systematic internal represen-

ation of the task demands. These TaskHidden andIRHidden layers, along with the perceptual inputs, all

eed into one large common hidden layer (representingosterior association cortex), which in turn projects to the

wo output layers. The PFC is bidirectionally connected toll relevant high-level processing layers (sensory input,

ask hidden, central hidden, and output), and its associatedG layers receive from all of these layers as well to provideontrol over the learning and execution of the dynamicating signals. Note that the shown PFC/BG system has

our stripes, with each stripe representing a selectivelypdatable component of working memory. More stripes

acilitate faster learning, but result in a larger, more compu-ationally costly model, so the exact number of stripes is aatter of pragmatic optimization in the model (in the brain, westimate that many thousands of stripes are present).

When sensory inputs are presented, activation flowshroughout the network in a bidirectional manner, so thatnternal posterior cortical “hidden” layers are affected byoth these bottom-input and maintained top-down activa-

ions in the PFC. In the Leabra algorithm that we use,ndividual units are modeled as point neurons, with simu-ated ion channels contributing to a membrane potential,hich is in turn passed through a thresholded nonlinearctivation function to obtain a continuous instantaneouspike rate output that is communicated to other units. The

nhibitory conductances are efficiently computed accordingo a k-winners-take-all algorithm (kwta), which ensures thato more than some percentage (typically between 15 and5%) of units within a layer is active at a time.

Outside of the BG system, learning occurs as a resultf both Hebbian and error-driven mechanisms, with therror-driven learning computed in a biologically-plausible

ashion based on the GeneRec learning algorithmO’Reilly, 1996). The learning mechanisms for the BGomponents (PVLV algorithm) were described earlier.

The model can be systematically lesioned in a wideariety of ways and its performance tested on each of theehavioral paradigms. In some cases, there are clear pa-

ient data that the model will be expected to simulate.
owever, much of the time, these lesions will not clearly c
ap onto any existing piece of empirical data, and willherefore stand as important testable predictions of theodel. In addition to the standard techniques of damagingnits and connections, we can also manipulate the dopa-inergic and other pathways in the BG and their projec-

ions to the PFC to simulate conditions such as Parkin-on’s, ADHD, and schizophrenia.

urrent progress and future directions

he model is currently able to perform a set of core tasks,ncluding the Stroop, AX-CPT, 1-2-AX, and WCST, in ad-ition to a set of more primitive component tasks (e.g.aming, matching, and comparing stimulus features, di-ensions, and locations) that also involve basic workingemory capacities. The network has replicated basic fea-

ures of these tasks as highlighted in Table 1 (e.g. theifferential slowing of the color-naming conflict condition inhe Stroop task), but more extensive detailed testing haset to be performed, awaiting further training of the modeln more tasks, and all of the tasks integrated together.hus, although we are optimistic that the network willucceed in simulating all of these tasks, considerable workemains to be done.

As mentioned earlier, a single instantiation of our com-rehensive MT model will provide important opportunitiesor exploring a wide range of issues in cognitive develop-ent and human intelligence generally. Perhaps the great-st mystery in cognitive processing is where all thesmarts” come from to control the system in a task-appro-riate manner. Specifically, a fundamental question weace is, How is it that people quickly adapt to performingovel cognitive tasks, when it takes monkeys months ofighly-focused training to learn a single new task? Weypothesize that people develop an extensive repertoire ofasic cognitive operations throughout the long develop-ental period into adulthood, and then, are able to rapidlynd flexibly combine these basic “building block” opera-ions to solve novel tasks. This may sound like commonense, but demonstrating even rudimentary capabilities inhis regard in a mechanistically explicit computationalodel remains an important challenge.

To explore these issues, we are planning to train theT model on a large corpus of primitive task-orientedxperiences (e.g. object naming, color naming, etc.) so aso establish a basic set of capabilities underlying the moreomplex task paradigms targeted above. A critical issueill be how much specific training on each task will still be

equired, in comparison to prior training on more basicognitive operations that may be combinatorially applied toach task. One extreme of this dimension is representedy studies conducted on monkeys, where many months ofighly specific training are required to achieve task perfor-ance. In contrast, adult human participants typically re-uire a simple set of verbal instructions, followed by amall number of practice trials, to achieve high levels oferformance. In the past, neural network models haveeen much closer to the monkey end of this spectrum (ororse), requiring extensive task-specific training. Other
ognitive models (e.g. production systems) typically rely on

tetfmmhtrwm

bocwgkaMa(dbec(f

movgtroowwcmttpt

crzmiotDdtN

s

tvlmsf(fso

F

TasapvgaPidiawngttthbcwpapp

m“nestfflfitctmf

ree


he intelligence of the modeler to program in all the nec-ssary task components by hand, while non-computationalheories often invoke constructs that amount to homunculi,or example the “central executive” of Baddeley (1986). Aajor goal of our overall research program is to move ourodels closer to the human end of the spectrum. A keyypothesis to be tested is that our model will learn complexasks significantly faster after being pre-trained on simpler,elevant ones. If we are successful in achieving this goal, itould represent a critical qualitative step forward in theodeling of human-like intelligence.

Some preliminary work using an earlier version of ourasic model provides reason to be optimistic regarding thisverall approach. In simulations of a task paradigm wealled cross-task generalization (XT; Rougier et al., 2005),e explored the ability of training on one set of tasks toeneralize (transfer) to other, related tasks. In general, theey to generalization in a neural network is the formation ofbstract (e.g. categorical) representations (O’Reilly andunakata, 2000; Munakata and O’Reilly, 2003). For ex-mple, the abstract representation of the category “dog”which must be abstracted over experiences with specificogs) can enable one to surmise that poodles eat dog foodased on seeing a German shepherd eating dog food,ven though this might not be obvious given the raw per-eptual differences among members of the dog categorye.g. you might otherwise be tempted to feed a poodle catood).

In the Rougier et al. (2005) model, we trained a PFCodel using a simple form of dynamic gating mechanismn a varying number of related tasks operating on simpleisual stimuli (e.g. name a “feature” of the stimulus along aiven “dimension” such as its color, shape, or size; match

wo stimuli along one of these dimensions; compare theelative size of two stimuli). To test for generalization, wenly trained a given task on a small percentage (e.g. 30%)f all the stimuli, and then tested that task on stimuli thatere trained in other tasks. We found that only the modelith an intact PFC and dynamic gating mechanism wasapable of significant levels of generalization. Further-ore, this model developed discrete rule-like representa-

ions in the PFC that clearly and uniquely encoded theask-relevant stimulus dimensions, and the generalizationerformance and formation of these rule-like representa-ions were strongly correlated throughout all the models.

We think this pattern of results reflects a general prin-iple for why the PFC should develop more abstract rep-esentations than PC, and thus facilitate flexible generali-ation to novel environments: abstraction derives from theaintenance of stable representations over time, interact-

ng with learning mechanisms that extract commonalitiesver varying inputs. Supporting this view are data showing

hat damage to PFC impairs abstraction abilities (e.g.,ominey and Georgieff, 1997), and that PFC in monkeysevelops more abstract category representations thanhose in PC (Wallis et al., 2001; Freedman et al., 2002;ieder et al., 2002).

As a preliminary test of our overall approach, we have
uccessfully implemented the core findings of the cross- l
ask generalization model (Rouger et al., in press) in aersion of the MT model described here to validate that the

atter can replicate the core results generated by thatodel. Specifically, we have been able to easily demon-

trate that the latest PBWM mechanisms, using the inputramework and training environment of the Rougier et al.2005) model, could quickly learn the core tasks (nameeature, match feature, compare feature) and could alsouccessfully learn to generalize across tasks just as in theriginal.

uture developmental directions for the model

he question of how the PFC is functionally organized islso prominent in the literature, and remains largely unre-olved. We think this path of research can shed consider-ble light on this issue as well. Previously, we have pro-osed that the anterior-posterior (and perhaps dorsal–entral) axis of the PFC might be organized along aradient from abstract to concrete, respectively (O’Reilly etl., 2002; O’Reilly and Munakata, 2000). However, theBWM model currently has no mechanism for encourag-

ng such a gradient (or any other kind of gradient) toevelop in the stripes of the PFC (all stripes are equipotent

n their access to information). Therefore, an additionalvenue of research we plan to explore is to look at variousays of biasing the model to develop a gradient of orga-ization along its PFC stripes, and see what types ofradients actually develop in response to the battery ofraining provided, while assessing any behavioral implica-ions this organization might have (e.g. does the organiza-ional principle actually facilitate processing, and if so,ow?). One particular organizational bias suggested by theiology is to have only the more posterior areas of PFConnected (bidirectionally) with posterior cortical areas,hile more anterior PFC areas connect only with theseosterior PFC areas. Thus, anterior PFC areas might beble to serve as more abstract biasing inputs to moreosterior PFC areas, which in turn bias more specificrocessing in PC.

Understanding the human capacity for generativityay be one of the greatest challenges facing the field of

higher-level” cognitive function. We think that the mecha-isms of the PBWM model, and in particular its ability toxhibit limited variable-binding functionality, may be criticalteps along the way to such an understanding. It may behat, over the 20 or so years it takes to fully develop aunctional PFC, people have developed a systematic andexible set of representation that supports dynamic recon-guration of input/output mappings according to main-ained PFC representations. Thus, these PFC “variables”an be activated by task instructions, and support novelask performance without extensive training. This andany other important problems remain to be addressed in

uture research.Finally, while our model addresses the computational

ole of BG DA in working memory, we are only beginning toxplore DA effects in PFC. In brief, we think that phasic DAffects are critical for learning in the BG, whereas longer

asting DA effects in PFC support robust maintenance of

wSdato

Apehattduissfii(msmos

asmricirfeac

Aa

A

A

A

A

A

B

B

B

B

B

B

B

B

B

B

B

C

C

C

C

C

C

C

D

D

D

D

E


orking memory representations (Durstewicz et al., 2000;eamans and Yang, 2004; Tanaka, 2002). In the modelescribed here, these dopaminergic effects in PFC werebstracted and subsumed by a simple intracellular main-enance current—but these currents are known to dependn a healthy level of tonic DA.

CONCLUSION

lthough many theoretical models have been developedurporting to explain aspects of working memory and ex-cutive function, the mechanistic basis underlying themas remained inadequately described, often amounting tohomunculus. In this paper, we have reviewed some of

he progress made by our colleagues and others in at-empting to deconstruct this implicit homunculus by eluci-ating the precise computational and neural mechanismsnderlying them, particularly the role of the PFC. These

deas can be specified at multiple levels. At a more ab-tract level, we outlined six key functional demands that weee underlying working memory, which need to be satis-ed by the neural system. We also described a detailedmplementation of these functional demands in the PBWMPFC BG working memory) computational model. Thisodel attempts to incorporate detailed biological con-

traints in addition to the more abstract functional de-ands, and is capable of learning complex working mem-ry tasks strictly as a function of experience (without task-pecific knowledge having to be built in by the modeler).

We are currently applying this computational model torange of different working memory tasks, which should

trongly test the cognitive neuroscience validity of theodel. For example, the model can be used to explore

oles of the individual neural systems involved by perturb-ng parameters to simulate development, aging, pharma-ological manipulations, and neurological dysfunction, and

t promises to be extensible to a broad array of otherelevant manifestations of working memory and executiveunction. Furthermore, we hope to use this platform toxplore fundamental questions regarding the neural basisnd origins of our uniquely flexible human intelligence andapacity for cognitive control.

cknowledgments—Supported by ONR grant N00014-03-1-0428,nd NIH grants MH069597 and MH64445.

REFERENCES

lexander GE, DeLong MR, Strick PL (1986) Parallel organization offunctionally segregated circuits linking basal ganglia and cortex.Ann Rev Neurosci 9:357–381.

mos A (2000) A computational model of information processing in thefrontal cortex and basal ganglia. J Cogn Neurosci 12:505–519.

nderson JR (1983) The architecture of cognition. Cambridge, MA:Harvard University Press.

tallah HE, Frank MJ, O’Reilly RC (2004) Hippocampus cortex andbasal ganglia: insights from computational models of complemen-tary learning systems. Neurobiol Learn Mem 82(3):253–267.

ubert I, Ghorayeb I, Normand E, Bloch B (2000) Phenotypical char-acterization of the neurons expressing the D1 and D2 dopamine
receptors in the monkey striatum. J Comp Neurol 418:22–32.
addeley AD (1986) Working memory. New York: Oxford UniversityPress.

eiser DG, Houk JC (1998) Model of cortical-basal ganglionicprocessing: encoding the serial order of sensory events. J Neuro-physiol 79:3168–3188.

erns GS, Sejnowski TJ (1995) How the basal ganglia make deci-sions. In: Neurobiology of decision-making (Damasio A, DamasioH, Christen Y, eds), pp 101–113. Berlin: Springer-Verlag.

erns GS, Sejnowski TJ (1998) A computational model of how thebasal ganglia produces sequences. J Cogn Neurosci 10:108–121.

ogacz R, Cohen JD (2004) Parameterization of connectionist mod-els. Behav Res Methods Instr Comput 36(4):732–741.

raver TS, Barch DM, Cohen JD (1999) Cognition and control inschizophrenia: A computational model of dopamine and prefrontalfunction. Biol Psychiatry 46:312–328.

raver TS, Bongiolatti SR (2002) The role of frontopolar cortex insubgoal processing during working memory. Neuroimage 15:523–536.

raver TS, Cohen JD (2000) On the control of control: The role ofdopamine in regulating prefrontal function and working memory. InControl of cognitive processes: Attention and performance XVIII(Monsell S, Driver J, eds), pp 713–737. Cambridge, MA: MITPress.

raver TS, Cohen JD, Nystrom LE, Jonides J, Smith EE, Noll DC(1997) A parametric study of frontal cortex involvement in humanworking memory. Neuroimage 5:49–62.

rown J, Bullock D, Grossberg S (1999) How the basal ganglia useparallel excitatory and inhibitory learning pathways to selectively re-spond to unexpected rewarding cues. J Neurosci 19:10502–10511.

urgess N, Hitch GJ (1999) Memory for serial order: A network modelof the phonological loop and its timing. Psychol Rev 106:551–581.

entonze D, Picconi B, Gubellini P, Bernardi G, Calabresi P (2001)Dopaminergic control of synaptic plasticity in the dorsal striatum.Eur J Neurosci 13:1071–1077.

ohen JD, Braver TS, O’Reilly RC (1996) A computational approachto prefrontal cortex, cognitive control, and schizophrenia: Recentdevelopments and current challenges. Philos Trans R Soc (Lond)B 351:1515–1527.

ohen JD, Dunbar K, McClelland JL (1990) On the control of auto-matic processes: A parallel distributed processing model of theStroop effect. Psychol Rev 97(3):332–361.

ohen JD, Perlstein WM, Braver TS, Nystrom LE, Noll DC, Jonides J,Smith EE (1997) Temporal dynamics of brain activation during aworking memory task. Nature 386:604–608.

ohen JD, Servan-Schreiber D (1992) Context, cortex, and dopamine:A connectionist approach to behavior and biology in schizophrenia.Psychol Rev 99:45–77.

ohen JD, Servan-Schreiber D, McClelland JL (1992) A parallel dis-tributed processing approach to automaticity. Am J Psychol105:239–269.

ontreras-Vidal JL, Schultz W (1999) A predictive reinforcementmodel of dopamine neurons for learning approach behavior.J Comp Neurosci 6:191–214.

ominey P, Arbib M, Joseph J-P (1995) A model of corticostriatalplasticity for learning oculomotor associations and sequences. JCogn Neurosci 7:311–336.

ominey PF, Georgieff N (1997) Schizophrenics learn surface but notabstract structure in a serial reaction time task. Neuroreport 8:2877

urstewitz D, Kelc M, Gunturkun O (1999) A neurocomputationaltheory of the dopaminergic modulation of working memory func-tions. J Neurosci 19:2807

urstewitz D, Seamans JK, Sejnowski TJ (2000) Dopamine-mediatedstabilization of delay-period activity in a network model of prefrontalcortex. J Neurophysiol 83:1733–1750.

riksen BA, Eriksen CW (1974) Effects of noise letters upon theidentification of a target letter in a nonsearch task. Percept Psy-
chophys 16:143–149.

F

F

F

F

F

F

F

F

F

G

G

H

H

H

H

I

J

J

K

K

K

K

K

L

L

M

M

M

M

M

M

M

M

M

M

M

N

N

N

N

N

N

O

O

O

O


ellous JM, Wang XJ, Lisman JE (1998) A role for NMDA-receptorchannels in working memory. Nat Neurosci 1:273–275.

rank MJ (2005) Dynamic dopamine modulation in the basal ganglia:A neurocomputational account of cognitive deficits in medicatedand non-medicated Parkinsonism. J Cogn Neurosci 17:51–72.

rank MJ, Loughry B, O’Reilly RC (2001) Interactions between thefrontal cortex and basal ganglia in working memory: A computa-tional model. Cogn Affect Behav Neurosci 1:137–160.

rank MJ, Rudy JW, O’Reilly RC (2003) Transitivity, flexibility, con-junctive representations and the hippocampus: II. a computationalanalysis. Hippocampus 13:341–354.

rank MJ, Seeberger L, O’Reilly RC (2004) By carrot or by stick:Cognitive reinforcement learning in Parkinsonism. Science 306:1940–1943.

reedman DJ, Riesenhuber M, Poggio T, Miller EK (2002) Visualcategorization and the primate prefrontal cortex: Neurophysiologyand behavior. J Neurophysiol 88:929–941.

unahashi S, Bruce CJ, Goldman-Rakic PS (1989) Mnemonic codingof visual space in the monkey’s dorsolateral prefrontal cortex.J Neurophysiol 61:331–349.

uster JM (1989) The prefrontal cortex: Anatomy, physiology andneuropsychology of the frontal lobe. New York: Raven Press.

uster JM, Alexander GE (1971) Neuron activity related to short-termmemory. Science 173:652–654.

erfen C (1992) The neostriatal mosaic: multiple levels of compart-mental organization in the basal ganglia. Ann Rev Neurosci15:285–320.

raybiel AM, Kimura M (1995) Adaptive neural networks in the basalganglia. In: Models of information processing in the basal ganglia(Houk JC, Davis JL, Beiser DG, eds), pp 103–116. Cambridge,MA: MIT Press.

ernandez-Lopez S, Bargas J, Surmeier DJ, Reyes A, Galarraga E(1997) D1 receptor activation enhances evoked discharge in neos-triatal medium spiny neurons by modulating an L-type Ca2� con-ductance. J Neurosci 17:3334–3342.

ernandez-Lopez S, Tkatch T, Perez-Garci E, Galarraga E, Bargas J,Hamm H, Surmeier DJ (2000) D2 dopamine receptors in striatalmedium spiny neurons reduce L-type Ca2� currents and excitabil-ity via a novel PLC�1-IP3-calcineurin-signaling cascade. J Neuro-sci 20:8987–8995.

ouk JC, Adams JL, Barto AG (1995) A model of how the basalganglia generate and use neural signals that predict reinforcement.In: Models of information processing in the basal ganglia (Houk JC,Davis JL, Beiser DG, eds), pp 233–248). Cambridge, MA: MITPress.

ouk JC, Wise SP (1995) Distributed modular architectures linkingbasal ganglia, cerebellum, and cerebral cortex: their role in plan-ning and controlling action. Cereb Cortex 5:95–110.

vry R (1996) The representation of temporal information in perceptionand motor control. Curr Opin Neurobiol 6:851–857.

ackson S, Houghton G (1995) Sensorimotor selection and the basalganglia: A neural network model. In: Models of information pro-cessing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds),pp 337–368. Cambridge, MA: MIT Press.

oel D, Niv Y, Ruppin E (2002) Actor-critic models of the basal ganglia:new anatomical and computational perspectives. Neural Netw15:535–547.

imberg DY, D’Esposito M, Farah MJ (1997) Effects of bromocriptineon human subjects depend on working memory capacity. Neuro-report 8:3581–3585.

oechlin E, Corrado G, Grafman J (2000) Dissociating the role of themedial and lateral anterior prefrontal cortex in human planning.Proc Natl Acad Sci U S A 97:7651

oechlin E, Ody C, Kouneiher F (2003) Neuroscience: The architec-ture of cognitive control in the human prefrontal cortex. Science
424:1181–1184.
ropotov JD, Etlinger SC (1999) Selection of actions in the basalganglia-thalamocoritcal circuits: Review and model. Int J Psycho-physiol 31:197–217.

ubota K, Niki H (1971) Prefrontal cortical unit activity and delayedalternation performance in monkeys. J Neurophysiol 34:337–347.

evitt JB, Lewis DA, Yoshioka T, Lund JS (1993) Topography ofpyramidal neuron intrinsic connections in macaque monkey pre-frontal cortex (areas 9 46). J Comp Neurol 338:360–376.

ovett MC, Reder LM, Lebiere C (1999) Modeling working memory ina unified architecture. In: Models of working memory: Mechanismsof active maintenance and executive control (Miyake A, Shah P,eds) pp. 135–182. New York: Cambridge University Press.

arr D (1971) Simple memory: A theory for archicortex. Philos TransR Soc (Lond) B 262:23–81.

auk MD, Buonomano DV (2004) The neural basis of temporal pro-cessing. Ann Rev Neurosci 27:307–340.

cClelland JL, McNaughton BL, O’Reilly RC (1995) Why there arecomplementary learning systems in the hippocampus and neo-cortex: Insights from the successes and failures of connectionistmodels of learning and memory. Psychol Rev 102:419–457.

cNaughton BL, Morris RGM (1987) Hippocampal synaptic enhance-ment and information storage within a distributed memory system.Trends Neurosci 10 (10):408–415.

iddleton FA, Strick PL (2000) Basal ganglia output and cognition:Evidence from anatomical, behavioral, and clinical studies. BrainCogn 42:183–200.

iller EK, Cohen JD (2001) An integrative theory of prefrontal cortexfunction. Ann Rev Neurosci 24:167–202.

iller EK, Erickson CA, Desimone R (1996) Neural mechanisms ofvisual working memory in prefontal cortex of the macaque. J Neu-rosci 16:5154

ink JW (1996) The basal ganglia: Focused selection and inhibition ofcompeting motor programs. Prog Neurobiol 50:381–425.

iyashita Y, Chang HS (1988) Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature 331:68–70.

ontague PR, Dayan P, Sejnowski TJ (1996) A framework for mes-encephalic dopamine systems based on predictive Hebbian learn-ing. J Neurosci 16:1936–1947.

unakata Y, O’Reilly RC (2003) Developmental and computationalneuroscience approaches to cognition: The case of generalization.Cogn Studies 10:76–92.

akahara H, Doya K (1998) Near saddle-node bifurcation behavior asdynamics in working memory for goal-directed behavior. NeuralComput 10:113–132.

akahara H, Doya K, Hikosaka O (2001) Parallel cortico-basal gangliamechanisms for acquisition and execution of visuomotor sequenc-es-a computational approach. J Cogn Neurosci 13:626–647.

ewell A (1990) Unified theories of cognition. Cambridge, MA: Har-vard University Press.

ieder A, Freedman DJ, Miller EK (2002) Representation of the quan-tity of visual items in the primate prefrontal cortex. Science298:1708–1711.

ishi A, Snyder G, Greengard P (1997) Bidirectional regulation ofDARPP-32 phosphorylation by dopamine. J Neurosci 17:8147–8155.

orman KA, O’Reilly RC (2003) Modeling hippocampal and neocorti-cal contributions to recognition memory: A complementary learningsystems approach. Psychol Rev 110:611–646.

’Reilly RC (1996) Biologically plausible error-driven learning usinglocal activation differences: The generalized recirculation algo-rithm. Neural Comput 8 (5):895–938.

’Reilly RC (1998) Six principles for biologically-based computationalmodels of cortical cognition. Trends Cogn Sci 2 (11):455–462.

’Reilly RC (2001) Generalization in interactive networks: The bene-fits of inhibitory competition and Hebbian learning. Neural Comput13:1199–1242.

’Reilly RC, Braver TS, Cohen JD (1999) A biologically based com-putational model of working memory. In: Models of working
memory: Mechanisms of active maintenance and executive control

O

O

O

O

O

O

P

R

R

R

R

R

S

S

S

S

S

S

S

S

T

W

W

WW

W

W

W

Y


(Miyake A, Shah P, eds), pp 375–411. New York: CambridgeUniversity Press.

’Reilly RC, Frank MJ (in press). Making working memory work: Acomputational model of learning in the frontal cortex and basalganglia. Neural Computation, in press.

’Reilly RC, McClelland JL (1994) Hippocampal conjunctive encoding,storage, and recall: Avoiding a tradeoff. Hippocampus 4 (6):661–682.

’Reilly RC, Munakata Y (2000) Computational explorations in cogni-tive neuroscience: Understanding the mind by simulating the brain.Cambridge, MA: MIT Press.

’Reilly RC, Noelle D, Braver TS, Cohen JD (2002) Prefrontal cortexand dynamic categorization tasks: Representational organizationand neuromodulatory control. Cereb Cortex 12:246–257.

’Reilly RC, Norman KA (2002) Hippocampal and neocortical contri-butions to memory: Advances in the complementary learning sys-tems framework. Trends Cogn Sci 6:505–510.

’Reilly RC, Rudy JW (2001) Conjunctive representations in learningand memory: Principles of cortical and hippocampal function. Psy-chol Rev 108:311–345.

ucak ML, Levitt JB, Lund JS, Lewis DA (1996) Patterns of intrinsicand associational circuitry in monkey prefrontal cortex. J CompNeurol 376:614–630.

escorla RA, Wagner AR (1972) A theory of Pavlovian conditioning:Variation in the effectiveness of reinforcement and non-reinforce-ment. In: Classical conditioning II: Theory and research (Black AH,Prokasy WF, eds), pp 64–99. New York: Appleton-Century-Crofts.

olls ET (1989) Functions of neuronal networks in the hippocampusand neocortex in memory. In: Neural models of plasticity: Experi-mental and theoretical approaches (Byrne JH, Berry WO, eds), pp240–265. San Diego, CA: Academic Press.

ougier NP, Noelle D, Braver TS, Cohen JD, O’Reilly RC (2005)Prefrontal cortex and the flexibility of cognitive control: Rules with-out symbols. Proc Natl Acad Sci U S A 102 (20):7338–7343.

ougier NP, O’Reilly RC (2002) Learning representations in a gatedprefrontal cortex model of dynamic task switching. Cogn Sci26:503–520.

umelhart DE, McClelland JL, PDP Research Group, eds. (1986)Parallel distributed processing. Volume 1: Foundations. Cam-bridge, MA: MIT Press.

chultz W (1998) Predictive reward signal of dopamine neurons.
J Neurophysiol 80:1.
chultz W, Apicella P, Ljungberg T (1993) Responses of monkeydopamine neurons to reward and conditioned stimuli during suc-cessive steps of learning a delayed response task. J Neurosci13:900–913.

chultz W, Romo R, Ljungberg T, Mirenowicz J, Hollerman JR, DickinsonA (1995) Reward-related signals carried by dopamine neurons. In:Models of information processing in the basal ganglia (Houk JC, DavisJL, Beiser DG, eds), pp 233–248. Cambridge, MA: MIT Press.

eamans J, Yang C (2004) The principal features and mechanisms ofdopamine modulation in the prefrontal cortex. Prog Neurobiol74:1–57.

ternberg S (1966) High speed scanning in human memory. Science153:652–654.

uri RE, Bargas J, Arbib MA (2001) Modeling functions of striataldopamine modulation in learning and planning. Neuroscience103:65–85.

utton RS (1988) Learning to predict by the method of temporaldiferences. Machine Learn 3:9–44.

utton RS, Barto AG (1998) Reinforcement learning: An introduction.Cambridge, MA: MIT Press.

anaka S (2002) Dopamine controls fundamental cognitive operationsof multi-target spatial working memory. Neural Netw 15:573–582.

allis JD, Anderson KC, Miller EK (2001) Single neurons in prefrontalcortex encode abstract rules. Nature 411:953–956.

ang X-J (1999) Synaptic basis of cortical persistent activity: Theimportance of NMDA receptors to working memory. J Neurosci19:9587.

ickens J (1993) A theory of the striatum. Oxford: Pergamon Press.ickens JR, Kotter R, Alexander ME (1995) Effects of local connec-

tivity on striatal function: simulation and analysis of a model. Syn-apse 20:281–298.

idrow B, Hoff ME (1960) Adaptive switching circuits. Institute ofRadio Engineers, Western Electronic Show and Convention, Con-vention Record, Part 4 (pp 96–104).

ise SP, Murray EA, Gerfen CR (1996) The frontal cortex-basalganglia system in primates. Crit Rev Neurobiol 10:317–356.

u Q, Reith M, Walker Q, Kuhn C, Caroll F, Garris P (2002) Concur-rent autoreceptor-mediated control of dopamine release and up-take during neurotransmission: an in vivo voltammetric study.J Neurosci 22:6272–6281.

eung N, Botvinick MM, Cohen JD (2004) The neural basis of errordetection: Conflict monitoring and the error-related negativity. Psy-
chol Rev 111 (4):931–959.
(Accepted 27 April 2005)(Available online 15 December 2005)

banishing the homunculus: making working memory work

Documents