Birdsong LearningI R Fiete, University of California, Santa Barbara, andCenter for Neural Systems, Pasadena, CA, USAH S Seung, Massachusetts Institute of Technology,Boston MA, USA

ã 2009 Published by Elsevier Ltd.

Introduction

Birdsong has been compared to human speechbecause both are examples of animal vocalizationsthat are at least partially learned. Indeed, some ofthe neural pathways underlying these vocal behaviorsmay be similar in songbirds and humans. More gen-erally, birdsong can be compared to any motor behav-ior that involves sequential control and learning andis an ideal playground for experimental and theoreti-cal explorations of the neural basis for motor learningand control.The neurons involved in song production and learn-

ing are clustered in a small number of discrete nucleiwithwell-defined boundaries, fairly well-characterizedconnectivity, and increasingly well-studied neuro-physiology. The various species of oscine songbirdsdisplay a vast spectrum of complexity in syllablesequencing (syntax), improvisation, and vocal acqui-sition. For example, song can be as simple as repeti-tions of a set of syllables arranged in fixed order(e.g., zebra finches), or it can be intricately impro-vised, with syllable ordering governed by probabilistictransitions between different syllables (e.g., domesti-cated bengalese finches). Mockingbirds can acquirenovel songs and learn novel nonbird sounds through-out life. Yet the basic neural song circuit is similar inall oscines. This diversity of behavior subserved by anunderlying similarity of structure makes birdsongan intriguing system for future comparative studiesof sophisticated motor sequencing and control.Models have so far focused on the zebra finch, a

songbird with one of the simplest of oscine song reper-toires, because most recordings of neural activity havebeen obtained from these birds. The male zebra finchlearns one song motif with fixed syllable order andsings repetitions of that motif throughout its life. Songacquisition takes place in a sensory phase, when thejuvenile bird acquires a mental template of the songof a possibly unrelated adult tutor, and an overlapp-ing sensorimotor phase, when the juvenile beginsto vocalize and tries to match its vocalizations withits mental template. In the laboratory the sensoryand sensorimotor phases can be separated: templateacquisition is fast and can be completed and the tutorbird removed from the juvenile’s presence before the

juvenile has begun to practice its song. Sensorimotorlearning is a slow process of self-generated trial anderror and can proceed successfully in complete audi-tory and social isolation after template acquisition.However, auditory feedback of the bird’s own voca-lizations is crucial during sensorimotor learningbecause a bird deafened at this stage cannot learn toreproduce its mental template of the tutor song.

This article focuses on the motor and sensorimotoraspects of song production, in particular the forma-tion of neural sequences to drive song, the learning ofthe motor map, and the reasons song may be encodedthe way it is by songbird premotor neurons. The aimis to illustrate how theoretical work has contributedto the understanding of the song system and highlightthe resulting predictions for experiment. Auditoryneural coding and dynamics and the physics (acous-tics and mechanics) of songbird vocal production arebeyond the scope of the article.

Basic Functional Anatomy ofthe Song Pathway

In the oscine vocal premotor circuit, high-level vocalcenter nucleus (HVC) projects to nucleus robustusarchistriatalis (RA). RA drives the motor neuronsthat innervate the syringeal and respiratory organsfor song production (Figure 1). Lesioning any ofthese nuclei immediately disables song production,while nuclei upstream of HVC are not necessary forsinging in adult zebra finches. Therefore, HVC origi-nates the top-level premotor drive for adult zebrafinch song. Although the circuitry between the pre-motor nuclei is feedforward, connectivity withinHVC and within RA is extensively recurrent. Inaddition to the premotor circuit, the song systemcontains the anterior forebrain pathway (AFP),which connects HVC to RA through an indirectpath. Lesion studies reveal that the AFP is crucialfor the sensorimotor phase of song learning andlong-term song maintenance but not for song produc-tion. The AFP is an avian homolog of the mammalianthalamocortical–basal ganglia pathway.

Based on anatomical, neurophysiological, andlesion data, it is believed that HVC generates thespatiotemporal premotor drive for sequential motoractivation in the form of sequential neural activity.The HVC activity is ‘abstract,’ in the sense that itencodes only temporal ordering, not song features(Figures 2(a) and 2(b)). The synapses from HVC toRA and from RA to the motor neurons convert thesequence of HVC activity patterns into the sequenceof motor commands that produces song (Figure 2).

Birdsong Learning 227

According to this perspective, there are two types oflearning. First, HVC acquires the capability of gener-ating a long sequence of neural activity patterns(Figures 2(f) and 2(g)). Second, the circuitry down-stream from HVC acquires the capability of mappingHVC activity patterns to the correct sound-producingpatterns of activity in RA. The HVC–RA synapsesshow extensive structural plasticity, with synapticgrowth followed by massive retraction, preciselyduring the sensorimotor learning period. Electro-physiological synaptic maturation accompanies suchstructural remodeling. Therefore, motor map learningfrom HVC to the vocal outputs is likely to involveplasticity in the HVC–RA synapses. Other possibleloci of plasticity for sensorimotor learning, not ruledout by experiments, include the recurrent RA–RAsynapses or the RA–motor neuron connections.

Sequence Generation

The high-level premotor area HVC displays clearsequential patterns of neural activation. In the awakesinging zebra finch, each recorded RA-projectingHVC neuron (also called an HVCRA neuron) fires atmost one brief burst of spikes (lasting �6 ms) persecond-long song motif (Figure 2(a)). These neuronsare otherwise inactive during singing. Different RA-projecting HVC neurons fire their single bursts atdifferent fixed times in the motif. Thus as a popula-tion, the HVCRA neurons cover time during song.In addition to RA-projecting neurons, HVC con-

tains inhibitory interneurons which ramify withinHVC and X-projecting neurons which send their out-puts to the AFP. Inhibitory interneuron activity is very

different from HVCRA activity: these neurons firetonically throughout song, with high rates and rela-tively little modulation in firing frequency across song(Figure 2(b)).

In the adult zebra finch, deafening causes no imme-diate effect on song production, suggesting that audi-tory feedback is not important for sequence generation.Also, lesion studies indicate that input from the higherinterfacial nucleus to HVC is not necessary for singingin zebra finches.What network connectivity and neuraldynamics underlie the sequence generation dynamicsobserved in the top-level premotor area, and howmight that connectivity become established as hatchl-ings mature?

The Synaptic Chain Model

The ‘synaptic chain’ model is an old idea for hownetworks of neurons might produce sequential pat-terns of activity. (The idea was already old when itwas critiqued by Lashley in 1951 under the name‘associative chaining’ model.) In the model’s simplestversion, the neurons in a network are divided intogroups, and the groups are ordered in a sequence.From each group of neurons, there are excitatorysynaptic connections to the neurons in the nextgroup of the sequence. The groups and their synapticconnections are like the links of a chain.

If the first group in the sequence is activated, itsends synaptic drive to the second group, whichthen becomes active. The second group in turn excitesthe third group to become active, and so on down thechain. (An alternative mechanism for the propagationof sequential activity in songbird HVC is based onthe removal of lateral inhibition between groups of

a

Premotor

AFP

HVC

RA

Motor

X

DLM

LMAN

b

HVC

NIfLMAN

XDLM

UVA

RA

Syrinx nXIIts

Excitatory

Inhibitory

Cortex

Basal ganglia

Thalamus

Figure 1 The neural pathways underlying song learning and song production. (a) Song learning and production involves a discrete set

of nuclei. (b) The same areas, unfolded to better illustrate their connectivity. Lines terminating in arrows designate excitatory or putative

excitatory connectivity. Lines terminating in circles designate inhibitory synapses. Blue nuclei are part of the premotor pathway. Green

nuclei are part of the anterior forebrain pathway (AFP) and are important for sensorimotor song learning but not song production. HVC,

high-level vocal center nucleus; LMAN, lateral magnocellular nucleus of the anterior nidopallium; NIf, interfacial nucleus; RA, nucleus

robustus archistriatalis.

228 Birdsong Learning

neurons; however, this mechanism does not allow forthe self-propagation of sequential activity, insteadrequiring an external periodic drive (clocked input)to induce transitions between activity states.)

More-complex versions of the synaptic chainmodel include inhibitory as well as excitatorysynapses or different types of synaptic currents withmultiple time constants. In still other versions, pre-cisely synchronous spiking of one group is required toactivate the next group. These are known as synfire(or ‘synchronous firing’) chain networks.

The role of intrinsic bursting Two recent modelshave implemented the synaptic chain idea with excit-atory conductance-based neurons to model sequencegeneration in the zebra finch HVC.

Recordings in the singing zebra finch show thatwhen an RA-projecting HVC neuron is active duringsong, it fires a high-frequency burst of four to sixspikes in a 6–10ms interval (Figure 2(a)). Previouslywe discussed the question of the mechanism ofsequential activation in HVC. Another basic questionconcerns the mechanisms that give rise to the burstingbehavior. One possibility is that the neurons burstbecause they are driven by synaptic input that lastsfor the duration of the burst. Alternatively, the neu-rons might burst as a result of the dynamics of theirintrinsic conductances.

Li and Greenside have advanced a model that doesnot use intrinsic bursting. It is consistent with the factthat previous brain slice studies of HVC neurons havenot revealed intrinsic bursting. The network model isable to produce sequential neural activity with multi-spike bursts (Figure 3(a)). However, the realism ofthis model is compromised by the fact that the peakfiring rate within the propagating multispike burstachieved by the model (�15ms interspike intervals)is approximately 10 times lower than seen in theactual HVC (�1.5ms interspike intervals).

In contrast, Jin et al. have advanced amodel inwhichHVC neurons are intrinsically bursting. The modelgenerates multispike bursts at peak firing rates consis-tent with those seen in experiments (Figure 3(b)). Forcomparison, Jin et al. also simulated a synaptic chainmodel without intrinsic bursting. Unlike Greensideand Li, they were able to achieve higher peak firing

100 ms

a

b

c

d

e

f g

HV

CR

AH

VC

IR

Aex

Son

g

HVC

RA

Motor

Time

abcxyzm1

m2

1234

1

a

m1

WHVC

WHVC-RA

WRA

WRA-motor

2 3 4

b c x y z

m2

Figure 2 Summary of data and schematic model. (a)–(e) Activ-

ity in the premotor pathway. Raster plot of single-unit activity in

nucleus robustus archistriatalis (RA)-projecting high-level vocal

center nucleus (HVC) neurons (HVCRA neurons; a), in HVC inter-

neurons (HVCI; b), and in RA (c). Consecutive lines of the same

color are multiple recordings of the same neuron. Ticks represent

single spikes. (d) Electromyogram of activity in the abdominal

expiratory muscle. (e) Spectrogram of resulting song. (a) and (b)

adapted from Hahnloser RH, Kozhevnikov AA, and Fee MS

(2002) An ultra-sparse code underlies the generation of neural

sequences in a songbird. Nature 419(6902): 65–70; (c) from Leo-

nardo A and Fee MS (2005) Ensemble coding of vocal control in

birdsong. Journal of Neuroscience 25(3): 652–661; (d) and (e)

from Suthers RA, Goller F, and Pytte C (1999) The neuromuscular

control of birdsong. Philosophical Transactions of the Royal Soci-

ety of London, Series B: Biological Sciences 354(1385): 927–939;

(f,g) Schematic of loci of song production and plasticity. (f) Flow of

activity: HVC drives RA, which drives muscle contraction via

motor neurons in a feedforward path. In addition, connectivity

within HVC and RA is recurrent. Dashed lines represent weights

that are likely to be plastic; solid lines with arrows designate

weights assumed to be fixed. HVC, with its recurrent connectivity,

produces the high-level sequential drive. Motor neurons, desig-

nated by motor neuron pools m1 andm2, sum the activity from RA

((g), middle eight traces) to generate drive for the muscles

((g), bottom two traces). Sequence learning involves plasticity in

the recurrent HVC connectivity. In this simple picture, sensori-

motor learning is dissociable from sequence learning and involves

plasticity in the HVC–RA and perhaps RA–RA connections.

Birdsong Learning 229

rates more consistent with experimental data, but thisrequired careful fine-tuning of parameters like synap-tic strengths. In contrast, the model with intrinsicbursting was quite robust to changes in parameterslike synaptic strengths.

Superficially, the Jin et al. model may seem incon-sistent with previous experiments in vitro, in whichsomatic current injection did not trigger intrinsicbursting in RA-projecting HVC neurons. However,one can imagine two scenarios in which this inconsis-tency is resolved. First, somatic current injectionmight be ineffective for revealing intrinsic burstingthat is caused by a dendritic calcium spike. Thisdepends on the strength of coupling between thesoma and dendrite and on the voltage threshold fora dendritic spike. Second, the intrinsic properties ofRA-projecting HVC neurons appear to be differentin vitro and in the brain of a singing bird. For exam-ple, the peak firing rates observed in vitro are never ashigh as those observed during song. Possibly, HVCneurons in vitro lack some neuromodulator secretedduring song, and this neuromodulator could be nec-essary for intrinsic bursting. Given the findings of Jinet al. regarding robustness, it seems important to lookagain for intrinsic bursting of HVC neurons.

Directionality of excitatory connectivity In the syn-aptic chain models mentioned earlier, there are excit-atory synapses from each group to the next group inthe chain. Unidirectionality is a fundamental feature.While the synaptic chain model is intuitively plausi-ble, it is important to subject it to experimental test-ing and to contrast the model’s predictions with thoseof competing models. For example, one can imagine abidirectional synaptic chain in which there are excit-atory synapses between adjacent groups in the chain,but the synapses go in both directions, ‘forward’ and‘backward’ (Figures 3(c) and 3(d)).

In both the unidirectional and bidirectional mod-els, it is assumed that song is initiated by activationof the first group in the chain. This causes theentire chain to be activated sequentially in the for-ward direction (Figures 3(c) and 3(d), red ‘electrode’).Both models can reproduce the sequential activitypatterns observed in HVC during singing. However,

20 ms

Time

100 ms

HVC1

HVC2

HVC3

HVC4

HVC5

1

c

a b

d

2 3 4 5 6 7

1 2 3 4 5 6 7

Figure 3 (a,b) Propagation of bursts in synaptic chain models of

high-level vocal center nucleus (HVC). (a) Neurons in the model

are not intrinsically bursting. Activity propagates due to synaptic

chain connectivity, and the burst of spikes in each neuron is due to

sustained synaptic drive from the network. Spike frequency during

the burst is �10 times lower than in nucleus robustus archistria-

talis (RA)-projecting HVC (HVCRA) neurons. Adapted from Li

M and Greenside H (2006) Stable propagation of a burst through

a one-dimensional homogeneous excitatory chain model of song-

bird nucleus HVC. Physical Review. E, Statistical, Nonlinear, and

Soft Matter Physics 74(1.1): 011918. (b) HVCRA neurons are

assumed to possess an intrinsic mechanism for burst generation.

A synaptic chain network can robustly produce stable sequences

of multispike neural activation. Adapted from Jin DZ, Ramazano-

glu F, and Seung HS (2007) Intrinsic bursting enhances the

robustness of a neural network model of sequence generation

by avian brain area HVC. Journal of Computational Neuroscience

23(3): 283–299. (c,d) Unidirectional synaptic chain versus bidirec-

tional connectivity. (c) In a synaptic chain, connections between

different groups of neurons project in a predominantly ‘forward’

direction. Activity initialized at group 1 (red ‘electrode’) will drive

sequential activity up to the end of the chain, at group 7. Stimula-

tion of the last group of the chain (blue ‘electrode’) will produce no

propagating activity or sequence playback. Activation of a group

near the middle of the chain will generate playback of a fragment

of the original activity sequence. The resulting vocalization should

correspond to normal song, but beginning midsong or midsyllable.

(d) Bidirectional synaptic connectivity. Activity initialized at group 1

(red ‘electrode’) will produce a unidirectional sequence of activity

propagating from beginning to end, as in the synaptic chain.

Stimulation at the end of the chain (blue ‘electrode’) will lead to

sequence playback in the reverse direction, from group 7 to 1. The

resulting vocalization should be the song or a syllable sung in

reverse. Stimulation of a central group in the chain will lead to

activity propagation in both forward and backward directions. The

resulting vocalization should be a superposition of the song (or a

syllable) sung forward and in reverse, starting in the middle.

230 Birdsong Learning

the bidirectional model can also produce other kindsof activity patterns that the unidirectional model can-not (Figures 3(c) and 3(d)), which could be used as thebasis of experimental tests that distinguish betweenthe two models.In these tests, activity in the HVC network would

be initiated by some means, and the subsequentresponse to stimulation would be observed. If the lastgroup in the chain is stimulated (Figures 3(c) and 3(d),blue ‘electrode’), there would be no subsequent activityin the unidirectional model. In the bidirectional model,the activitywould propagate backward along the chainand would result in a sequence of motor commandsthat is reversed from normal song. Another experimentwould be to stimulate an intermediate group in thechain (Figures 3(c) and 3(d), green ‘electrode’). Accord-ing to the bidirectional model, subsequent activitywould simultaneously propagate both forward andbackward along the chain. (Note that the playbackwill not continue from the forward to the backwarddirection, or from the backward to the forward direc-tion, because of the existence of a refractory period foractivation. This point is well known in the theory ofexcitable media.) In the unidirectional model, activitywould propagate only forward.While such experiments may sound straight-

forward, they are actually technically demanding. Sofar no topographic organization of RA-projectingHVC neurons has been detected. In other words, thelocation of a neuron in HVC does not seem to berelated to the time at which it is activated during asong motif. If coactive neurons were co-located inHVC, it would be relatively straightforward to acti-vate a hypothetical group in the synaptic chainmodelsby stimulation through a microelectrode. Instead, theneurons of a hypothetical group appear to be scatteredthroughout HVC. Therefore, a group would be bestidentified through optical imaging of activity, and beststimulated through optical means also.So far, no one has attempted to precisely stimulate a

group of HVC neurons. However, cruder perturbationexperiments have been performed. For example, HVCactivity during song has been perturbed by stimulatingRA, which leads to retrograde effects on HVC. Stimu-lation leads to skips in motif sequencing, but thesubsequent vocalization is a recognizable fragment ofthe motif sung in the forward direction, beginning at adifferent positionwithin the motif. This is more consis-tent with the unidirectional model than with the bidi-rectional model. But more-conclusive tests will requireprecisely controlled stimulation of activity in HVC.A more direct way of testing the synaptic chain

models is to check their predictions concerning thesynaptic connectivity of HVC. Since the real HVC is

no doubt ‘messier’ than in these idealized models,the predictions should be phrased probabilistically.Suppose A and B are twoRA-projectingHVC neuronsthat are activated in succession. In the unidirectionalmodel, the probability of a connection from A to B isexpected to be higher than that of a connection fromB to A or of a connection between two neurons chosenat random. (Alternatively, the predicted asymmetry ofcoupling between A and B might be reflected in syn-aptic strength rather than in the presence or absence ofconnections.) In the bidirectional model, the probabil-ity of a connection from A to B is equal to the proba-bility of a connection from B to A.

Testing this kind of prediction could be done bycomparing physiological and neuroanatomical mea-surements. Optical imaging could be used to find twoneurons A and B that are activated in succession. Thenoptical stimulation could be used to see whether andhow they are connected to each other. Alternatively, acompletely neuroanatomical approach would be toreconstruct the entire connectivity ofHVCusing serialsection electron microscopy, look for a chain withinthis connectivity, and determine its directionality.

The role of inhibition The synaptic chain modelsabove were based on synaptic excitation, withoutreference to the role of synaptic inhibition. Inhibitionis often included in such models to globally ‘dampen’activity in a temporally nonspecific way and canenhance the stability of sequential activity in theexcitatory neurons. To implement such a global inhi-bition effect, each inhibitory interneuron receivesexcitatory input from a large random set of projectionneurons, and each projection neuron in returnreceives synapses from the interneurons. If the bursttimes of the projection neuron population weredistributed across the entire song motif, then eachinterneuron would receive roughly continuous driveand would spike continuously.

Consistent with this picture, the temporal firingpatterns of inhibitory interneurons in HVC aremuch less specific than those of the HVCRA neurons:an inhibitory interneuron is typically active through-out most of the motif. This suggests that the connec-tivity of inhibitory neurons is also much less specificthan that of excitatory neurons.

Sequence Learning

Hand in glove with the question of what intrinsicneural properties and network architectures couldunderlie the stable propagation of activity in HVC isthe challenge of understanding how such architecturescould emerge from initially less-structured networks.

Birdsong Learning 231

Intuitively, associative (Hebbian) learning rulessuch as spike-timing-dependent plasticity (STDP)seem ideally suited to produce the kinds of asymmet-ric connectivity characteristic of synaptic chains: ifneuron A fires before B, the connection from A toB is strengthened, but that from B to A is weakened. Ifa naı̈ve network is trained with this rule for synapticplasticity on a set of input patterns presented sequen-tially, one would expect network weights to encodethe temporal correlations in the training patternsand later to autonomously reproduce the same long,stable chains of neural activity when stimulated.But contrary to intuition, past modeling studies

show that associative learning rules are not sufficientto configure networks to autonomously generate longneural sequences. Networks trained in this way usu-ally require finely tuned excitatory and inhibitoryconnectivity that is continuously renormalized to bal-ance network activity during learning, as well asadditional pacemaker inputs to propagate activityfrom one time step to the next. The situation is evenworse if the network does not have access to traininginputs that are themselves sequential. In such cases,the best response even with fine-tuning of parametersis the formation of extremely short chains (fewer thanfive steps) using thousands of neurons.The reason for this is that associative learning

rules do not balance activity equitably across space(neurons) and time; instead, they tend to amplifyinequities. For example, a neuron that by chancereceives strong synapses from its inputs will fireoften. Because of its frequent firing, it will tend torapidly form strong outgoing synaptic connections aswell. As a result, activity tends to accumulate arounda few overly active neural hubs, producing epilepticor synchronous burstlike network states rather thanlong, orderly chains. Empirically, these problemsremain unsolved with the imposition of local con-straints on the maximum strengths of individualsynapses. For these reasons, the question of howplasticity rules that are local in both space and timecan impose global constraints on activity balanceover all space and all time is nontrivial.Two independent models provide a conceptual

framework for how associative learning rules mustbe augmented to allow initially unstructured networksto self-organize and generate stable, long sequencesof neural activation. Both models use STDP as theprimary activity-dependent mechanism for synapticchange. But crucially, in addition to STDP, both mod-els rely strongly on competitively enforced synapticbounds that are not computed on the level of singlesynapses but are triggered by neuronwide constraints.Although the rules governing synaptic remodeling are

not local to the individual synapses, they are localto single neurons. Remarkably, these local neuralbounds nevertheless serve to uniformly and globallydistribute activity throughout the network in bothspace and time.

The qualitative nature of these synaptic bounds,and how they are triggered and enforced, is wherethe two models diverge. The model of Jin and Junassumes that the formation of a finite number ofstrong connections from a neuron triggers axonremodeling and elimination of weak synapses. Fieteet al. propose two separate schemes based on neuralresource constraints: one assumes that the productionrate of synapse-building resources is limited,so potentiation in one synapse triggers a uniformheterosynaptic weakening of all other synapses atthat neuron. The other scheme assumes a cap on thetotal resources a neuron can devote to synaptic main-tenance; therefore all synapses at a given neuronundergo uniform weakening whenever STDP drivestheir summed synaptic strength to exceed the cap.

Both models make predictions for the rules gov-erning synaptic plasticity within HVC and for thestatistics of the resulting connectivity within HVC.The Fiete et al. model produces several sequencesof different lengths, with a definite distribution ofchain lengths. Determining whether this distributionmatches syllable length distributions in birdsong couldindicate whether songbird syllables attain their specificlengths largely due to bottom-up (low-level network)properties or top-down (behavioral, evolutionary, orrespiratory) constraints.

Sensorimotor Learning of the Motor Map

The main goal-directed aspect of learning in zebrafinches takes place during the sensorimotor phase,when the juvenile learns to match, through auditoryfeedback and template comparison, its own string ofvocalizations with its memorized mental copy of thetutor song. Because the goal-directed aspect of songacquisition is so striking and because natural goal-directed learning behaviors that are easy to quantifyare rare, several models have focused on this aspect ofsong learning. Yet the microscopic synaptic plasticityrules that underlie such learning remain experimen-tally undiscovered and theoretically debated. Predic-tions from existing theoretical or computationalmodels can serve to greatly narrow, in a principledway, the dauntingly large number of possible plastic-ity paradigms to investigate in experiment, and theyare now specific enough to be tested in detail inslice preparations of the song system. (Experimentaltesting of the proposed paradigms is contingent on

232 Birdsong Learning

the identification of a reinforcement or critic signal,which is widely expected to exist but has not yetbeen found.)Existing models of goal-directed sensorimotor song

learning share the essential schema of the cartoonmodel (Figure 2) for the roles assigned to the dif-ferent premotor nuclei: HVC produces a high-levelsequence, and sensorimotor learning involves mappingthat abstract sequence into the correct vocal motorsequence through plasticity in the HVC–RA synapses.The primary differences between models lie in theroles assigned to the AFP (Figure 4), in the particularrules for how activity triggers plasticity in HVC–RAsynapses, and in their levels of biophysical realismand detail. The plausibility of different assumed rolesfor the AFP can already begin to be evaluated withexisting electrophysiological and lesion results.Early experiments showed that lesioning the ter-

minal lateral magnocellular nucleus of the anteriornidopallium (LMAN) of the AFP, which projects backinto the premotor pathway, abolishes the ability ofjuvenile songbirds to learn song. Moreover, the imme-diate and lingering effect of the LMAN lesion, besidesstoppage of song learning, is to dramatically reduce thelarge trial-to-trial variability of juvenile song. Theseresults suggested that the role of LMAN in songlearning is possibly to drive vocal experimentationthrough perturbation of the premotor song pathway.An early, and in several respects prescient, com-

putational model is based in part on these observa-tions. The model applied the specific algorithm ofweight perturbation (with momentum) from the rein-forcement learning literature to suggest that LMANdirectly generates small static weight changes in theHVC–RA connections (Figure 4(a)). According tothe model, these LMAN-driven static weight changeslast for the duration of one song rendition, afterwhich they are selectively retained, based on correla-tion with a song evaluation (reinforcement or critic)signal. The reinforcement (critic) signal is a functionof the match between the actual and desired (tutor)songs. Because the weight perturbation rule has beenproven by others to perform gradient climbing onthe scalar reinforcement signal, it will improve songperformance and do so independent of most parametervalues.Numerical simulationof a small networkmodelof the birdsong network, consisting of rate-based unitsor neural assemblies with time ‘coarse-grained’ intosyllable-long chunks, has demonstrated how this mayhappen. Experimental studies have supported severalfeatures of the model, including the high-level schemaof actor–critic reinforcement learning (Figure 5), therole of perturbations in song learning, and the role ofthe AFP in driving perturbations.

HVC

RA

LMAN

HVC

HVC

LMAN

LMAN

RA

RA

Spatially varying, temporally static perturbation andspatially uniform reinforcement

Spatially uniform reinforcement

Spatially and temporally varying perturbation

a

b

c

Figure 4 Proposed roles for lateral magnocellular nucleus of the

anterior nidopallium (LMAN) in sensorimotor learning. The dashed

line indicates a plastic synapse between high-level vocal center

nucleus (HVC) and nucleus robustus archistriatalis (RA) neurons.

(a) The hypothesis of Doya and Sejnowski: LMAN must produce

different, independent perturbations in each HVC–RA weight ori-

ginating from different HVC neurons. These perturbations must be

static over the duration of one song rendition but vary across

renditions. LMAN must also broadcast a separate, spatially

uniform reinforcement signal to the premotor pathway. (b) The

hypothesis of Troyer and Doupe. LMAN’s sole role is to provide

a spatially uniform reinforcement signal to the premotor pathway.

Learning is associative (Hebbian). (c) The hypothesis of Fiete and

colleagues: LMAN must inject different perturbations to different

RA neurons. The perturbations are excitatory synaptic inputs to

RA and vary in time over one song rendition. (A spatially uniform

reinforcement signal is assumed to arrive at RA from elsewhere

in the network.)

Birdsong Learning 233

What are the biophysical assumptions and implica-tions of this model? According to the model, boththe song evaluation (reinforcement) signal and theperturbative drive are generated in the AFP, and deliv-ered to the premotor pathway by LMAN. This meansthat LMAN must carry and convey two independentsignals, for perturbation and reinforcement, to theHVC–RA synapses (Figure 4(a)).The model is based on batch mode weight pertur-

bation, and it assumes that LMAN can somehowdirectly perturb the weights of the HVC–RA syn-apses in a way that the perturbations vary from songrendition to rendition but are static over the courseof a rendition. Moreover, for the weight perturbationscheme to explore enough of the motor space forsong learning, HVC–RA synapses that originate

from different HVC neurons must receive different,and independent, perturbations from LMAN. Thereare a few experimental inconsistencies with theserequirements in the birdsong system. First, single-and multiunit activity in LMAN varies rapidly duringa song motif, without an unusually large transient atthe beginning. It is therefore difficult to imagine howLMAN neurons could provoke an instantaneous per-turbation in the HVC–RAweights at the beginning ofa song rendition, which is held constant for the rest ofthe rendition. (It may be possible for weight perturba-tions to be driven by slow neuromodulatory action ofLMAN onto the HVC–RA synapses; within themodel, reinforcement would then have to bemediated either by ordinary synaptic transmissionfrom LMAN to RA or through a second neuromodu-lator released by LMAN neurons into RA.) Second,each RA neuron receives an estimated 1000 inputsfrom up to 200 HVC neurons but only approximately20–50 inputs from LMAN. Thus, it is unlikely thatsynapses from different HVC neurons could receiveindependent perturbations to fulfill the requirementsfor weight perturbation.

Finally, Doya and Sejnowski found that learningwith purely random inputs from LMAN was tooslow, with poor convergence to the tutor song, eventhough the simulated network was far smaller thanthe songbird motor pathway and even though a songmotif consisted of very few time steps (each timestep represented an entire syllable). Therefore, theyassumed that LMAN activity itself evolves by rein-forcement learning on the same global reinforcementsignal, so that its output is composed of a term pro-portional to the true HVC–RA weight gradient,superimposed on a random component. With thisassumption, the AFP must perform at least threefunctions: (1) generate and carry random perturba-tion signals whose dimension might be as large as thenumber of HVCRA neurons, (2) generate and carry atrue gradient estimate after correlating its randomactivity with reinforcement, and (3) function as acritic, generating and carrying the reinforcement sig-nal. With these assumptions, the model indeed learns;however, the AFP is assumed to possess these capabil-ities de facto, without modeling how the necessarycomputations are performed or how the resultingsignals are carried by LMAN and delivered to thepremotor pathway, leaving crucial questions unre-solved. Whether the AFP actually possesses these cap-abilities is an experimentally open question.

A model proposed by Troyer and Doupe in 2000suggests that plasticity in HVC–RA synapses andRA–RA synapses may be largely associational orHebbian in nature. According to the model, plasticityis gated by a reinforcement signal, which LMAN

Critic Experimenter

Actor

a

b

HVC

LMANRA

Plastic?

Empiric?R

Figure 5 The empiric synapse hypothesis. (a) The empiric syn-

apse hypothesis for goal-directed learning consists of three parts:

The actor network, which is responsible for performing the desired

task and has plastic synapses, and two of its inputs, one an

experimenter, whose role is to drive perturbations in the actor

through empiric synaptic inputs, and the other a critic, which

provides through a reinforcement signal a scalar assessment of

the performance of the actor on the desired task. Actor neurons

are hypothesized to correlate experimenter-driven perturbations

with the reinforcement signal to derive a signal for synaptic

change, in the direction of the gradient of the reinforcement signal.

(b) In the song system, the actor network is hypothesized to be the

premotor pathway including high-level vocal center nucleus (HVC)

and nucleus robustus archistriatalis (RA) and plastic HVC–RA

synapses. The experimenter is hypothesized to be the AFP.

A critic-generated reinforcement signal (R) is widely assumed to

exist in the song system, although its neural identity is unknown.

Adapted from Fiete IR, Fee MS, and Seung HS (2007) Model of

birdsong learning based on gradient estimation by dynamic per-

turbation of neural conductances. Journal of Neurophysiology 98:

2038–2057.

234 Birdsong Learning

delivers to the premotor network (Figure 4(b)). Theassociational rule also includes a subtraction term forsynaptic weakening, bringing it closer to reinforce-ment learning rules, but the specific form used is notproven in the literature to guarantee improved perfor-mance. As a result, there are no general assurances ofconvergence or stability, even in networks of simplifiedrate-based neurons. The model requires fine-tuning byhand of several homeostatic mechanisms to maintainstability during learning. The authors demonstratedthrough numerical simulation in rate-based networksthat under particular parameter settings, with a smallset of RA assemblies, HVC–RA connections, and afew syllable-long time steps, the network can learn tomatch a desired output.A strength of this model is that, unlike others in

the literature, it stresses the importance of lateralconnections within RA, which, when subject to thesame plasticity rules as HVC–RA synapses, formassemblies that are capable of pattern completioneven with partially correct inputs, to generate ‘good’output syllables. The AFP’s only role in the model isto generate and convey a spatially uniform reinforce-ment signal to RA (Figure 4(b)). However, in thezebra finch, LMAN activity is spatially inhomoge-neous (neural activity is not strongly correlated acrossthe nucleus), and connections to RA are not highlydivergent, which implies that different RA neuronsare likely to receive fairly uncorrelated inputs fromLMAN rather than a uniform signal. As formula-ted, the model cannot explain why, in experiments,LMAN lesions reduce song variability. Experimentsfail to reveal qualitative changes in LMAN activitywhen an adult bird’s auditory feedback is altered,which suggests that LMANmay not primarily conveyan error or match signal. (If LMAN does generatean error signal, but bases it on learned predictionsabout auditory feedback, it is possible that prolongedrather than transient exposure to distorted auditoryfeedback is necessary to produce changes in LMANactivity.)Subsequent informal proposals, not grounded in

numerical simulation or modeling, have espousedmore information-rich primary roles for LMAN insong learning. These include suggestions that LMANmay provide detailed instructive or supervisory errorsignals to RA. Unfortunately, these suggestions lackmechanistic proposals or models of how the AFPmight correctly compute and generate these informa-tion-rich, time-varying supervisory signals, leavingthe bulk of the computational burden of song learn-ing unsolved and at the door of the AFP. In reality itis possible that the AFP supplies such a supervisorysignal to RA, but no experimental or computationalstudies exist yet to suggest that it does.

A recent model of song learning by Fiete and colla-borators returns to a reinforcement learning frame-work and attempts to do two things: (1) propose that,and illustrate how, spiking RA neurons might performreinforcement learning by using rapidly time-varyingLMAN synaptic inputs that perturb RA membranevoltages (rather than using static synaptic pertur-bations of the plastic HVC–RA weights; Figure 4(c))and (2) show as a proof-of-principle that although it iswidely thought to be slow, reinforcement learningcan be fast enough to explain song acquisition ina full-scale model of the birdsong network with spik-ing, conductance-based neurons, delayed feedback,scalar reinforcement, and purely random independentperturbations.

The model assumes that LMAN perturbs the pre-motor pathway by injecting independent randomcurrents into different RA neurons (Figure 4(c))through ordinary glutamatergic synaptic transmission(Figures 5, 6(d), and (e)). This is consistent with recentstimulation and pharmacological inactivation experi-ments in LMAN, which show that LMAN helps drivesong variability through glutamatergic synaptic projec-tions to RA. The model requires at most as manyindependent LMAN inputs as there are RA neurons.The LMAN inputs are assumed to be time varyingthroughout song. Reinforcement is assumed to be com-puted and delivered from elsewhere in the brain, withlearning following the high-level, actor–critic schema.Under these conditions, the derivation of the rules forhow activity should modify synaptic weights, and therules themselves, are quite different from weight–perturbation-like algorithms (Figures 6(d)–6(f)): eachHVC–RA synapse must somehow deduce an instruc-tion for change based on time-varying fluctuationsin the conductance of its postsynaptic RA neuronrather than simply decide whether to retain or discarddirect static perturbations of its own weight throughcorrelationwith the reinforcement signal. The resultingrule is guaranteed to robustly (without parameter tun-ing) improve song performance, even in recurrent net-works of conductance-based spiking neurons and withdelayed reinforcement.

The model is able to produce learning in a biologi-cally realistic network model of the song system,with spiking neurons and an impoverished, delayedreinforcement signal (Figures 6(a) and 6(b)). Learningtakes place online, and activity in the network evolveson the millisecond timescale, driven by 5–6ms burstsin the HVC inputs. Within 2000 iterations, the net-work output resembles the tutor song, which is arecording of an actual zebra finch song (Figure 6(c)).Fiete et al. have shown that learning time even withrandom LMAN input should scale well in the full-sized premotor network and may converge in far

Birdsong Learning 235

fewer iterations than sung by the typical zebra finch(�100 000 renditions) during song acquisition.The model’s central prediction is a specific paradigm

for the induction of bidirectional, heterosynaptic activ-ity-dependent synaptic plasticity in the premotor vocalpathway (Figure 6(f)) that should be possible to testin slice experiments.

The model above is consistent with a large bodyof experimental data in the song system and providesconcrete predictions for synaptic plasticity. How-ever, LMAN’s role in the model is to provide purelyrandom exploration. This assumption does not con-tradict the data but is a limited view of what theAFP, a complex neural pathway, might be capable

300

400

00

100

200

Iterations

Err

or

Tutor

c

b 75 ms

5kH

z

HV

CR

AM

otor

Time (ms)a

LMAN

HVC

R

LMAN

ΔW > 0 ΔW < 0 ΔW = 0 ΔW = 0

RAd

f

Empiric LMANsynapse

HVC

R

e

0

−60−48

−56

arb

units

mV

mV

10

R

Before After

30 50 70

500 1000 1500 2000

Figure 6 Empiric synapses: results and predictions. (a) Activity of two model high-level vocal center nucleus (HVC) neurons (top), two

representative nucleus robustus archistriatalis (RA) neurons (middle), and one of the motor output pools (bottom; black: tutor, blue:

network response) during song learning. (b) Spectrograms of the tutor song and the output of the model song network before and after

1200 iterations of learning, together with the respective sound pressure waves. (c) Mismatch between the network output and the tutor

song decreases relatively rapidly. Learning has reached an approximate asymptote by 1000 iterations. (d,f) Prediction for bidirectional

synaptic plasticity from the empiric synapse model. (d) Possible experimental design for testing the empiric synapse rule in the birdsong

pathway involves focused stimulation in HVC, intracellular recording in RA, and fiber bundle stimulation of the lateral magnocellular

nucleus of the anterior nidopallium (LMAN)–RA axons, together with control of the reinforcement signal (R). (e) Magnified schematic of a

single RA neuron and its inputs. Each RA neuron receives a few empiric synapses from LMAN and several from different HVC neurons; in

addition, RA receives a reinforcement signal. Blue (gray): active (quiescent) premotor inputs; green: empiric input; red: reinforcement

signal. (f) Protocol for induction of bidirectional synaptic plasticity in active HVC–RA synapses: Simultaneous stimulation of an HVC

neuron and the LMAN axon bundle, followed by administration of reinforcement, should lead to potentiation of all stimulated HVC

synapses.

236 Birdsong Learning

of. More-sophisticated strategies for AFP-deliveredperturbation may speed up learning. This avenue iswide open for discovery by theory and experiment.More generally, models of song learning based

on the AFP as a source of random or directed pertur-bations to the premotor system could be used ascomparative tools for studying the role of the basalganglia thalamocortical loop in mammalian sensori-motor learning.

Premotor Codes and Representations

Up to 30 000 neurons are devoted to song productionand learning in the zebra finch. Together, these neu-rons drive only eight to ten muscles that producesong. What are some of the functional requirementsand costs that determine why so many neurons arerequired for song production?The ultrasparse or unary coding of song by HVCRA

neurons (Figures 2(a)) exacts a large energetic andspace cost in terms of the number of neurons requiredto encode a given set of patterns. To produce a sequenceof 100 steps (L) with 200neurons per time, unarycoding requires 20000neurons. At the other extreme,a combinatorial codewith a downstreamreadout capa-ble of distinguishing between different combinatorialpatternswould require far fewer neurons, scaling as log(L) instead of L; intermediate possibilities wouldrequire intermediate numbers of neurons. What aresome of the properties of a unary code thatmight offsetits capacity costs and explain why it exists in HVC?A possible explanation for ultrasparse HVC

sequences could be mechanistic: recurrent networkscannot store many dense patterns because of increas-ing interference between patterns. Studies show thatsparsely active attractor networks can store morepatterns. Nevertheless, in these studies, ‘sparse’ isfar denser than unary, so the reason for ultrasparsecoding must lie elsewhere.Because oscine birdsong is learned, a natural place

to investigate the role of unary HVC codes is inlearning. There are actually two distinct roles thatunary codes could play in song learning.First, an analytical and numerical study by Fiete

et al. shows that unary coding within HVC enablesoptimally fast learning of the feedforward motormap from abstract sequences in HVC to sequencesof muscle activation. If each HVC neuron were to firetwice instead of once, at random times in a songmotif of fixed length, the fastest possible speed withwhich the input–output map can be learned is twiceas slow (Figures 7(a)–7(d)). There is an intuitiveexplanation of the underlying mathematical reasonfor this effect. Suppose an HVC neuron and itscorresponding HVC–RA synapses are active twice,

when the tutor song has opposing characteristics(e.g., pitch or amplitude), at those two times. Thesynapses will be pulled in opposing directions to bestmatch the tutor song (Figure 7(c)) at each time.Because the learning task is realizable, there is somevalue of synaptic strength that achieves the desiredoutput, and in the end this value will be reached.But the conflicting demands on synapse strength atthe two times in the motif make approach to thesolution with a hill-climbing rule slower, even if theparameter controlling learning step size is optimized toproduce the fastest possible learning without networkinstability.

Yet answers to why questions, however compel-ling, risk being just-so stories unless they generatespecific testable predictions. The work describedabove argued that unary codes may exist in HVCbecause of their utility for fast song learning. How-ever, single-unit HVCRA activity and unary codeshave been observed only in adults. Thus, the centralprediction – which if false would invalidate thehypothesis – is that activity in HVCRA neurons,even if nonsparse initially (early in the sensoryperiod), should be unary near the onset of senso-rimotor learning in juvenile zebra finches instead ofemerging gradually after or toward the end of senso-rimotor learning.

The analysis leads to an additional prediction. Toaccommodate a larger song repertoire, HVC couldchange in two a priori equally likely ways: (1) keepHVC volume (number of neurons) fixed but increasethe number of bursts per neuron or (2) increaseHVC volume but maintain a unary code. Energeticsmight favor the first possibility. But according to thelearning speed analysis, if zebra finches are underpressure to learn their songs quickly, then unary cod-ing should be conserved, even at the cost of a largerHVC. This suggests that coding in HVC across zebrafinches and related songbirds with somewhat largerrepertoires (measured by number of unique syllables)should remain unary while HVC size scales with rep-ertoire size. Indeed, existing experimental findingscorroborate these predictions: HVC size scales withrepertoire size after correcting for overall brain size,while RA, LMAN, or other song nuclei do not displaysimilarly consistent relationships.

Second, analysis and simulation by Fiete et al. hasshown that if the HVC code is unary and if, duringa song rendition, performance is continuously evalu-ated and a reinforcement signal continuously delivered(albeit delayed), then the time taken to learn a songis independent of the length of the learned song.Because the HVC code is unary in zebra finches, thisprediction can be used in experiments (where learningtime is monitored for birds trained on tutor songs of

Birdsong Learning 237

different lengths) as an indirect probe of whether theelusive reinforcement signal in the song pathway isdelivered online or in batch mode. If the reinforcementsignal is evaluated and delivered only once, at the end

of song, then the time taken to learn a tutor songshould scale linearly with its length. Similarly, if theHVC code at each time consisted of a random patternof activations, with half of all neurons active per time

HVC1

a

b

c

d

e

f

g

HVC2

HVC3

HVC2

Time

Wj

Wi

Unary representation

RA

HVC3 RA

HVC1

HVC1

HVC2

RA

HVC1 RA

HVC2 RA

Random nonunary representation

Figure 7 Unary coding in high-level vocal center nucleus (HVC) helps speed the acquisition of the motor map. (a) The activity of

three hypothetical HVC neurons. The first two are each active only once in the song motif, but the third is active twice. (b) Hypothetical

pitch of the tutor song (black) and the pupil network (gray). (c) Direction in which synapses should change to reduce error between tutor

and pupil pitch. Synapses from HVC neuron 3 should be strengthened at the first and weakened at the second activity burst to improve the

tutor–pupil match. Such conflicting demands on the synapses of neurons that are active at two or more random times in a motif cause a

slowdown in learning speed. (d) Contours of iso-error in the learning surface. Learning a feedforward map in a network with unary coding

in the top layer is like learning on an isotropic cost surface and can be fast. Denser coding in the input layer produces correlations and

makes the learning surface anisotropic. To keep the error from diverging along the steep directions, the learning rate must be kept low.

As a result, best-case learning is slower than in the isotropic case. HVC activity tuned to acoustic features may be helpful for generalizable

learning. (e) If HVC neurons fired multiple bursts at selected points when the acoustic features of tutor song are similar, rather than at

random times, there would be no interference in the learning update, (f). If HVC neurons acquired such a tuning to features in the tutor

song, it could be easy for the bird to quickly reproduce a specific heard sound by activating the requisite sound-tuned HVC neuron. RA,

nucleus robustus archistriatalis.

238 Birdsong Learning

(dense code), learning time should scale with songlength (even if reinforcement is delivered online).Do other songbirds also display unary HVC repre-

sentations of song? At present, single-unit recordingsof RA-projecting HVC neurons are available onlyfrom zebra finches. Learning with a pure time codeis nongeneralizable: like one piece of magnetic tape,each pattern is used to produce song at only onespecific time. An entirely different set of HVC neu-rons and HVC–RAweights must be used and trainedto produce the same sound if it recurs elsewhere in themotif. Therefore, songbirds that rapidly acquire andimitate new songs (albeit after slowly acquiring afirst song) may use a different encoding strategy(e.g., tuning codes based on acoustic features) froma time-code (Figures 7(e)–7(g)). In addition, song-birds with vastly larger repertoires, regardless ofthe speed of song acquisition, may display differentHVC codes because of capacity constraints.

Conclusions

Advances in the acquisition of behavioral and anato-mical data and neurophysiological recordings fromawake, singing birds are rapidly turning the birdsongcircuit into an ideal system for unraveling themechanisms underlying motor control and learning.Theoretical models, by quantitatively validating oreliminating candidate explanations of network func-tion, have helped lay a groundwork for understand-ing the dynamical and coding principles that lead tosuch functionality. Theoretical models also provide afunctionally motivated set of predictions about neuralactivity, connectivity, coding, and plasticity that nar-row the choices of future experiments and are pres-ently ripe for experimental testing. Future experimentsand modeling in songbird species with more-flexiblelearning behaviors should greatly enhance understand-ing of generalizable motor learning and control.

See also: Bird Song Systems: Evolution; Birdsong

Learning: Evolutionary, Behavioral, and Hormonal Issues;

Birdsong: The Neurobiology of Avian Vocal Learning;

Hebbian Plasticity; Learning, Action, Inference and

Neuromodulation; Motor Sequences; Signal Production

and Amplification in Birds; Spike-Timing-Dependent

Plasticity Models; Spiking Neuron Models; Synaptic

Mechanisms of Learning; Vocal Communication in Birds.

Further Reading

Brainard MS and Doupe AJ (2000) Interruption of a basal ganglia–

forebrain circuit prevents plasticity of learned vocalizations.

Nature 404(6779): 762–766.Doya K and Sejnowski TH (1995) A novel reinforcement model of

birdsong vocalization learning. In: Tesauro G, Touretzky DS,

and Leen TK (eds.) Advances in Neural Information ProcessingSystems, pp. 101–108. Cambridge, MA: MIT Press.

Fiete IR, Burger L, Senn W, and Hahnloser RHR (2007) Enforcing

global coding constraints by synaptic competition – a model for

the formation of long ultrasparse sequences in the songbird.Submitted, 2007. (See also Society for Neuroscience Abstracts,

2005.)

Fiete IR, Fee MS, and Seung HS (2007) Model of birdsong learning

based on gradient estimation by dynamic perturbation of neuralconductances. Journal of Neurophysiology 98: 2038–2057.

Fiete IR, Hahnloser RH, Fee MS, and Seung HS (2004) Temporal

sparseness of the premotor drive is important for rapid learningin a neural network model of birdsong. Journal of Neurophysi-ology 92(4): 2274–2282.

Jin DZ, Ramazanoglu F, and Seung HS (2007) Intrinsic bursting

enhances the robustness of a neural network model of sequencegeneration by avian brain area HVC. Journal of ComputationalNeuroscience 23(3): 283–299.

Jun JK and Jin DZ (2007) Development of neural circuitry for

precise temporal sequences through spontaneous activity, axonremodeling, and synaptic plasticity. PLoS ONE 2(8): e723.

Li M and Greenside H (2006) Stable propagation of a burst

through a one-dimensional homogeneous excitatory chain

model of songbird nucleus HVC. Physical Review. E, Statistical,Nonlinear, and Soft Matter Physics 74(1.1): 011918.

Troyer TW and Doupe AJ (2000) An associational model of

birdsong sensorimotor learning I. Efference copy and thelearning of song syllables. Journal of Neurophysiology 84(3):

1204–1223.

Birdsong Learning 239