performance variability enables adaptive plasticity ‘crystallized’ adult song

Performance variability enables adaptive plasticity ‘crystallized’

adult song

E. C. Tumer and M. S. Brainard Nature, 2007

DIE 2014.06.03 by K. Sasahara

Question and Hypothesis

Why residual variability exists even in the most practiced skills?

Such variability is simply a ‘noise’ that the nervous system is unable to control or that remains below threshold for behavioral relevance.

Such variability enables trial-and-error learning, in which the motor system generates variation and differentially retains behaviors that produce better outcomes.

This paper experimentally tested the second possibility.

insight review articles

NATURE | VOL 417 | 16 MAY 2002 | www.nature.com 353

The sensory exposure required for tutor song memorization canbe surprisingly short. Nightingales can almost fully reproduce tapesof 60 songs that they have heard only once a day for 20 days27, andzebra finches can learn well with less than a minute of tutor songexposure per day28. In this respect, sensory learning of song resembles‘imprinting’, in which animals very rapidly and irreversibly learn torecognize an animal or object of critical behavioural relevance.

Closure of the sensitive period is affected by experienceThe sensitive period for song learning does not have a strict age limit.Rather, experience itself is centrally involved in closing the sensitiveperiod. For instance, songbirds tutored with only heterospecificsongs can incorporate new songs from their own species at a timewhen birds raised with conspecifics will no longer learn1,29. For somespecies, even more deprivation, such as raising birds in complete isolation, can result in adults that will still incorporate new song elements29,30. Thus, a lack of normal experience leaves the brain opento be shaped by the appropriate input for longer than usual. In mostcases, however, plasticity seems not to last indefinitely, even in theabsence of experience. Presumably, circuits poised to be shaped byactivity-dependent events ultimately stabilize in some state, even ifdriven only by spontaneous activity.

Attentional or motivational factors also influence the timing ofthe sensitive period. Birds will learn from live, countersinging tutorsfor longer than they learn from taped tutors31,32. Hormonal factorsmay be important as well, as manipulations that delay the onset ofsinging and decrease testosterone levels seem to extend the sensitiveperiod1,33.

Auditory neurons shaped by song experienceWe do not yet know where and how in the brain the memory of thetutor song is stored during the process of sensory learning, nor howthis memory is accessed during the evaluation of auditory feedbackthat guides vocal practice. However, the use of behaviourally relevantauditory stimuli has revealed neurons that clearly have been shapedby the individual bird’s unique auditory experience during songlearning. These ‘song-selective’ neurons, which are found through-out the adult male song system, respond more strongly to the soundof the bird’s own song (BOS), and in some cases to the tutor song,than to other equally complex auditory stimuli, such as conspecificsongs or BOS played in reverse or out of order16–18 (Fig. 4).

Although song-selective neurons reflect the individual bird’sexperience, it is not clear which aspects of that experience are

responsible for generating selectivity. In principle, these neuronsmight be shaped by the tutor song during sensory learning and/or byfeedback of BOS during sensorimotor learning. The former possibil-ity is especially intriguing; if tutor song selectivity arises during sensory learning, then this selectivity itself may be a manifestation ofthe tutor song memory. Moreover, such tutor-tuned neurons couldparticipate directly in the subsequent evaluation of auditory feedback during sensorimotor learning: as a bird practises his song,auditory feedback from those variants that more closely resemble thetutor’s song would be differentially effective in activating tutor-selective neurons. Hence, the degree of activation of these neuronsduring vocal practice could signal the degree of success in the youngbird’s attempts to mimic the tutor song.

This simple scenario, in which sensory learning generates tutor-selective neurons that can then guide feedback evaluation, faces aserious challenge. Developmental studies suggest that robust songselectivity does not emerge during sensory learning, but instead aris-es in parallel with the bird’s own motor production. Moreover, theemerging song selectivity is characterized by greater response to BOSthan to the tutor song, or by similarly strong responses to both thesesongs18,34,35. These observations are consistent with the possibilitythat the critical experience that shapes song selectivity is exposure tofeedback of BOS. The tutor song responses observed could arise simply because of similarity between the bird’s learned song and thetutor song to which it was exposed; neurons tuned to BOS would tendto respond well to the acoustically similar tutor song (Fig. 4c).

Because of this problem of acoustic similarity, and because thetutor song is only an indirect representation of what the bird has actu-ally memorized, the relative strength of neural responses to BOS andtutor song in normal adults cannot unambiguously reveal whichexperiences have shaped song selectivity35 (Fig. 4c). This problem hasbeen partly addressed by studying birds that were prevented fromproducing a good copy of the tutor song by denervating the vocalapparatus. These birds produce very abnormal songs, without theusual acoustic similarity to tutor song. Song-selective neurons inthese birds, at least in the AFP, develop sensitivity to the sound of theabnormal songs produced by the bird35. This indicates that BOSshapes song-selective neurons during sensorimotor learning.

But some AFP neurons in birds with deafferented vocal organs arestrongly responsive to the tutor song as well as to BOS, despite theacoustic differences between the two35. Thus, some song-selectiveneurons seem to reflect independently both sensory and sensorimo-tor learning. Such joint selectivity for BOS and tutor song could be a

Figure 2 Timelines for songlearning. a, In many seasonalspecies, such as the white-crownedsparrow, the sensory andsensorimotor phases of learning canbe separated in time. The initialvocalizations, or ‘subsong’,produced by young birds arevariable and generic acrossindividuals, akin to the babbling ofhuman infants. Subsong graduallyevolves into ‘plastic song’, whichremains highly variable from onerendition to the next, but also beginsto incorporate some recognizableelements of tutor songs. Plasticsong is progressively refined untilthe bird ‘crystallizes’ its stable adultsong. b, Zebra finches developrapidly, and their two phases oflearning overlap extensively. c, ‘Open learners’, such as canaries, can continue or recapitulate the initial learning process as adults.

Hat

ch Sensory Sensorimotor

Spring Autumn Spring

Crystallized

Sensitive period ends Subsong Plastic song

Hat

ch Sensorimotor

Sensorimotor Sensorimotor

Sensory

Sensory Sensory?

Crystallized

Crystallized Crystallized

25 60 Day 90

Hat

ch

Spring Autumn AutumnSpring SpringPlastic

Sparrows

Zebra finches

C anaries

© 2002 Macmillan Magazines Ltd

Birdsong

A complex learned behavior that requires the song system in the brain for precise motor control and auditory feedback.

There is a critical period for learning (i.e., only acquired early in life).

M. S. Brainard and A. J. Doupe Nature 2002

interfere with the performance of birds in tasks that require songmemorization and discrimination45–47.

Cellular and synaptic changes correlated with sensory learningTutor song memorization could be distributed across a number ofbrain areas, but because disruptions of the AFP affect learning, manystudies have focused on this circuit in the search for neural mechanisms underlying the sensitive period for sensory learning48.In zebra finches, the AFP and its connection to RA undergo numerous regressive changes by 60 days of age, when the sensitiveperiod closes in this species. The synapses from LMAN to the motorpathway decrease in number when HVc innervates RA49, and the ini-tially coarse topographic projection from LMAN to RA undergoesrefinement50. Elimination of connections is also prominent withinthe AFP itself: LMAN neuron spine density decreases between 25 and60 days of age51, and thalamic arbors in LMAN are pruned52. This isaccompanied by decreased N-methyl-D-aspartate (NMDA) recep-tors in LMAN53, faster NMDA currents at synapses from thalamus toLMAN30, and loss of activity-dependent synaptic potentiation anddepression at synapses within LMAN54.

These regressive changes could potentially underlie an experi-ence-dependent narrowing of song responsiveness as birds encode aparticular tutor song memory. But in zebra finches, the period of sensory learning also overlaps with the onset of vigorous singing,sensorimotor rehearsal and refinement of auditory selectivity forBOS (Fig. 2), making it difficult to specifically attribute any changesto sensory learning. In addition, the song system is still developingduring this time, such that many of the observed changes couldreflect developmental events that are independent of learning. So far,only a small number of observations have been tested and found to

insight review articles

354 NATURE | VOL 417 | 16 MAY 2002 | www.nature.com

useful property for song learning, which involves comparing thesetwo stimuli. There is as yet little evidence in the song system for the simpler idea of auditory neurons with strong suprathresholdselectivity to tutor song alone.

Neurons with responses to BOS playback in anaesthetized orsleeping animals do not always show these responses when birds areawake, indicating that the strength, and perhaps the nature, of audi-tory responses to sounds are ‘gated’ by the behavioural state of thebird36,37. In other sensorimotor systems, for instance locomotion inmammals or flying in insects, sensory responses related to a behav-iour are ‘gated’ by the motor activity that generates the behaviour38.That is, responses are diminished unless the animal is also engagedin the behaviour. Similarly, for songbirds as for humans, auditoryfeedback of self is available only when the animal is actually vocaliz-ing. Thus, anaesthesia or sleep may artificially open a gate that is normally operated by the act of singing. Ultimately, an understand-ing of the neural mechanisms for evaluation of auditory feedback ofBOS is likely to require recording neural activity when that feedbackis produced — that is, during singing.

Forebrain auditory areas and sensory responses Complex stimulus selectivity is also found in some auditory forebrain regions that provide input to the song system15,39–41. In particular, the high-level auditory areas (Fig. 3) known as the caudo-medial neostriatum (NCM) and the caudal portion of the ventralhyperstriatum contain neurons that show more immediate earlygene induction or neurophysiological activity in response to conspecific songs than to heterospecific songs39,40. For the most part,responses within these regions, unlike those within the song system,do not seem to be restricted specifically to BOS or tutor song stimuli.Hence, these forebrain regions may contribute to a general process-ing of conspecific sounds. However, one recent study found that,within NCM, some auditory responses seem to reflect the individualbird’s song-learning experience42,43. It therefore remains possiblethat some of the sensory learning of song occurs within this networkof auditory forebrain areas.

This conclusion seems especially plausible as many animals thatare not vocal learners, including some birds, are nevertheless capable of perceptual learning. Perceptual learning, including thatof tutor song, may rely on sensory processing pathways that are phy-logenetically widespread. In contrast, the sensorimotor componentof vocal learning, which has appeared only rarely, may have requiredthe evolution of specialized vocal areas such as the song system.

Assessing the functional role of brain regions in sensory learningLesion studies are problematic for identifying brain regions that arespecifically involved in the sensory phase of song learning. This isbecause the main assay for what a bird has memorized is the songthat the bird ultimately produces; any song abnormalities arisingfrom lesions are therefore difficult to attribute specifically to disrup-tion of sensory learning, as opposed to disruption of subsequentsensorimotor learning or song production. One attempt to circum-vent this problem in investigating the role of the AFP has been toreversibly inactivate the AFP nucleus LMAN (lateral magnocellularnucleus of the anterior neostriatum) during tutoring sessions, butnot during song rehearsal44. Song learning in these experimentalbirds is reduced relative to controls. However, the decrease is small,and the songs of treated birds are not isolate-like, as might be expect-ed if song memorization were completely prevented. Nonetheless,this experiment provides perhaps the most direct evidence ofinvolvement of a brain area in song memorization, and could beextended usefully to testing the role of other brain areas.

Another approach is to study the effects of lesions in purely per-ceptual tasks, where aspects of sensory learning can be measuredindependently of song production. Although such studies have notaddressed the issue of memorization of tutor song, they have found that lesions of song nuclei, including HVc and LMAN,

RA

DLM

Syrinx

HVc

Area X

LMANL

nXIIts

Respiratorymuscles

NIf

Dopamineneurons

Figure 3 Neural substrates for learning: the song system. The motor pathway (black)is necessary for normal song production throughout life, and includes HVc(abbreviation used as proper name) and the robust nucleus of the archistriatum(RA)9. RA projects to the tracheosyringeal portion of the hypoglossal nucleus (nXIIts),which controls the bird’s vocal organ or syrinx, and to nuclei involved in control ofrespiration during song7–9. Additional nuclei afferent to HVc, including the nucleusinterfacialis (NIf), are likely to be part of the motor pathway, but their role is less clear.HVc sends a second projection to the anterior forebrain pathway (AFP, red). The AFPincludes Area X, which is homologous to mammalian basal ganglia19,20, the medialnucleus of the dorsolateral thalamus (DLM), and the lateral magnocellular nucleus ofthe anterior neostriatum (LMAN; a frontal cortex-like nucleus). LMAN sends aprojection back into to the motor pathway at the level of RA. Like basal ganglia inother vertebrates, Area X is the target of strong midbrain dopamine projections19;LMAN, HVc and NIf also receive dopamine inputs (purple). The Field L complex is theavian primary forebrain auditory area and projects to a complex network of higherauditory areas14 (green), including the caudomedial neostriatum and caudal portionof the ventral hyperstriatum (not labelled). Auditory inputs likely enter the songsystem at the level of NIf and possibly HVc15.

© 2002 Macmillan Magazines Ltd

Methods

RESULTSA set of lightweight headphones was custom-fit to each bird in thestudy to generate online shifts in the pitch of auditory feedback (anexample of crystallized song from one bird in our study is shown inFig. 1a). A microphone in each bird’s cage relayed acoustic signalsthrough sound-processing hardware capable of generating arbitraryshifts in pitch. These pitch-shifted acoustic signals were then playedback through speakers in the headphones (Fig. 1b) with an averageprocessing delay ofB7 ms. Shifts in the pitch of auditory feedback andthe resulting changes in the pitch of song are both measured in units of‘cents’ (see Online Methods), where 1,200 cents corresponds to anoctave and 100 cents represents the same pitch interval as a semitone(approximately a 6% change in absolute frequency). A 100-centupward shift in pitch applied to several song syllables is shown inFigure 1c.We consistently found that shifting the pitch of auditory feedback

led to adaptive changes in song (that is, changes in the directionopposite the imposed pitch shift). We monitored changes in vocaloutput by repeatedly measuring pitch at particular times (or spectralframes; Fig. 2a) during song syllables. In the spectral frame chosen for

each syllable, we quantified changes in pitch by measuring changes ineither the fundamental frequency or the frequency of one of the higherharmonics (a harmonic feature, see Online Methods). In a typicalSpeakers

Head

Pitch shift

Microphone

A B C

100 ms

Fre

quen

cy (

kHz)

1

8

A A

B B

C C

Songoutput

Shiftedfeedback

a c

b

Figure 1 Technique for manipulating auditory feedback. (a) Crystallized songfrom an adult Bengalese finch. Spectrographic representation shows thepower at each frequency (color scale) as a function of time. Three harmonicfeatures are labeled A, B and C. (b) Each bird was fit with a set ofheadphones that housed a pair of speakers. A microphone in the cage (seeinset) provided input to online sound-processing hardware, which was used tomanipulate the pitch of song. Processed acoustic signals were then relayed tothe headphone speakers via a flexible cable (not shown in photograph) andplayed through the speakers. (c) An upward (+100 cents) shift in the pitch ofauditory feedback was introduced by the headphone system. For each of theharmonic features labeled in a, the left spectrogram shows the bird’s acousticoutput and the black triangle shows the frequency of the harmonic feature.The right spectrogram shows the pitch-shifted auditory feedback playedthrough the headphones and the red triangle shows the frequency of theharmonic feature in the shifted song. Black triangles are repeated next to thespectrograms on the right for comparison.

DB C5

10

Freq

(kH

z)

100 ms

3.3 4.10

0.5

2.2 2.6 7.5 8.8 3.5 4.1

1

–100

–50

0

50

100

Day14 24 67

Pitc

h(c

hang

e fr

om b

asel

ine,

cen

ts)

Mean pitch

Pitch of ABCD

BaselineShift Recovery

Freq (kHz)

+

–Freq

(kH

z)Fr

eq (

kHz)

A

a

b

c

d

Pro

babi

lity

Baseline Day 14 Difference

4.1

3.640 ms

D

8.8

7.740 ms

C

* ** *Figure 2 Vocal error correction driven by an upward shift in the pitch ofauditory feedback. (a) Baseline song of Bird 1 (mean spectrogram).Arrowheads above the spectrogram indicate the spectral frames(measurement times in each syllable) at which four harmonic features (A–D)were measured to quantify changes in the pitch of song. (b) Changes in pitchin response to a 100-cent upward shift (red line) in the pitch of auditoryfeedback and subsequent recovery back to baseline. Colored lines show themean ± s.e.m. change in pitch (measured in cents, see Online Methods) ofeach harmonic feature across time and the black line shows the mean changein the pitch of song (mean ± s.e.m. pitch change averaged across harmonicfeatures). After 14 d of shift exposure (gray box), unshifted auditory feedbackwas restored and the bird was monitored for an additional 10 d. Pitch wasalso measured on day 67 to assess any long-term changes. (c) Pre- and post-shift distributions of the frequencies of the harmonic features shown in a. Foreach feature, the probability distribution of frequencies during baseline(dashed lines) and day 14 (solid lines) differed significantly (* P o 10!5,one-tailed t test). Color conventions for each feature as in b. (d) Pitch shift-induced changes in mean spectral structure. Left, mean spectrograms forharmonic features C (top) and D (bottom) during the baseline epoch. Middle,mean spectrograms for features C and D on shift day 14. Right, differencespectrograms obtained by subtracting the baseline spectrograms from the day14 spectrograms.

928 VOLUME 12 [ NUMBER 7 [ JULY 2009 NATURE NEUROSCIENCE

ART ICLES

Sober and Brainard, Nature Neursci 2009

with the median decreasing from 91% to 21% by the third day. Thesedata provide the first evidence that auditory reinforcement signals candirect specific, adaptive changes to adult song. As a corollary, theydemonstrate that birds were able to effectively associate small naturalvariations in vocal production with differential outcomes.

Changes elicited by differential reinforcement occurred rapidly. Adetailed examination of one experiment reveals that fundamentalfrequency approached the asymptotic range within one day(Fig. 2a). Similarly, for six of seven birds, significant adaptive changesto fundamental frequency occurred within 7 h (Fig. 2b, ‘morning’)and for all birds significant changes occurred within the first day(Fig. 2b, ‘evening’). The median number of syllables sung withinthe first half-day and full day were 605 and 1,179, respectively. Byday 3, fundamental frequency stabilized at nearly asymptotic values(Fig. 2b).

The induced changes in fundamental frequency recovered rapidly.After at least 3 days of reinforcement, contingent white noise burstswere terminated. In every case, the fundamental frequency revertedtowards its original range (Fig. 2c). Hence, the nervous system retainsa representation of the initial song and both the capacity and impetusto return song towards its original structure in the absence of extern-ally imposed drive.

c

Fundamental frequency (kHz) Fundamental frequency (z-score)

Pro

babi

lity

0.2

0.1

0

2.2 2.3

2.2 2.3

Baseline

Day 3

Escape Hit

Pro

babi

lity

d

Escape Hit

a

100 ms

5 kH

z

a b c d e e

Mean

+1σ

–1σ

Pro

babi

lity

–5 –2.5 0 2.5 50

0.2

0.4


0.2

0.1

0

–5 –2.5 0 2.5 50

0.2

0.4

Pro

babi

lity

e Baseline

Day 3

Downwardescape

Upwardescape

Downwardescape

Upwardescape

f

250 ms

*Hitb Escape

*

5 kH

z

Trigger time

Figure 1 | Differential reinforcement can adaptively alter features ofadult song. a, Syllables (a, b, c, d, e, e) are normally produced with littlevariation. Three songs are shown for which the fundamental frequency of ‘a’spanned 2 standard deviations of the baseline distribution (SupplementaryRecording 1 contains corresponding audio files). b, White noise bursts(‘hits’) were targeted at higher pitched versions of ‘a’. c, Baselinefundamental frequency distribution for ‘a’, showing overall mean (triangle)and mean for escapes (line). d, Fundamental frequency distribution after 3days white noise. e, f, Baseline (e) and day 3 (f) distributions for sevenexperiments in which white noise directed either downward (red) or upward(blue) shifts in fundamental frequency. Fundamental frequency is expressedin units of the standard deviation of the baseline distribution (z-score, seeMethods).

-2 -1 0 1 2 3 4 5

2.2

2.3

2.4

Time since feedback on (days)

White noise on

a

c

White noise on

Bas

elin

e

Mor

ning

Even

ing

Day

3

Day 1feedback

on

{ Last

day

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

0

1

2

3

–1

Last

day

feed

back on

Day

3fe

edba

ck off

Whitenoise off

0

1

2

3

–1

b

Fund

amen

tal

frequ

ency

(kH

z)

Figure 2 | Adaptive shifts in fundamental frequency occur rapidly andrecover. a, Fundamental frequency of targeted syllables for one adult bird(age 334 days). Fundamental frequency progressively increased during thefirst day. b, Upward (blue) and downward (red) shifts in fundamentalfrequency. Filled symbols indicate significance (P, 0.05). After 3 days,fundamental frequency changed little, indicating that shifts had reachednear asymptotic values. c, Fundamental frequency for last day with whitenoise feedback on (‘Last day, feedback on’) and third day followingtermination of feedback (‘Day 3, feedback off ’).

NATURE |Vol 450 |20/27 December 2007 LETTERS

1241Nature ©2007 Publishing Group

White noise (Negative reinforcer)

No noise

Binary signal

F0

Adult Bengalese finch (11 birds) A computerized system, in real-time, detects the target syllable and deliver white noise if the pitch is higher (or lower) than the threshold.

Example of a crystallized adult song

Differential Reinforcement Can Adaptively Alter Features of Adult Song




c


Pro

babi

lity

0.2

0.1

0

2.2 2.3

2.2 2.3

Baseline

Day 3

Escape Hit

Pro

babi

lity

d

Escape Hit

a

100 ms

5 kH

z

a b c d e e

Mean

+1σ

–1σ

Pro

babi

lity

–5 –2.5 0 2.5 50

0.2

0.4


0.2

0.1

0

–5 –2.5 0 2.5 50

0.2

0.4

Pro

babi

lity

e Baseline

Day 3

Downwardescape

Upwardescape

Downwardescape

Upwardescape

f

250 ms

*Hitb Escape

*

5 kH

z

Trigger time


-2 -1 0 1 2 3 4 5

2.2

2.3

2.4


White noise on

a

c

White noise on

Bas

elin

e

Mor

ning

Even

ing

Day

3

Day 1feedback

on{ La

st d

ay

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

0

1

2

3

–1

Last

day

feed

back on

Day

3fe

edba

ck off

Whitenoise off

0

1

2

3

–1

b

Fund

amen

tal

frequ

ency

(kH

z)




Before

After

F0 = 2,281± 27 Hz Variation: ~ 1%

n=7

Hit rate: 91% → 17 %

n=1Even crystallized adult song has small natural variations. Hit rate was largely reduced in a few days. The negative reinforcement contingency directs either increases or decreases in pitch.

Adaptive Shifts in F0 Occur and Recover Rapidly




c


Pro

babi

lity

0.2

0.1

0

2.2 2.3

2.2 2.3

Baseline

Day 3

Escape Hit

Pro

babi

lity

d

Escape Hit

a

100 ms

5 kH

z

a b c d e e

Mean

+1σ

–1σ

Pro

babi

lity

–5 –2.5 0 2.5 50

0.2

0.4


0.2

0.1

0

–5 –2.5 0 2.5 50

0.2

0.4

Pro

babi

lity

e Baseline

Day 3

Downwardescape

Upwardescape

Downwardescape

Upwardescape

f

250 ms

*Hitb Escape

*

5 kH

z

Trigger time


-2 -1 0 1 2 3 4 5

2.2

2.3

2.4


White noise on

a

c

White noise on

Bas

elin

e

Mor

ning

Even

ing

Day

3

Day 1feedback

on

{ Last

day

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

0

1

2

3

–1

Last

day

feed

back on

Day

3fe

edba

ck off

Whitenoise off

0

1

2

3

–1

b

Fund

amen

tal

frequ

ency

(kH

z)




F0 approached the asymptotic range within one day (7hrs = 600 songs) (a). The nervous system retains the initial song representation and the capacity to return to it. (b, c)




c


Pro

babi

lity

0.2

0.1

0

2.2 2.3

2.2 2.3

Baseline

Day 3

Escape Hit

Pro

babi

lity

d

Escape Hit

a

100 ms

5 kH

z

a b c d e e

Mean

+1σ

–1σ

Pro

babi

lity

–5 –2.5 0 2.5 50

0.2

0.4


0.2

0.1

0

–5 –2.5 0 2.5 50

0.2

0.4

Pro

babi

lity

e Baseline

Day 3

Downwardescape

Upwardescape

Downwardescape

Upwardescape

f

250 ms

*Hitb Escape

*

5 kH

zTrigger time


-2 -1 0 1 2 3 4 5

2.2

2.3

2.4


White noise on

a

c

White noise on

Bas

elin

e

Mor

ning

Even

ing

Day

3

Day 1feedback

on

{ Last

day

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

0

1

2

3

–1

Last

day

feed

back on

Day

3fe

edba

ck off

Whitenoise off

0

1

2

3

–1

b

Fund

amen

tal

frequ

ency

(kH

z)




n=1 n=7

Changes are Restricted to Targeted Song Features

In principle, birds could escape white noise equally well by alteringthe fundamental frequency of only the targeted syllable or of largersegments of song. Nevertheless, we consistently found that changesto fundamental frequency were restricted exclusively to targetedsyllables (Fig. 3a, b, red symbols). Other syllables, even when theyoccurred within a few tens of milliseconds, did not change (Fig. 3a, b,blue symbols). Furthermore, the changes were specific to fun-damental frequency and did not affect other features such as dura-tion, volume and spectral entropy (Fig. 3c). Hence, even though theexperimentally imposed contingency between performance and feed-back was not revealed to the bird in any direct sense, the nervoussystem was able to detect and respond precisely to that contingency.

This specificity indicates an impressive capacity of the nervoussystem to modify discrete features of song independently. This isappropriate for vocal learning, where birds match their song torapidly varying features of an acoustic model. If modification ofone song feature generalized to causemodification of others, learningmight still proceed, but such interference would probably slow itsprogress26.

In theory, reinforcement signals can drive learning even at longlatencies to the actions that precipitate them15. For complex motor

skills, however, the nervous system might detect the contingencymore effectively at shorter delays. We tested the importance of delayby varying the time betweenmeasurement of fundamental frequency

c

Fundamentalfrequency

Duration Volume Entropy

–1

0

1

2

3

Mea

n sh

ift in

feat

ure

(z-s

core

)

Target

Control

b

Mea

n sh

ift in

fund

amen

tal

frequ

ency

(z-s

core

)

Mean time from target syllable (ms)

a Target


–400 –300 –200 –100 0 100 200 300 400

0

2

4

6

8

10

–4

–2

0

2

4

Freq

uenc

y (k

Hz)

–400 –300 –200 –100 0 100 200 300 400

Figure 3 | Changesarerestrictedtotargetedfeaturesofsong. a, Spectrogramillustrating analysed features for target (red bar) and control (blue bars)syllables of an individual experiment. b, Mean changes in fundamentalfrequency for target (red) and control (blue) syllables. Squares represent twoexperiments for song illustrated in a. Data from 3 additional birds (circles,triangles, diamonds) are shown without corresponding spectrograms. Filledand open symbols indicate experiments with upward and downward shifts infundamental frequency, respectively. c, Spectral characteristics other thanfundamental frequencywerenot altered for either target (red) or control (blue)syllables. Bars indicate mean6 standard deviation.

5 kH

z

100 ms+100 ms delay

Targetsyllable

Short delay

Targetsyllable

a

b

Detect timeTrigger time

d Shortdelay

+100 ms delay

–1

0

1

2

3

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

Shortdelay

+100 ms delay

0

0.2

0.4

0.6

0.8

1

Hit

rate

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

0 3 7 15 21 252.2

2.3

2.4

Time (days)Fund

amen

tal f

requ

ency

(kH

z)

c Feedback off Feedback on

Short delay +100 ms delay Short delay

Downwardescape

Downwardescape

Upwardescape

28

Figure 4 | Delayed feedback prevents adaptive pitch shifts.a, b, Spectrograms illustrating short delay (a) and 1100ms delay(b) feedback for one bird. c, Fundamental frequency (mean 6 standarddeviation) for this bird. Initially, short delay feedback directed a downwardshift in fundamental frequency. After recovery, the same contingency withdelayed feedback was ineffective. Subsequently, an upward shift was drivenusing a new short delay contingency. Shading indicates baseline61 standarddeviation. d, Summary of shifts in fundamental frequency and hit rate.Symbols indicate 4 birds subjected to both short delay and 1100ms delayreinforcement. Shifts were prevented when feedback was delayed.

LETTERS NATURE |Vol 450 |20/27 December 2007






c



–1

0

1

2

3

Mea

n sh

ift in

feat

ure

(z-s

core

)

Target

Control

b

Mea

n sh

ift in

fund

amen

tal

frequ

ency

(z-s

core

)


a Target


–400 –300 –200 –100 0 100 200 300 400

0

2

4

6

8

10

–4

–2

0

2

4

Freq

uenc

y (k

Hz)

–400 –300 –200 –100 0 100 200 300 400


5 kH

z

100 ms+100 ms delay

Targetsyllable

Short delay

Targetsyllable

a

b


d Shortdelay

+100 ms delay

–1

0

1

2

3

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

Shortdelay

+100 ms delay

0

0.2

0.4

0.6

0.8

1

Hit

rate

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

0 3 7 15 21 252.2

2.3

2.4

Time (days)Fund

amen

tal f

requ

ency

(kH

z)



Downwardescape

Downwardescape

Upwardescape

28




The nervous system can: detect and respond to the imposed contingency: Higher/lower pitch→white noise modify song features independently

Delayed Feedback Prevents Adaptive Pitch Shifts

Short delay (< 30ms)→Adaptive pitch shifts Long delay (+100ms)→No systematic changes Normally predictable timing is between premotor activity and sensory consequence (< 70ms)





c



–1

0

1

2

3

Mea

n sh

ift in

feat

ure

(z-s

core

)

Target

Control

b

Mea

n sh

ift in

fund

amen

tal

frequ

ency

(z-s

core

)


a Target


–400 –300 –200 –100 0 100 200 300 400

0

2

4

6

8

10

–4

–2

0

2

4

Freq

uenc

y (k

Hz)

–400 –300 –200 –100 0 100 200 300 400


5 kH

z

100 ms+100 ms delay

Targetsyllable

Short delay

Targetsyllable

a

b


d Shortdelay

+100 ms delay

–1

0

1

2

3

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

Shortdelay

+100 ms delay

0

0.2

0.4

0.6

0.8

1

Hit

rate

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

0 3 7 15 21 252.2

2.3

2.4

Time (days)Fund

amen

tal f

requ

ency

(kH

z)



Downwardescape

Downwardescape

Upwardescape

28








c



–1

0

1

2

3

Mea

n sh

ift in

feat

ure

(z-s

core

)

Target

Control

b

Mea

n sh

ift in

fund

amen

tal

frequ

ency

(z-s

core

)


a Target


–400 –300 –200 –100 0 100 200 300 400

0

2

4

6

8

10

–4

–2

0

2

4

Freq

uenc

y (k

Hz)

–400 –300 –200 –100 0 100 200 300 400


5 kH

z

100 ms+100 ms delay

Targetsyllable

Short delay

Targetsyllable

a

b


d Shortdelay

+100 ms delay

–1

0

1

2

3

Ada

ptiv

e sh

ift in

mea

nfu

ndam

enta

l fre

quen

cy (z

-sco

re)

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

Shortdelay

+100 ms delay

0

0.2

0.4

0.6

0.8

1

Hit

rate

Bas

elin

eD

ay 3

Day

5

Bas

elin

eD

ay 3

Day

5D

ay 8

Day

12

0 3 7 15 21 252.2

2.3

2.4

Time (days)Fund

amen

tal f

requ

ency

(kH

z)



Downwardescape

Downwardescape

Upwardescape

28




Incremental Adjustment of Threshold Drives Large Pitch Changes

and delivery of feedback. White noise typically was delivered within16–30ms. In this case, white noise overlapped with the targetedsyllable (Fig. 4a, ‘short delay’). For some experiments, white noisewas delayed by an extra 100ms. In this case, white noise started aftercompletion of the targeted syllable (Fig. 4b, ‘1100 ms delay’).Syllables targeted at short delay rapidly shifted and reached asymp-totic values after roughly 3–4 days. In contrast, the same syllables,when targeted with delayed feedback, exhibited no directed changeto fundamental frequency even over longer periods (Fig. 4c, d).Hence, even small delays profoundly attenuated adaptive responses.Likewise, in the 1100ms condition, there were no systematicchanges to syllables following the targeted syllable, for which whitenoise was delivered in a non-contingent manner (see SupplementaryFig. 3). Hence, rapid changes to fundamental frequency requiredboth that white noise was contingent on the structure of a syllableand that it occur at short delay. These data suggest that the vocalcontrol system may take advantage of normally predictable timingbetween premotor activity and resultant sensory consequences;normally, activity of premotor neurons influences song and resultantfeedback at latencies, 70ms (ref. 27). Therefore, the nervoussystem may not readily detect reinforcement contingencies whenfeedback is delayed beyond this range. Although such timing con-straints are not necessary for reinforcement learning, they potentiallyenable more efficient learning and may be a general feature ofsensorimotor systemswhere there is predictable delay between actionand consequence28.

Our results indicate that small natural variations present in adultsong can enable adaptive modification of targeted features. Thesechanges exceed the frequency discrimination threshold for birds25.Moreover, the reversion of fundamental frequency after terminationof feedback (Fig. 2c) indicates the nervous system is sensitive to thediscrepancy from the original song. Nevertheless, our reinforcementscheme constrained shifts in fundamental frequency to be similar inmagnitude to the initial range of variation (see Supplementary Fig.2). This raises the question of whether changes in adult song arerestricted to subtle fine-tuning, or whether more dramatic remodel-ling is possible. We tested this by progressively altering the reinforce-ment contingency so that continued escape from white noise wouldrequire progressively larger shifts in fundamental frequency (Fig. 5).We were able to incrementally drive large changes in fundamentalfrequency, such that syllables eventually were produced in a rangecompletely non-overlapping with the baseline range (SupplementaryMovie 1 demonstrates the salience of these changes). This findingimplies that following each incremental shift in fundamental fre-quency a new range of behaviourally relevant variation was estab-lished that enabled differential reinforcement of more extremevocalizations. The range of variation remained relatively constantover time (for example, see Figs 1, 4, 5). Hence, as learning pro-gresses, current performance is continuously surrounded by a ‘halo’of variation that enables continuous adaptive modification.

In summary, our results demonstrate that binary feedback signalscan drive rapid plasticity of normally stable adult song. In contrast toprevious studies in which disruption of feedback led to a deteriora-tion of song10–14, we show that differentially delivered feedback candirect precise, adaptive changes to adult song. The changes are precisein that they are restricted to targeted features. They are adaptive inthat birds alter those features systematically to escape disrupted feed-back. These data provide empirical support for models suggestingthat song might normally be shaped by reinforcement signals16–18.More generally, they indicate that the trial-and-error process of rein-forcement learning can efficiently guide adaptive modification ofhighly complex and tightly controlled motor skills.

The adaptive plasticity reported here also demonstrates that birdscan accurately associate small variations in fundamental frequencywith resultant consequences. In principle, variation could be mon-itored by sensory feedback (auditory or proprioceptive). Alter-natively, if variation is centrally generated, it could be monitored

internally by an efference copy of premotor activity17. Consistentwith this possibility, studies in both birds andmammals indicate thatsomemovement variability derives from central neural activity ratherthan the periphery7,8,29,30, suggesting that variability may be activelygenerated for motor exploration. Regardless of mechanism, ourdata indicate that natural variations present in crystallized adult songare not simply noise but rather can be exploited for trial-and-error

c

d

–2 0 2 4 6 8 10 12

2.3

2.4

2.5

2.6

Time since feedback onset (days)

Fund

amen

tal f

requ

ency

(kH

z)

2.3 2.4 2.5 2.6 2.70

0.1

0.2

Fundamental frequency (kHz)

Pro

babi

lity

White noise

a

b

100 ms

2.5

kHz

Baseline Day 13Day 13

–15 –10 –5 0 5 10 150

0.1

0.2

0.3

Pro

babi

lity

Fundamental frequency (z-score)

Baseline

Last day

–15 –10 –5 0 5 10 150

0.1

0.2

0.3

Pro

babi

lity


Figure 5 | Incremental adjustment of threshold drives large pitch changes.a, Changes to fundamental frequency for one experiment. Points indicatemean (6 standard deviation) on each day. Shading indicates threshold forescapes. b, Fundamental frequency distributions for baseline (black) and day13 (blue). Spectrograms show the average syllable for corresponding days.c, d, Baseline (c) and final (d) fundamental frequency distributions for threeexperiments with thresholds incrementally adjusted upward (blue) ordownward (red). Over 11–14 days, mean fundamental frequency (triangles)shifted by 7.0–10.5 standard deviations of the baseline distribution (226–376Hz). For comparison, vertical shaded regions show the range of meanchanges driven with a fixed reinforcement contingency.



Video from supplemental materials

n=1

(Cont.)

and delivery of feedback. White noise typically was delivered within16–30ms. In this case, white noise overlapped with the targetedsyllable (Fig. 4a, ‘short delay’). For some experiments, white noisewas delayed by an extra 100ms. In this case, white noise started aftercompletion of the targeted syllable (Fig. 4b, ‘1100 ms delay’).Syllables targeted at short delay rapidly shifted and reached asymp-totic values after roughly 3–4 days. In contrast, the same syllables,when targeted with delayed feedback, exhibited no directed changeto fundamental frequency even over longer periods (Fig. 4c, d).Hence, even small delays profoundly attenuated adaptive responses.Likewise, in the 1100ms condition, there were no systematicchanges to syllables following the targeted syllable, for which whitenoise was delivered in a non-contingent manner (see SupplementaryFig. 3). Hence, rapid changes to fundamental frequency requiredboth that white noise was contingent on the structure of a syllableand that it occur at short delay. These data suggest that the vocalcontrol system may take advantage of normally predictable timingbetween premotor activity and resultant sensory consequences;normally, activity of premotor neurons influences song and resultantfeedback at latencies, 70ms (ref. 27). Therefore, the nervoussystem may not readily detect reinforcement contingencies whenfeedback is delayed beyond this range. Although such timing con-straints are not necessary for reinforcement learning, they potentiallyenable more efficient learning and may be a general feature ofsensorimotor systemswhere there is predictable delay between actionand consequence28.

Our results indicate that small natural variations present in adultsong can enable adaptive modification of targeted features. Thesechanges exceed the frequency discrimination threshold for birds25.Moreover, the reversion of fundamental frequency after terminationof feedback (Fig. 2c) indicates the nervous system is sensitive to thediscrepancy from the original song. Nevertheless, our reinforcementscheme constrained shifts in fundamental frequency to be similar inmagnitude to the initial range of variation (see Supplementary Fig.2). This raises the question of whether changes in adult song arerestricted to subtle fine-tuning, or whether more dramatic remodel-ling is possible. We tested this by progressively altering the reinforce-ment contingency so that continued escape from white noise wouldrequire progressively larger shifts in fundamental frequency (Fig. 5).We were able to incrementally drive large changes in fundamentalfrequency, such that syllables eventually were produced in a rangecompletely non-overlapping with the baseline range (SupplementaryMovie 1 demonstrates the salience of these changes). This findingimplies that following each incremental shift in fundamental fre-quency a new range of behaviourally relevant variation was estab-lished that enabled differential reinforcement of more extremevocalizations. The range of variation remained relatively constantover time (for example, see Figs 1, 4, 5). Hence, as learning pro-gresses, current performance is continuously surrounded by a ‘halo’of variation that enables continuous adaptive modification.

In summary, our results demonstrate that binary feedback signalscan drive rapid plasticity of normally stable adult song. In contrast toprevious studies in which disruption of feedback led to a deteriora-tion of song10–14, we show that differentially delivered feedback candirect precise, adaptive changes to adult song. The changes are precisein that they are restricted to targeted features. They are adaptive inthat birds alter those features systematically to escape disrupted feed-back. These data provide empirical support for models suggestingthat song might normally be shaped by reinforcement signals16–18.More generally, they indicate that the trial-and-error process of rein-forcement learning can efficiently guide adaptive modification ofhighly complex and tightly controlled motor skills.

The adaptive plasticity reported here also demonstrates that birdscan accurately associate small variations in fundamental frequencywith resultant consequences. In principle, variation could be mon-itored by sensory feedback (auditory or proprioceptive). Alter-natively, if variation is centrally generated, it could be monitored

internally by an efference copy of premotor activity17. Consistentwith this possibility, studies in both birds andmammals indicate thatsomemovement variability derives from central neural activity ratherthan the periphery7,8,29,30, suggesting that variability may be activelygenerated for motor exploration. Regardless of mechanism, ourdata indicate that natural variations present in crystallized adult songare not simply noise but rather can be exploited for trial-and-error

c

d

–2 0 2 4 6 8 10 12

2.3

2.4

2.5

2.6

Time since feedback onset (days)

Fund

amen

tal f

requ

ency

(kH

z)

2.3 2.4 2.5 2.6 2.70

0.1

0.2

Fundamental frequency (kHz)

Pro

babi

lity

White noise

a

b

100 ms

2.5

kHz

Baseline Day 13Day 13

–15 –10 –5 0 5 10 150

0.1

0.2

0.3

Pro

babi

lity


Baseline

Last day

–15 –10 –5 0 5 10 150

0.1

0.2

0.3

Pro

babi

lity


Figure 5 | Incremental adjustment of threshold drives large pitch changes.a, Changes to fundamental frequency for one experiment. Points indicatemean (6 standard deviation) on each day. Shading indicates threshold forescapes. b, Fundamental frequency distributions for baseline (black) and day13 (blue). Spectrograms show the average syllable for corresponding days.c, d, Baseline (c) and final (d) fundamental frequency distributions for threeexperiments with thresholds incrementally adjusted upward (blue) ordownward (red). Over 11–14 days, mean fundamental frequency (triangles)shifted by 7.0–10.5 standard deviations of the baseline distribution (226–376Hz). For comparison, vertical shaded regions show the range of meanchanges driven with a fixed reinforcement contingency.



Large changes in pitch is achieved by an incremental differential reinforcement.

Current performance is surrounded by a ‘halo’ of variation that enables trial-and-error exploration for continuous adaptive modification.

n=3

Fixed case

Summary

Reinforcement with a binary feedback can direct precise, adaptive changes to crystallized adult song.

Residual variability in well learned skills is not entirely noise but rather reflects meaningful motor exploration that can support continuous learning and optimization of performance.

Graphon, Nature Neursci 2008