Speech & Music Perception

‘talking about music is likedancing about architecture’

Topics

• Tritone Paradox• Sometimes Behave So Strangely• Speech & Music: Syntax• Speech & Music: Semantics• Whistled Languages• Talking Drums• “Ringing my Phone”

Pitch Circularity: Shepard Tones• Shepard (1964) decoupled these dimensions

– Minimized tone-height cues• 10 octave-spaced sine wave partials; spectral envelope

– Subjects judged tone pairs’ pitch: ‘’ or ‘’– Determined by pitch class proximity

Scale heard as endlessly ascending!

Shepard spectral envelope

From Shepard, R. N., J. Acoust. Soc. Am., 1964

Pitch Circularity in MusicCompositional technique (Braus, 1995)

– Evident in Western art music by 1500’s– Baroque, Romantic, & 20th Century– Modern composers used Shepard tones

∞Reflects infinity-seeking compositionalaesthetic; climactic; builds tension

Ascending and Descending, M.C. Escher

Tritone Paradox• Judging Shepard

tone-pairs:• What happens w/o

proximity cues(tritone)?

• Consistent perceptswithin subjects

Tritone: examples• Systematic variation

between subjects– Geographic region

(dialects)– Pitch range of

speech– Mother-child

similarities

Tritone Paradox: Data Sheet• For each tone pair:

– ⇑ if the 2nd pitchsounds higher thanthe 1st

– ⇓ if the 2nd pitchsounds lower thanthe 1st

Tritone Paradox: Key 1• 1st pitch of

each tone pair• For each ⇓

record thenote name

• Count # eachpitch class

Tritone Paradox: Key 2• 1st pitch of

each tone pair• For each ⇓

record thenote name

• Count # eachpitch class

Tritone Paradox: Key 3• 1st pitch of

each tone pair• For each ⇓

record thenote name

• Count # eachpitch class

Sometimes Behave So Strangely• A repeated phrase of normal speech• Begins to sound like song• The musical reinterpretation persists indefinitely

Crying babies

The Studies:

• ProcessingSyntactic Relationsin Language andMusic: An Event-Related PotentialStudy (1998)

• Ani Patel, EdwardGivson, Jen Ratner,Mireille Besson, PhilHolcomb

• Neural Processingof a WhistledLanguage (2005)

• Manuel Carreiras,Jorge Lopez,Francisco Rivero,David Corina

Key Theoretical Issues

• Processing: domain-general vs.modularity

• Syntactic rules?Predictions/expectancy?

• What is language?

Structure of Music: Syntax• Syntax: set of rules for combining discrete

elements (e.g. notes & chords) intosequences

• Syntactic knowledge allows the brain toperceive a sequence of elements in termsof hierarchical relations

• Musical syntax enables musicalexpectations. E.g. is a specific chord thefinal chord or is there more to come?

• Music and language both have discreteelements that are organized according tosyntactic principles

• In language, syntax allows one tounderstand meaning from word order andfrom dependent relationships betweendifferent words– The dog bit the man.– The man bit the dog.– The man bit by the dog was upset.

Implicit Knowledge of Syntax• Experienced perceivers show implicit

knowledge of these principles in a varietyof ways, including the ability to detectstructural incongruities in novel sequences

• Agreement errors in language– “Our baby love his books."

• Wrong notes in music

Function of Syntax• Syntactic knowledge allows the mind to

perceive a linear sequence of elements interms of hierarchical relations that conveyorganized patterns of meaning

• In language, one meaning supported bysyntax is ‘who did what to whom’

• In music, one meaning supported bysyntax is the pattern of tension andresolution experienced as the musicunfolds in time

Tonal HierarchyI

IV V

VI II

VII III

High ProbabilityTransitions

Comparative Research• Recent experiments suggests an overlap

in the cognitive processing of syntacticrelations in language and music

• The ERP components evoked whensyntactic errors are encountered inlanguage and music are statisticallyindistinguishable

• Overlap in the neural areas involved inprocessing musical & linguistic syntax

• Processing of musical syntax has beenshown to activate 'language areas' of thebrain, including Broca's area, Wernicke'sarea

Comparative Research

Parallel Aspects of Language & Music

• Numerous parallels– Similar brain areas & cognitive processes for

musical and linguistic syntax– Semantic meaning is conveyed by both– music can facilitate the processing of words– Emotion is conveyed by the same acoustic

cues in both language and music

Tonal Pitch Space Theory• Lerdahl's Tonal Pitch Space theory (TPS)

concerns the perception of pitch in a musicalcontext

• We acquire highly structured mentalrepresentations of musical pitch via exposure totonal music

• E.g., musical keys, such as C-major, carry ahierarchy of importance, such that some pitchclasses are perceived as more central or stablethan others– the first (C) being the most stable, followed by the fifth

(G) and third (E) pitch class

• Chords built on this scale are alsoperceived in a hierarchy of stability, withchords built on the first, fourth and fifthpitch class being the most stable

• Entire musical keys are also perceived interms of an orderly set of distances fromeach other, as depicted by the 'circle offifths'

• provides a model for quantifying the tonaldistance between any two musical chords

• incorporates the distances of pitchclasses, chords and keys

• method for deriving tree structures– hypothesis for the perceived relations

between chords

Tonal Pitch Space

Syntactic Structure: Music & Language• the overlap in linguistic

& musical syntax is notat the level ofrepresentation– long-term structural

knowledge in adomain

• syntactic processing– operations

conducted on thatknowledge for thepurpose of buildingcoherent percepts

• Using the tree structure, one computes thedistance of each chord from the chord towhich it attaches in the tree

• Thus each chord is associated with anumerical distance value from anotherchord

• This distance plays an important role inpredicting the perceived ebb and flow oftension in musical sequences

• E.g., tension drops between two chords when the musiccomes to a harmonic resting point in a given key

• when chords enter a new key area, tension increases• The numerical predictions of TPS can be compared to

'tension profiles' produced by listeners who rateperceived tension over time in musical passages

• Such experiments provide support for TPS, and suggestthat listeners do in fact hear relations between chords ina hierarchical rather than a purely sequential manner

TPS Continued

Convergence of syntacticprocessing in language and music

• structural integration is a key part ofsyntactic processing– mentally connecting each incoming element to

a previous element in the evolving structure

• Both Language & Music: integration isinfluenced by the distance between twoelements in an abstract cognitive space

Shared Syntactic IntegrationResource Hypothesis

• Overlap in the syntactic processing of languageand music can thus be conceived of as overlapin the neural areas and operations which providethe resources for syntactic integration

• Patel proposes the 'shared syntactic integrationresource hypothesis' (SSIRH). According toSSIRH, the brain regions providing theresources for syntactic integration are'processing regions' that serve to rapidly andselectively bring items in 'representation regions'up to the activation threshold needed forintegration to take place

Language Specificity of theP600 ERP (Patel, 1998)

• The P600 reflects reanalysis ofstructural relations, corresponding tothe difficulty of structural integration inrule-governed sequences

• Separate studies previously foundP600’s in response to syntacticincongruities in language, and harmonicanomalies in music

Experiment 1: Methods

• Within groups design, 15 English-speaking, musicallytrained, right-handed adults

• 150 sentences: target phrases easy, difficult, orimpossible to integrate with prior context, e.g:

“Some of the senators had promoted an old idea of justice”“The senators endorsed promoted an old idea of justice”“The senators endorsed the promoted an old idea of justice”

Exp. 1 Methods cont’d

• 144 musical phrases: groups of threebased on same root phrase, with targetchord as tonic, related key, or distantkey (next slide)

• Latency, polarity, scalp distribution ofelicited potentials = dependent variables

• B-A and C-A waveforms comparedbetween domains

Example of musical stimuli

Results• Positivities elicited by

incongruous words &chords: statisticallyindistinguishable inamplitude & scalpdistribution in P600 latencyrange, at moderate & highanomaly levels

• Strongly suggests theprocess eliciting P600’s isnot language-specific.

Language

Music

• Subsequent neuroimaging research has supported syntactic overlap– musical syntactic processing activates 'language areas' of the

brain• Maess et al., using MEG, found an early right anterior negativity

(ERAN) associated with harmonic processing in music– located in left frontal language area (Broca's) and its RH

homologue• fMRI studies of harmonic processing report activation of these areas

as well as Wernicke's language area in harmonic processing

Imaging and Syntax

Semantic Priming & the N400• When the brain encounters a word that is

semantically unpredicted by the context, itemits an N400 ERP component

• “He buttered his bread with jam and shoe.”

• If the target word is predicted by thesemantic context, the amplitude of theN400 is reduced

N400 is specific to semantic meaning

• The N400 is not evoked by just anyunexpected stimulus, only unexpectedsemantic items

• A word in an unexpected type facedoesn’t evoke the N400

• “He buttered his bread with JAM.”• Grammatical errors do not evoke the N400• “He buttered his bread on jam.”

Can music modulate the N400response?

• Words in isolation evoke the N400(since the context doesn’t predict them)

• When a word is preceded sequentiallyby a related word, the N400 is reduced

• Would music that somehow depicts acertain word act as a semantic prime,reducing the N400?

Music Affects the N400• When the musical passages were

unrelated to the primed word, the N400effect was present

• When the musical passages were relatedto the primed word, the N400 effect wassignificantly reduced

• suggests music can communicate specificideas

• Referentiality- music can represent &arouse emotions/images

(Koelsch, 2004)

Neural processing of awhistled language

• Traditionally recognized “speech processing” brain areas (lefttemporal, inferior frontal lobes) become active in other domains,such as sign language & non-linguistic acoustic signals

• To what extent can such brain regions adapt to other signallingforms?

• Silbo Gomero, a rare form of communication used byshepherds in the Canary Islands, condenses the phonemicinventory of Spanish into two vowels and four consonants,which are reformed into whistles varying by pitch & rhythm

Silbo Gomero sound clips

• “Domingo is sick”• Domingo está enfermo

• “John milks the goats”• Juan ordéñame las cabras

Experiment 2: Methods

• Functional neuroimaging was acquiredwhile Silbadores and non-whistlers:

• listened passively to Silbo and Spanishsentences vs. a reversed-Silbo baseline

• monitored cycles of Silbo and Spanishwords for target words vs. a silentbaseline

Experiment 2: Results

• Processing of Silbo activated left hemisphere regionsassociated with spoken-language function only inSilbadores

• No common cortical language areas found for Silboand speech in non-whistlers

• Language-processing regions adapt to a wide rangeof signalling forms

Hemodynamic response to passivelistening task:

General Conclusions

• Aspects of syntactic and semantic processingtranscend traditional limited conceptions of modularspeech processing

• Syntactic rules and predictions/expectancy areevident in linguistic and musical contexts

• Results indicate domain-general activity, and/or needfor a broader definition of “language”

Whistling a Language

• Whistles ≈ sine waves• (no timbral variation, consonants)• Pitch glides: vary range, contour, length• Silbo Gomero: 5 vowels, 4 consonants• Populations: typically isolated, sparse• Terrain: difficult, mountainous• Long distances (1-5 km)

Talking Drum• Long distance

communication• Imitates language• West African tone

languages, e.g.Yoruba, Bantu