music: a science of the mind? - mcgill university...music: a science of the mind? 3 the...

Contemporary Music Review, © 1987 Harwood Academic Publishers GmbH1987, Vol. 2 pp. 1-61 Printed in the United KingdomPhotocopying permitted by license only

Music: A science of the mind?

Stephen McAdamsInstitut de Recherche et Coordination Acoustique/Musique, Paris, France (psychologist)

There is an increasing interest in psychological studies of music for the advancement ofboth musical and scientific thought. An historical perspective of psychological considera-tions of music reveals a trend leading from physical thought through theories of sensationand finally up to modern cognitive psychology. What might truly be called the field ofmusic psychology can be shown to exist by an overview of the domains of researchcovered by contemporary theorists and researchers. Within the framework of cognitivepsychology one can demonstrate the importance of mental representation of the variousdimensions and structures of music and of organizational processes underlying musiclistening. The relation between a musical structure and the form that is "accumulated" by alistener depends to a large extent on his or her musical experience within a given culture.A difference in experience may be hypothesized to result from the nature of mentalschemata acquired by listeners. A great deal of work is still needed, however, to approachmore affective and aesthetic aspects of musical experience.

KEY WORDS music psychology, cognition, internal representation, organizationalprocesses, musical form, musical structure.

There is one real, and graded, distinction between sciences like the biologiesand the physical sciences. The former are unrestricted and their investigatormust be prepared to follow their problems into any other science whatsoever.

C.F.A. Pantin (1968, p.24)

Introduction

Music is a fertile ground for the development of thought in the cognitivesciences, but why would a scientist of the mind dare attempt an analysisof the nature of human musical experience? Can there be any realm ofcollective human production and response more obscure, mysteriousand, finally, so absolutely personal and individual? At the same time,can any human science ignore one of the species' most uniquecapacities? The answer to this last question being an obvious "No!,"the question arises of how to,proceed.

With current knowledge drawn from the various human sciences,from diverse theories of music and from the speculations of practising

Downloaded By: [McGill University] At: 22:23 22 August 2008

2 Stephen McAdams

composers, we must take stock of what we know and what we canformulate as a starring point for a psychology of music. A thoroughscientific consideration of musical experience necessitates drawing frommany fields including acoustics, psychology, brain sciences, artificialintelligence and music theory. Although, any single researcher orthinker must necessarily limit the scope of his investigations, much ofwhat today calls itself the psychology of music runs the opposite risk ofnarrowness of focus and lack of appropriate musical culture. There is aneed at this point in the development of cognitive explorations intomusic to have a plan of approach and a vision of the totality with whichto judge the relevance of more specific considerations. Included in thisglobal vision is an intimate knowledge of music itself.

The study of music has a great deal to offer an understanding of thehuman mind. The starting assumption of the psychologist of music isthat the structure and process of music can indicate the nature of certainmental structures and processes. The study of mental structures andprocesses is the domain of cognitive psychology, which seeks tounderstand internal (or mental) representations and the things thatthese representations allow one to do with music. The nature of therepresentations is inferred from the ways people listen to, memorize,perform, create and react to music. The psychologist of music seeks theneatest and most economical manner of describing music's structure in away that most closely resembles the psychological processes by whichmusic is created, reproduced and understood (Sloboda, 1985, p.ll). Wemay suppose, then, that since music is a uniquely human product, wecan logically draw conclusions about the connection between theobserved structure of music and the nature of the human mind thatproduces it to be heard by others. What might these conclusions be, andwhat do they contribute to a general understanding about humanmental activity and about the possibilities of music itself?

Another of the underlying beliefs of this endeavour is that psychologi-cal investigation and theory also have something to offer the musicaldisciplines. There are two main constraints, however: the work mustconform to the accepted procedures of scientific investigation and must,as well, succeed in being relevant to the experience of real music and notonly limited to that of overly reduced pre- and proto-musical collectionsof sounds, though these lower-level investigations will always beimportant in analysing the contribution of various psychologicalprocesses to the comprehension of music.

I will attempt to demonstrate to both psychologists and musicians therelevance of music psychology for contemporary musical and scientificthought. Starting with an historical perspective that traces psychologicalconsiderations of music from the ancient Greeks to modern cognitivepsychology, I will then describe some of the main problem areas in therealm of music psychology in order to situate the framework for a moredetailed discussion of the two main problems in cognitive science asapplied to an understanding of musical experience. These include themental representation of the dimensions and structures of music, and


Music: a science of the mind? 3

the organizational mechanisms underlying the mental processing ofmusical structures. The aim is to arrive at an understanding of theexperience of musical form, as rich or poor as that may be depending onone's previous experience with the music of a given culture. Thepsychology of music clearly includes more areas of study than aretreated in this article, but I have deliberately limited the scope of thisvolume to reception and processing of musical structure and the experienceof musical form. Also, while this article is consciously biased towardsthe theory of psychological issues, it is balanced by the more specificallymusical considerations of my musical colleagues in the other articles ofthis issue.

The cognitive trajectory

The history of psychological considerations of music reveals a trajectoryfrom physical thought to sensation, perception and cognition (Figure I).1

The Greeks of the 6th Century BC attempted an integration of thelaws of nature with theories of aesthetics. For Pythagoras, there was animportant relation of ratios of small numbers between pitches (stringlengths) to musical consonance. The apprehension of these ratios was amanifestation of a "higher" sense he called "pleasure-in-proportion."This sense was to be distinguished from the more base senses involvedin ordinary perception. The idea was developed by Plato who made apsychological distinction between mere perception and an inbuiltresponse to proportion and the rhythms of the universe (the harmony ofthe spheres). This response naturally leads one, he mused, to a state ofbeing where the flux of the base senses is resolved in a higher state ofequilibrium. In contemporary times this classical notion of resolution asequilibrium or a state of quiescence has yielded to a more dynamichomeostatic model of perception which takes into account the necessitywithin an organism of fluctuations around an optimal level. At aphysiological level, things must constantly be in a state of flux or thesensory systems cease to respond. The best known example of this is thefact that the eyes constantly make small random movements to keep theimage of the environment moving across the retina. If this image is heldin tiie same place, by pressing one's finger against the eyeball, forexample, it slowly fades away. Similar kinds of "adaptation" or"fatigue" to constant sound stimuli have been shown in auditoryperception. What this may imply at a higher, more musical, level is thatthe tensing and releasing functions in the development of musical ideasmay characterize an aesthetic response by first defining a certain"optimal" state, and then departing from and returning to this state as away of modulating the experienced tension.

Another assumption in Pythagorian thought that is implicitly psycho-logical is the notion that there is an identical correspondence betweensome specific external, or physical, property (string length) and theresulting internal, or perceptual, property (sensation of pitch). This


4 Stephen Me Adams

Physicalists

"Natural"Intonations

andMicrotonal

Systems

Psychoaesthetics Psychoacoustics Music Cognition

Figure 1 This diagram represents a possible reading of the historical trajectoryof music psychology (with time progressing from top to bottom) from thephysically based thought of the ancient Greeks to modern cognitive science. Thearrows indicate transitions to subsequent schools of thought and the double barsindicate a rupture. At the bottom are some current areas of systematic orscientific investigation of music.



"naive realism," as it is called in philosophy, has been systematicallymodified in Western thought by the development of psychologicalmethods for investigating the relations between physical facts (thevibration frequency resulting from a string of certain length, tension anddensity, for example) and facts of sensation (pitch). This approach hasshown that the correspondences between physical and perceptualdimensions are far from being simple and are very dependent on otherpsychological processes in the listeners such as attention.

Some 18th Century music theorists such as Rameau and Tartini madeattempts to emphasize sensation and more affective qualities of sound toa greater extent than did their predecessors of the 17th Century whowere more mathematically and mechanistically inclined. However,certain aspects of this earlier rationalism persevered even into the 19thCentury with the claim of Delezenne that "the sense of the interval mustbe innate and not a product of convention because 'our scale . . . can befound in identical form throughout Europe'," (Spender, 1980, p. 389).To his credit, and in spite of what Spender calls his anthropologicalparochialism, Delezenne (1826-7) did devise rigorous experimentaltechniques for measuring peoples abilities to tune octaves, 5ths, 3rdsand 6ths and to correlate these with their degree of musical training. Wefind, even in the latter part of the 20th Century, a resurging of some ofthese earlier notions in the concern of various composers and theoristsfor the relative aesthetic and emotional "value" of different tuningsystems based on whether or not they contain "pure" or "irrational"interval ratios. See, for example, Lou Harrison's notion of the Surd (fromthe Latin for deaf) which he uses to refer to irrational intervals: "Theintervals of equal temperament are surds. Our word 'absurd' carries thesuggestion 'from deafness'." (1971, p. 7; see also Partch, 1974, for thedevelopment of a compositional system based on small number pitchratios, and Makeig, 1982, for a study of affective perception of intervals.)

After 1850, psychology departed from philosophy and purely theore-tical considerations to establish itself in the experimental laboratory,adopting scientific method, borrowing theoretical models from thephysical sciences and beginning to combine an interest in the variousaspects of sensation with a numbr of advances in human physiology,particularly the physiology of the sense organs and of the nervoussystem. One might say that the field of psychophysics (of whichpsychoacoustics is the auditory part) was born in Germany with thework of physicists and physiologists such as Fechner and Helmholtz.Psychophysics is a branch of experimental psychology which attemptsto relate physical descriptions of stimuli with measurable behaviorsbelieved to be related to sensation, such as saying that one light stimulusis brighter than another, or that one sound is higher in pitch thananother. The sensationists believed that the primacy of sense implied thatscience should study sense organs. They conceived of the mind as adevice for recording and combining sense impressions. Intelligence,then, was a product of impression, memory and association (Langer,1942).


6 Stephen McAdams

Fechner, in addition to creating the field of psychophysics anddeveloping many of its experimental methods (I860), established thefield of what is today called experimental aesthetics (1876). This was anattempt to bring objectivity into the assessment of personal preferenceand to search for a sensationist definition of beauty and aesthetics basedon liking and disliking. It essentially represents a conception of art as thesatisfaction of taste. In particular, Fechner developed a method ofpreference judgements for "pleasingness" regarded as appropriate tomusical aesthetics and which survives to some extent today. In spite ofthe fact that increasingly sophisticated methods of data analysis havemade possible a clearer definition of some of the dimensions of artisticattributes, several aspects of artistic experience that are not specificallydue to the structure of the art object, such as the influence of culture andpersonal associations, confound to a great extent any generalizationsthat one might make about aesthetic experience. Fechner's 16 principlesof psycho-aesthetics were not entirely empirically based, and as yet notruly scientific theory of musical aesthetics has been developed, thoughsome researchers still continue in this direction (Frances, 1984; Berlyne,1971). According to Langer (1942), " . . . it seems to be an essentiallybarren adventure" (p. 180).

In contrast to Fechner, Helmholtz (1877/1885) investigated moregeneralizable psychoacoustic aspects of musical sounds rather thanindividual taste. From his studies of the mathematical structure of theinner ear, he developed a model of pitch perception by resonancewhich, though no longer accepted in its original form, still shows itsinfluence on several modern theories of pitch. His theory was basedboth on Ohms' Acoustical Law which states that the ear performs aFourier, or frequency, analysis on the incoming complex sound wave,and on Muller's Law of Specific Nerve Energies which states that eachnerve fibre leaving the inner ear reacts to but one small range of themany frequencies that stimulate the ear. This model led him to presumethat each discriminable pitch stimulated one single nerve. He also noteda problem with this model that is only now beginning to be understood,which was: why then do we tend to hear the complex sound comingfrom a single musical sound source as integrated, fused or unanalyzed,i.e. as a perceptual whole rather than as a collection of pitches?Helmholtz also developed many notions concerning the sensory basisfor the perception of consonance and dissonance, of the physical originsof differences in timbre, and of the role of unconscious perceptualprocesses in the perception of. melodic succession and harmonicprogression. He already intuited the importance of higher levelpsychological processes in the creation and perception of art, not findingsufficient evidence to "reduce" aesthetic perception to the laws ofsensation.

. . . J have endeavoured to shew that the construction of scales and ofharmonic tissue is a product of artistic invention, and by no means furnishedby the natural formation or natural function of our ear, as it has been hitherto



most generally asserted. Of course the laws of the natural function of our earplay a great and influential part in this result; these laws are, as it were, thebuilding stones with which the edifice of our musical system has been erected,and the necessity of accurately understanding the nature of these materials inorder to understand the construction of the edifice itself, has been clearlyshewn by the course of our investigations upon this very subject. But just aspeople of differently directed tastes can erect extremely different kinds ofbuildings with the same stones, so also the history of music shews us that thesame properties of the human ear could serve as the foundation of verydifferent musical systems.

Helmholtz (1877/1885, pp. 365-6)

Mach (1886) worked in the realms of both auditory and visualpsychology. In addition he did important work in the perceptualanalysis of temporal order and musical rhythms. His work laid theground for later research by Fraisse (1978, 1982). Another colleague ofthe same era, Stumpf (1883), wrote an important work on thepsychology of the qualities of musical tones. But as Susanne Langer(1942) has observed, Stumpf gave us a Tonpsychologie and not really aMusikpsychologie, a statement which summarizes well the limits of thesensationist approach to music psychology.

The mentalist school, active between the two World Wars, adhered tothe belief that body and mind were separate but parallel entities. Two ofthe greater thinkers of this period were Wundt (1896) and Titchener(1909) who believed that conscious experience could be analyzed intoatoms of mental feelings and sensations. According to this view the earpresents acoustic information to the great repository of musical talent,the mind. This clear dichotomy is difficult to maintain in the face of morecontemporary evidence that the information received by the sensorysystems is known to be progressively integrated at all levels ofpsychological processing along the sensory pathways. "The notion thatthe ear conveys a sensation to the mind which the mind in turn listens tobegs the question of how the mind might 'listen', and obscures the factthat the coded pattern of brain activity during perception represents themusical experience" (Spender, 1980, p. 390; my emphasis). Threeimportant works on the psychology of music were published during thisperiod: Mursell (1937) concerning the elements of music, particularlyrhythm, Seashore (1938) mapping out a certain number of musicalphenomena such as instrumental timbre and vibrato and developingseveral measures of musical aptitude, and Schoen (1940) concerning theaffective and aesthetic influences of music.

The main thesis of the Gestalt theorists was that the perception of formor pattern was due to an innate (rather than learned) response of thenervous system as a whole to the entire pattern of energy stimulating itsreceptors. There is a one-to-one (or isomorphic) relation between thepattern of stimulation and the pattern of nervous activity in the brain(Kohler, 1929). Current knowledge of neurophysiological processesmakes this claim untenable, but what remains from the work of the


8 Stephen McAdams

Gestaltists are two sets of analytic tools which are relevant to theperception of the properties of objects: 1) the figure-ground phenomenonthat distinguishes some "object" to which we might pay attention (thefigure), in spite of the presence of a lot of other information which isrelegated by perception to the background, and 2) a series of laws ofperceptual organization that attempt to resolve ambiguities about what isfigure and what is ground. While these tools may serve as guidingprinciples for experimentation, they are merely descriptions of organiza-tional tendencies rather than explanations of the psychological mechan-isms underlying perception (Hochberg, 1974). At the beginning ofGestalt psychology many musical examples were cited, but little musicalexperimentation was actually done and most of the work concernedvisual grouping. This thought has, however, influenced a number ofcontemporary researchers interested in the problems of musical organi-zation (see below).

The mentalist and Gestalt stances were fiercely attacked by thebehaviorists who adopted the empiricist principles of the association ofideas and more specifically applied them to the association betweenenvironmental stimuli and behavioral responses. Few behavioristsapproached the problems of music, thus leaving a gap between the low-level experiments of psychophysics and the high-level aesthetic theoriesderived from Gestalt psychology or Freudian principles. The basicelement of behaviorist theories of learning is the stimulus-response unit,which is considered to be an irreducible unit in a chain of learnedassociations. This principle is tacitly used in psychoacoustic discrimina-tion and detection experiments where rather than tell the subject whichdimension of a sound should be paid attention to, reinforcement(feedback) is given in the form of a light blink that indicates when thecorrect response was received. The subject then does what ever isnecessary to maximize the number of light blinks. Spender (1980)remarks that the formulation of the notion of stimulus-response isconfounded by a disregard for individual differences in musicaleducation, leading in part to a present day lack of rapport betweenpsychoacoustic and musical experimentation. A behaviorist approach tothe psychology of music is found in Lundin (1967).

A more recent approach to the psychology of music that picks upsome of the trends of mentalist and Gestalt theory is what might becalled the cognitive approach. Cognitive science has emerged from thecross-fertilization of a number of disciplines including psychology,linguistics, neurophysiology, philosophy and computer science. Itsmain concern is to characterize the mental capacities of human beings(and other organisms in some cases). Its most important goals includethe attempts to understand the nature of mental representation andmemory, the processes of organization of perception and thought, andthe ability to reason and solve problems. A recent work written by acomposer/theorist and a linguist (Lerdahl & Jackendoff, 1983) makes itvery clear that with appropriate conceptual tools, the concerns of musictheory can be expressed in such a way as to render this domain



accessible to experimental psychology. In particular they borrowedmany ideas from linguistics in the forming of their theory of thecognitive structure of musical forms belonging to the Western tonalidiom. The success of this approach is witnessed by the growing numberof experimental studies on the claims made by the theory. As Sloboda(1986) has remarked, it is with this work that the psychology of musiccomes of age!

The realm of music psychology

The psychology of music as a field studies all the possible forms ofmusical behavior with the accepted scientific methods of experimentalpsychology (see Spender, 1980; Sloboda, 1985; Dowling & Harwood,1986). The many and varied concerns of the field include some which fallwithin the realm of experimental psychology such as:

— the perception of musical qualities of sound,— the cognitive processes of organization, representation and storage

of music,— the acquisition and exercise of musical skills in performing and

listening,— the emotional and aesthetic responses to music,— the processes of creation (composition and improvisation), and— the nature of inborn and learned musical aptitudes;

some which are slightly more physiological such as:— the brain functions underlying musical processes, and— the role of music in healing mental and physical illness;

and some which are more social and historical such as:— the social psychology of collective music and the cultural contexts

of musical experience, and— the historical development of musical material and forms.Clearly, not all of these areas have been explored in depth, despite the

ancient Greek origins of psychological considerations of musical experi-ence. The areas most extensively explored up to the present include theperception of musical qualities, organizational processes in auditoryperception and the nature of musical aptitudes. How human beingsmake sense of the complex and highly structured acoustic message wecall music has been the principle task of experiments since the beginningof the 1970's. It is from that time that the cognitive framework began toexert its influence on music psychology.

Music psychology shares some problems with other experimentalfields such as psycholinguistics and ethnomusicology. Music psycholo-gy and psycholinguistics attempt to explain the capacity to produce andcomprehend grammatical utterances and to relate the reception ofacoustic properties to understanding. Many theorists have consideredthe relations between language and music at levels both of the structureand syntactic relations of the products of these activities and of theirnotational systems. They have also considered the role of culture in the


10 Stephen McAdams

acquisition of musical and linguistic skills. The psychologist's task is tounderstand the mental representations and processes that make thispossible (see, for example, Sloboda, 1985, chaps. 2 & 6).

Psychologists of music are equally as interested in the question ofwhich aspects of musical experience are universal or culture specific andhow a given cultural experience throughout one's life limits theunderstanding of aspects of musical form specific to the music of other,very different cultures. Here there is an evident overlap with someproblems in comparative ethnomusicology and the possibility to bridgepart of the gap between psychoacoustics and ethnology. Nonetheless, alarge gap still exists between what musicians recount of their experi-ence, what psychoacousticians say about human auditory discrimina-tory powers, and what perceptual psychologists say about limits ofhuman auditory organization.

Some of the areas listed at the beginning of this section, such asaffective and aesthetic response to music and all of the questions thathave to do with "meaning" in music, will be approached only with greatdifficulty in the near future. This is primarily due to the limits in simplydefining what the real problems are that need to be addressed andinvestigated.

Healthy research paradigm

In a remarkably dear-sighted paper entitled "Cognition and real music:The psychology of music comes of age," John Sloboda (1986)2 insists thatit is only within the last 10 years that a true scientific discipline deservingthe title "psychology of music" has come into existence. The reason isthat a coherent field of investigation, what Thomas Kuhn in his seminalbook The Structure of Scientific Revolutions (1972) calls normal paradigmaticscience, is only now taking form. The coherence of this form, i.e. theparidigm, requires a number of criteria to be satisfied. Researchers andtheorists must have

(1) an agreed set of experimental problems,(2) an agreed set of methods for working on the problems,(3) agreed theoretical frameworks within which to discuss them,(4) techniques and theories which are specific to the paradigm, and(5) research programs which are appropriate to the whole range of

phenomena in the domain the paradigm addresses.

I will attempt in this section to elaborate upon his thoughts on these fivecriteria as his is one of the most clear'statements to date of what it meansto have a field called "psychology of music."3

There are three major experimental problem areas that are being orneed to be addressed in our cognitive considerations about musicalexperience. 1) What is the nature of musical knowledge and representa-tion? 2) What are the processes involved in music production and



comprehension? 3) How and why do these representations andprocesses figure in the aesthetic and emotional effects of music? Themajority of the work to date has been in the first area, with increasingactivity in the second. The lack of experimental work in the third seemsprimarily due to difficulties in defining the nature of aesthetic andemotional effects. According to Sloboda, these kinds of questionsexclude from the psychology of music a good number of areas ofconsideration including music education and pedagogy (that do tests oneducational techniques which often make simplistic assumptions aboutthe nature of musical experience), as well as experiments which derivedata from musical situations but do not attempt to explain their musicalsignificance, as is the case with a great deal of psychoacoustic andauditory perception work.

While Sloboda is absolutely correct in placing emphasis on realmusical materials and experimental tasks that resemble normal musicalbehavior, this is too strong of a position for a couple of reasons. One isthat it has become increasingly possible to mingle education andexperimentation with the use of computers. This is already taking placein teaching musical skills to children (Seymour Papert at MassachusettsInstitute of Technology) and to college students (Gerald Balzano at theUniversity of California at San Diego). The second reason is related tothe open-ended nature of psychology invoked in the quote at thebeginning of this article. In order to explain a certain number ofobservations related to representation, processing and response tomusic, we cannot, without forsaking the scientific status of musicpsychology, refuse to descend into the lower auditory levels associatedwith pre-musical processing. After all, musical hearing is first of allhearing.

One of the seminal publications that did move the field in thedirection of being truly musical is a paper by Longuet-Higgins (1976) inwhich a cognitive model of musical abstraction took as input idealizedinformation about a performance (pitches and times of notes) andproduced a sensible musical notation with some degree of success. Thecognitively interesting features of the system are that it assigned thepitches and time values within a tonal/metric system using heuristicsthat embody psychological hypotheses about musical organization. Theanalysis that resulted from this "artificial musical intelligence" systemaccord reasonably well with the intuitions of musically trained listeners.

In searching for an agreed set of experimental methods, earlyresearchers in music psychology have had to summon the courage (or tohave the maturity as Sloboda says) to leave the experimental paradigmsof their less musical colleagues that are often limited to judgments of"same or different" between two tones. Some of the newer methodscurrently being employed include

— direct recording of performances on specially prepared instru-ments that allow precise determination of performance parameterssuch as time of attack, duration, velocity of attack on a key,variation in air pressure in a wind instrument, and so on,


12 Stephen McAdams

— methods for measurement of memory performance by reproducingmusical fragments on an instrument or notating them on paper inconventional music notation,

— synthesis of sounds by computer in order to have precise controlover the way a sound varies and then to be able to relate thesechanges to variations in reported musical experience,

— studies of verbal behavior as a way to explore more aesthetic andqualitative aspects of musical experience and to collect verbalprotocols in order to follow to some extent the thought processes ofsomeone in the act of creative musical problem-solving (thoughthis is of course fraught with the danger of the interaction betweenanalytic introspective and creative processes),

— recording of physiological correlates in response to music listeningor generated in performing, and

— computer-based simulations of musical behavior as a means offormalizing our understanding of the way a human processes agiven musical piece — the attempt being to get the computer toproduce some abstract structure from the musical signal thatcorresponds with the responses gathered from musical subjects.

Sloboda emphasizes, above all, the importance of experimentation onreal musical material that embodies the real complexities of music andwhich are performed in the context of response situations that are moreclosely related to normal musical activity. This constraint wouldseparate truly musical psychology from many types of experimentationand theory that one might call pre-musical or proto-musical where thematerial is not truly musical and the tasks demanded of the experimen-tal subjects are quite different from those actually performed in the act ofmusic listening, reading, performing, writing or improvising.

Here again, I find Sloboda a bit restrictive, particularly with respect tothe task of listening. Many of the response situations listed above(reproduction and notation) can only be performed by trained musiciansand the relation between physiological recordings and psychologicalexperience is far from beginning to be understood with any degree ofclarity for even very simple sound stimuli. Several of the more classicalexperimental paradigms relying on comparison and recognition couldstill be quite relevant to the experience of untrained musical listeners.

A somewhat more difficult criterion for any new science to fulfil is thecreation of a set of agreed theoretical frameworks. Among those listedby Sloboda (and slightly modified here) are included the assumptions

— that music is lawfully constructed and these laws are used bylisteners to decode the "meaning" of a musical "utterance,"

— that the internal representation of music listening has a hierarchicalcomponent,

— that scales, meter, and rhythm are psychologically real organizingprinciples and are possibly instantiations of musical universalswhich may be found in similar forms (to some degree) in almostany musical culture,

— that the processes of music composition, performance and



perception access the same core representations in the mind.These theoretical assumptions should account for a number of

observable aspects of musical behavior, such as the nature and postionof errors in performance, the distribution of expressive parameters inperformance, the differential memorability of musical passages, thedifferential salience in the perception of musical events, the judgmentsof well-formedness by listeners, cultural differences in music percep-tion, the process of enculturation and learning (how people acquirecompetence in musical language4), and, eventually, the structural andcognitive determinants of musical meaning and emotion.

The next criterion insists that the techniques and theories of theprevious two criteria be specific to the paradigm. Sloboda states thatthere is a strong tendency among researchers to study music because itis a good example of something else, such as a complex motor skill, alanguage-like phenomenon, a complex auditory phenomenon or a settheoretic entity. It is or can be, of course, all these things and often thefinancial needs of music researchers require that they disguise theirwork as these other things in order to convince national funding bodiesof the "scientific validity" of the proposed studies. The courageousmusic psychologist would, however, try to understand where music iswholly unlike anything else, all the while keeping an eye on relationswith other fields since no human mental activity is entirely divorced fromthe-rest. Some of the unique aspects of music include the fact that itcreates a non-referential (or perhaps, self-referential) world, practicallymaking it unlike language and several other arts; that it does so through,for example, psychological dimensions which are essentially unique tomusic in the way they are used such as pitch hierarchies and meter; andthat despite its lack of specific reference, it can have deep emotionalsignificance.

The last criterion states that the paradigm must generate research thatis appropriate to the whole domain. It is easy to show that, even in itsinfancy, music psychology demonstrates a wide range of musicalactivities. Most research to date consists of work on perception, but oneis starting to see work on performance, memory, analysis, compositionand improvisation. There is a great effort on the part of an increasingnumber of researchers to use real music (at least fragments, if not smallpieces) rather than impoverished music-like material. What makes thiswork difficult is using the "real thing" in order to move towards agreater "ecological validity," or simply musical relevance in the research,without giving up rigorous scientific control. This often means supple-menting musical material with specifically constructed material thatvaries along pre-determined dimensions. There is also an increasing useof subjects and behaviors that are close analogues of normal musicalactivities such as memorizing for recall, sight reading, extendedlistening, transcribing and improvising. Most importantly, this trendtoward musical relevance stimulates the interest of musicians toparticipate in the larger research project. A side effect of this newinterest is the appearance of courses on psychoacoustics and music


14 Stephen Me Adams

psychology in music curricula and of bonafide psychologists in musicdepartments and conservatories.'

We would like to use these criteria in the exploration of composition,performance and listening. Let us now consider the products of musicalactivity and unfold from these the central concerns and the majorsubstance of the field of music psychology.

The products of musical activity

There are essentially three products of musical activity that need to beinvestigated and explained from a psychological standpoint: 1) anotation or text resulting from composition and from which thegeneration of sound can proceed by a third party, 2) a series of motorpatterns (or patterns of muscle activity) generated by a performer inorder to control a musical instrument or voice, and 3) the perception andaccumulation of musical images and musical form in memory and in theconscious mind.

Notation or text includes, of course, the many kinds of musical scoreswith their varying degrees of precision of notation. An important featureof notational schemes is that they embody a large number of culturalconventions and assumptions about how the notation is to be interpre-ted. Thus, there are aspects of the performance that are not madeexplicit in the text but are implied in the cultural practice and usuallytransmitted from teacher to student either orally or by demonstration.

In this, a certain degree of freedom is allowed for modifying theprecise values expressed in the text. This modification is, in fact, expectedof the performer: a musical text played exactly as written is oftenconsidered to be mechanical and musically uninteresting. One expectscertain deviations in tempo, timing and dynamics, as well as micro-inflections of pitch and timbre to bring into relief the phrasing of a piecewhich serves to enhance one's appreciation of its form. The problem ofthe psychologist is to understand how three different things contributeto the final sonic result: 1) the notated structure of the piece, 2) theknowledge structures acquired by the performing musicians relating tothis piece, to the work of its composer, to the musical idiom of which it isan example, and to the culture beyond the idiom, and 3) the activity ofother musicians.

There is a relatively new kind of notation which is a text describing aprocedure for the generation of sound sequences by a computer. Thistext can include everything from a list of values to be plugged into asynthesizer at given times to a whole computer program that has tomake various decisions on exactly what and how to synthesizedepending on what else is going on in the musical environment at agiven moment. The case of the list of values fed into a synthesizer is onewhere there is no further musical interpretation of the notation. Thecommands are explicit and are executed as such, i.e. there is an absoluteinvariability in the realization of the score. One might say, then, that the



notation is complete with respect to its realization. The problem posed forthe composer in working in such a medium is that the realm ofexpressive interpretation from a musical text that is usually left to theperformer must be made absolutely explicit in the values fed into thecomputer synthesizer. They must, for example, include the smallvariations in tempo, timing and dynamics and are necessary for musicalphrasing. The role of the composer here is one of both creator andinterpreter.

In the case of an interpretive program, two types of text are usuallypresented. One is a score in the more classic sense of a text to beinterpreted musically and synchronized with other players. The other isa text that describes the nature and behavior of the "syntheticperformer" and its musical decision-making (interpretive) process.Attempts to formalize this understanding (or at least intuitions about it)have resulted in computer programs that interpret musical texts(Sundberg, Askenfelt & Fryde"n, 1983), accompany other musicians(Bloch & Dannenberg, 1985; Vercoe, 1985; Chabot, Dannenberg & Bloch,1986), or improvise in interaction with human or other "synthetic"musicians (unpublished work by George Lewis, STEIM, Amsterdamand David Wessel, IRCAM, Paris). In these cases one is obliged to modelthe various behaviors of a performer including perceiving sound,abstracting musical structure, making interpretive decisions and pro-ducing commands that generate sound. One quickly realizes thesimplicity of one's assumptions when one hears the relative lack ofmusicality in the early versions of these programs. In refining theprograms, one realizes the enormous number of rules necessary first toextract an abstract structure from a stream of physical signals, and thento realize an invented abstract structure in sound in a musicallyacceptable way.

One of the main psychological interests in the production of motorpatterns in performance is the relation between the temporal sequencingof muscle activity and the mental representation of the music beingperformed. In other words, how is a musical structure that is read from apage or stored in memory unfolded in time as an organized series ofcommands to the appropriate muscles. There are interesting features ofthis with respect to the learning of performance skills. For example, itseems that as one passes from the stage of novice to expert a greater andgreater "vocabulary" of motor sequences becomes automatic and, as aresult, more easily commanded. One might imagine a little musclepattern program in the brain that is triggered by a single commandwhich then chains together all of the necessary movements to realize it,whereas the novice is obliged to execute each movement with a separatecommand. In fact, the reason for the long hours of practising scales andexercises is to encode more and more efficiently and permanently thesevarious kinds of patterns such that they become a more or less automaticsequence and no longer require the intervention of the conscious mindwhich slows things down and can only deal with a certain amount ofinformation at a time. The fact that the sequence itself becomes


16 Stephen Me Adams

automatic does not necessarily imply, however, that it cannot bemodified by higher level processes to make it cohere expressively withits surrounding musical context, such as creating tempo fluctuations.What is implied is that such expressive modifications are more efficientlyrealized by perhaps simply modulating the rate at which the sequenceunfolds in time rather than recomputing the temporal relations of all theconstituent movements necessary to play a fast run for example.

The role of auditory and proprioceptive feedback during the executionof a motor pattern also interests psychologists. That is, how does thesound and feel of what one is producing at a given point affect what isproduced shortly thereafter? This is interesting from the standpoint ofthe interaction between the mental representations of the motorcommands, the incoming auditory information, and the sensoryinformation coming from the parts of the body being used to producethe sound, such as the fingers, mouth, tongue, diaphragm and foot inthe case of a musician playing a clarinet and tapping her foot to keeptime.

Another problem of interest that is related to the production of soundis the translation of notation through the visual system into a mentalrepresentation of what is to be produced and what should be heard as anend result. One can easily imagine the development of automatismsthrough the repeated experience of correlating the visual representationand the imagined auditory result with the actual sound produced by thereading.

The third product of musical activity, and the one that most concernsthe remainder of this article specifically and this volume in general, ismusical perception and the accumulation of musical form in memory. Incognitive psychology, we talk about the internal representations ofperceptual objects and events, of concepts, and so on. It seems fairlyclear that what is "in the head" when we perceive and remember is not acomplete and identical copy of something that is or was in the world,but rather a kind of abstract description of it. To illustrate this, imagine asmall girl singing a melody she heard her mother sing previously. Mostlikely, in order to learn the melody she has to have heard it sung severaltimes by her mother and no two of these times would be identical. Shemight start on different pitches from one day to the next, the sound ofher voice might be clear one day and then froggy another if she has acold, she might sing it slower one time and faster another and so on.And the mother is entirely capable in her own right of deciding whetheror not the girl has correctly reproduced the melody even though she hasnever heard her sing it before. The fact that this is possible suggests thatthe way at least this type of melody is represented is somehowindependent of the exact starting pitch, tempo or timbre of the voice.The representation most probably contains aspects of pitch intervalrelations, pitch contour, scale, etc. One can imagine that otherdimensions such as rhythm, harmony and timbral relations would havetheir representations as well. The representation of a melody must alsohave the property that one can perform a certain number of transforma-



tions on it and have listeners still recognize these transformations asbeing related to the original. Such transformations might includetransposition to another scale step in the same key (thereby changingsome of the intervals by a half step), or various kinds of ornamentationand elaboration of the melody's theme. Without this capacity, all ofwhat we call musical development would not be possible.

What actually gets representated internally depends to a certainextent on the experience we have acquired up to the present as well ason a whole gamut of processes that organize the incoming sensoryinformation into comprehensible entities such as sound sources andmusical forms. The framework of cognitive psychology rests on animportant distinction between mental representation and the processesof mental computation that are responsible for organization andcomprehension. These two concepts,5 while being formally distinct, areentirely complimentary. Neither could exist without the other and bothare necessary for comprehension (Sperber & Wilson, 1986). Thefollowing quote reveals this for the realm of visual cognition:

. . . piston is the process of discovering from images what is present in theworld, and where it is. Vision is therefore, first and foremost, an information-processing task, but we cannot think of it just as a process. For if we arecapable of knowing what is where in the world, our brains must somehow becapable of representating this information — in all its profusion of color andform, beauty, motion and detail. The study of vision must therefore includenot only the study of how to extract from images the various aspects of theworld that are useful to us, but also an inquiry into the nature of the internalrepresentations by which we capture this information and thus make itavailable as a basis for decisions about our thoughts and actions. This duality— the representation and the processing of information — lies at the heart ofmost information-processing tasks and will profoundly shape our investiga-tion of the particular problems posed by vision.

(Marr, 1982, p. 3)

The remainder of this article will consider in more detail the nature ofthe internal representations and organizational processes involved inmusic listening.

Internal representation of musical dimensions andstructures6

A primary presumption in music psychology is that there are underlyingpsychological principles that channel musical cognition. The psycholog-ist is interested in the nature of these principles and how they areacquired, if they are not innate. There are many kinds of musical activitywhere cognitive skills are in evidence: hearing, performing, remember-ing, imagining, reading and creating. Some of the key features of theskills behind these activities are worth noting. They have access to a vast


18 Stephen McAdams

array of mental structures (such as knowledge of the names andrelations in a pitch system, their notation, and ways to reproduce themon various instruments, and knowledge of the individual differences intone quality obtained when the same pitch is played on severalinstruments, etc.). These structures must be easily and quickly deployedin reading, listening and performing.

In this section I will first address some general considerations on thenature of mental representations and memory processes7 and thenbriefly survey current understanding of these for pitch, rhythm, timbreand musical structure.

One of the major contentions of the cognitive approach to psychologyis that all mental activity is mediated by internal (or mental) representa-tions. What the actual forms and nature of these representations are isthe major axis of cognitive research and there is anything but aconsensus, even about whether they exist or not!8 Let us take, however,the point of view of Pylyshyn (1985) that we must refer to "representa-tions" in order to express certain cognitive generalizations that are, forour present purpose, useful in explaining people's capacities in musiclistening. As he notes, "In science, the process of seeking to understandis called explanation" (p. 1).

The importance of the notion of representation as an explanatory toolfor symbolic function in human thought is beginning to be understoodby philosophers and scientists alike. According to Langer (1942), we usesymbols to attain and organize belief, and intelligence is related to the*power of using symbols for conception and expression. She states thatsymbolization is a basic, characteristically human need, that is an actessential to thought and prior to it, and indeed, that it is the essential actof mind.

[Symbolization] is the starting point of all intellection in the human sense, andis more general than thinking, fancying, or taking action.... The current ofexperience that passes through [the brain] undergoes a change of character, notthrough the agency of the sense by which the perception entered, but by virtueof a primary use which is made of it immediately: it is sucked into the stream ofsymbols which constitutes a human mind.

Langer (1942, p. 46)

As an element of explanation, a representation may be defined as "aformal system for making explicit certain entities or types of informa-tion, together with a specification of how the system does this" (Marr,1982, p. 20). An example of this in the external, or public, domain is therepresentation of musical events and pieces of music by a symbol systemwith conventions for notating temporal ordering, pitch, dynamics,specific instruments and various sound production techniques specificto a given instrument. This is a formal system because it is a set ofsymbols with rules for putting them together, where the symbols areused to stand for, or represent, things. Any degree of complexity can be,in theory, attained with a symbol system since a symbol can also standfor a lawful combination of other symbols. Internal representations, on



the other hand, are physical causes of behavior in the form of physicalcodes or symbols, i.e. patterns of brain activity. Acts are governed byrepresentations which are symbols of various kinds. Certain of theserepresentations correspond, for example, to the perception or imagina-tion of a major chord, or to the connecting of a sequence of clarinet notesinto a melody, or to the expectation of a tonic chord to resolve thetension state evoked by a dominant seventh chord. "To be in a certainrepresentational state is to have a certain symbolic expression in somepart of memory" (Pylyshyn, 1985, p. 29).

One of the tasks of the psychologist is to determine the forms theserepresentations take. The form is important in that different forms ofrepresentation, even though they carry the same meaning, may havedifferent properties with respect to what the processes of transformationcan do with them. Certain aspects of reality are more apparent in somerepresentations than in others since representation makes certaininformation explicit at the expense of other information. As a physicalanalogy, consider for a moment the various kinds of visual representa-tion of acoustic information such as the oscilloscope, the Fouriertransform and the sonogram (Figure 2). There are different aspects ofthe perception of sounds represented in these ways that becomeapparent with each one. The moment at which a sound event starts andstops is more easily seen in an oscilloscope trace and a sonogram than ina classic long-term Fourier transform since the two former have the timedomain more readily apparent in their representation and the Fouriertransform computes the frequency content of a sound over a long timeperiod, in a sense collapsing the analysis over this period of time. Thetime domain is implicit in the Fourier spectrum but not visually explicit.As such, we have to explicitly take separate Fourier transforms overdifferent time windows if we wish to contrast the spectral characteristicsat one moment with those at another. On the other hand, certain aspectsof timbre related to the form of resonance structure of a sound source aremore easily "read" from the Fourier spectrum, to a great extent from thesonogram and not at all on the oscilloscope trace. Again, the frequencyresonance characteristics of the sound source are present in theoscilloscope's time trace, but are not readily accessible due to the form ofthe representation.

Psychoacousticians have the problem of finding the most appropriateand relevant acoustic representation for their stimuli that allow them todraw conclusions about the relation between the measured responses ofexperimental subjects and the perceptual representation that results fromthe stimuli. The psychoacoustician measures both the acoustic signaland the response of the experimental subject, compares them and thentries to infer what is happening inside the subjects head. For thecomposer working with computer sound synthesis, it would thereforebe necessary to have a familiarity with several ways of representingsounds in order to focus on the aspects that are relevant to thecompositional decisions at hand for a given sound context.

Before returning to mental representations, let us diverge for a


20 Stephen McAdams

event 1 event 2 OSCILLOSCOPETRACE

time

FOURIERTRANSFORM

freq freq

SONOGRAM

time

Figure 2 Three different visual representations (based on acoustic analyses) ofa musical signal. The first shows the equivalent of an oscilloscope tracerepresenting time on the horizontal axis and amplitude (or pressure) on thevertical axis. The second shows 2 Fourier transforms representing frequency on..the horizontal axis and amplitude on the vertical axis, one each taken from themiddle portion of sound events 1 and 2 as indicated in the diagram above. Thethird shows a sonogram representing time on the horizontal axis, frequency onthe vertical axis and amplitude by the degree of darkening.



moment to consider the two major kinds of memory that are relevant toour concern with musical memory: short-term memory (STM) and long-term memory. STM is what we use for temporarily storing incominginformation or recently activated information from long-term storage inorder to make calculations on it or make comparisons with succeedinginformation or with other elements in long-term memory. Whatis stored in STM is already an interpretation of what has occurred in theworld and is not a complete image of the incoming sensory information.This memory is also of a very limited capacity: only about 5-9 items canbe stored at any one time (Miller, 1956). The storage capacity of STM isalso a limited duration. If no effort is made to refresh the information, byrepeating or rehearsing it (as one does when trying to remember atelephone number, by saying it to oneself over and over again), theinformation very quickly "fades" after about 6-9 seconds or is soonreplaced by new information. By contrast, LTM is more like permanentinformation storage and has no known upper bound on the quantitythat can be stored. Information in LTM is highly structured. It takes agreat deal of time and effort to store new material in LTM, and then, it isoften not so easily recalled, though it might be recognized immediately.

In a loose way we may consider representations to include thestructures assumed by memories of things we have experienced, theconclusions and assumptions we have arrived at that stand for beliefs ormodels we have about how the world works, and so on. Thesestructures and models may be conceived of as schemata that combinesymbols for concepts and relations between these concepts. All of theseare operated upon by processes or rules whose goal it is to build upmodels of what is transpiring in the world in order best to predict whatwill happen next and to interpret and comprehend the meaning of whatis happening (Neisser, 1976; Sowa, 1984).

Both schemata as representations of knowledge and the constructiveand interpretive processes that operate upon this knowledge have someinteresting formal properties that makes them reasonable candidates forthe way the human mind works in the perception of complex acousticstructures like music. In this view, memory may be represented as akind of storage that is itself a model of the evolving physical world. Theknowledge that has been acquired from the world is represented by thestate of the model at any given moment. This, of course, implies that thestorage is structured in relevant ways with respect to what is importantin the world.

The perception process depends a great deal on different levels ofstorage and representation of incoming information. Sound impingingon the ear is encoded by the inner ear into a flow of neural informationto the brain. Sensory processes in the brain that detect various featuresin this neural information prepare a description of the characteristics ofevents in the environment that stay in what is called auditory storage (orechoic memory) for a very short period of time. This sensory representa-tion is easily disturbed by the arrival of new information (a processcalled masking).


22 Stephen Me Adams

From this representation organizational and pattern recognitionprocesses use rules that are based on the current context for combiningthe features. These rules are guided by attention which operates frompreviously acquired schemata to assemble the auditory image which isthen stored in short-term memory for comparison with previously heardsounds, and subsequently arriving sounds. Items in this short-termstorage can come from the senses or can be recalled from long-termmemory. The information in short-term memory is in a special activesta,te that allows it to be used. An important feature of this active state isthat only a limited amount of information can be kept active. However,the amount of information that can be kept active by a given persondepends on the experience that person has had with the soundstructures in question (cf. Anderson, 1985, chaps, 3,4, 6). Someone whoknows a large number of Haydn symphonies in great detail will be ableto more easily store passages from another Haydn symphony since hewould have schemata that allow him to structure the information inefficient ways for storage. These schemata would not, however, helphim store passages from music by Babbitt or Xenakis, the organizationalprinciples being very different.

Conflicting schemata that may each partially match the incominginformation and the musical form that is gradually accumulating inmemory may generate illusions or ambiguities. This is, of course, one ofthe most commonly used "cognitive" tools of composers that areinterested in structural richness and ambiguity.

Multiple levels of perception help deal with novelty and help processcomplex structures. In cases of complexity, a more bottom-up approachmay be used wherein the total object is constructed from more familiarlower-level percepts. When more familiar objects or structures areencountered, though, large-scale schemata are activated where possibleso the most global percept is the first to occur. This would be the casewith face recognition (i.e. we immediately recognize a face as a wholewithout having to put together two eyes, a nose and so on) andrecognition of familiar pieces of music based on a few notes. The partimmediately evokes the whole, in a certain sense.

This raises the question of what is retained by a listener and of how thedegree of complexity of retention depends on musical experience andtraining. There are a couple of important differences between novicelisteners and those with much experience in a particular musical idiom(Sloboda, 1985, chap. 1). One is in the number and complexity ofstructural features in terms of which the listener is capable ofrepresenting music. Expert listeners can notice, remember and oftenreproduce on an instrument much finer levels of expressive sublety inthe structure of musical phrases, and much more global levels of musicalform that escape naive listeners. Secondly, expert listeners have agreater awareness of the structures they are using, implying that theyhave a more extensive vocabulary of description which enhancesmemory capacity, or at least the access to remembered things.

The capacity of memory structures in music listening is of paramount



importance since musical structures are extended in time. The percep-tion of movement, of transformation and of musical significance dependon the perceived element being heard in relation to rememberedelements. We might say that perception really only becomes musical when itis "in relation to" events, sequences, progressions and structuring in memory.The form of a piece of music is what gets accumulated in memory, andthus the richness of that form depends very heavily on one's capacitiesand experience as a listener.

This model of the perceptual process as a building up of representatio-nal schemata has some important assumptions embedded within it. Oneis that percepts are pre-fabricated building blocks that are derived fromexperience. Another is that a schemat is a pattern for assemblingperceptual units or other schemata into larger structures or unitarywholes. And, finally, these schemata can operate on various levels todiscern structures in the sensory information either at a level ofexpressive variation or at a level of global form. All of these assumptions

• seem to have their psychological reality with respect to the way werepresent and appreciate musical forms.

Evidence that visual images can be transformed or examined inmental operations supports the notion that they are derived frommodels or regenerated from some kind of representation like schemata(cf. Kosslyn, 1980; Shepard & Cooper, 1982; Pinker, 1984). Neisser (1976)has proposed that images can serve as anticipations (or as perceptualreadinesses) for what we expect to find in the surrounding environmentat a given moment in a given context. This is supported by evidence thatfamiliar forms are matched by ready-made percepts (previously assem-bled structures stored in long-term memory that can be recognized veryquickly, and do not need to be reconstructed from low-level percepts),whereas unfamiliar forms are reconstructed from percepts for theirparts. There is most likely a parallel processing going on here, where aswe build up a musical form in LTM, from assembled fragments in STM,we also look up these fragments in LTM. This comparison affects howthe new fragment gets placed in the whole form.

A key process in music listening is the way we select certain thingsfrom the mass of information that comes to us as a musical surface. Aswe listen, our attention fluctuates through the many paths of thissurface. What we remember, what gets stored away for futurecomparison with later moments on the surface, depends to a greatextent on what we have attended to in listening. There are two basickinds of attention. Automatic attention seizes upon striking or salientfeatures of the incoming information. This kind of attending isinvoluntary and under little influence by the conscious processes of thelistener. Willful attention, by contrast, seeks cognitively importantfeatures of the information flow which are determined by what weexpect to hear or want to hear. This kind of attention is involved infollowing the line of a particular instrument in a complex musicaltexture. In this case attention is guided by knowledge structures that havebeen developed through experience. These important structures of


24 Stephen McAdams

memory and comprehension are what we have called mental schemata(Bartlett, 1932; Neisser, 1976; Sowa, 1984).

The ultimate result of our ability to perceive, remember, conceptualize, and acton musical information is the formation of internal frameworks, or schemata,for representing and reproducing more complex musical knowledge. Schemata... are flexible. They certainly change as we get older and as our familiaritywith musical tradition grows.

(Dowling & Harwood, 1986, p. 4)

A general feature of schemata as mental representations of knowledgeis that they embody both general and specific knowledge structures.They incorporate general knowledge of musical stimulus propertiescommon to many pieces of a given culture, such as tonal scales andharmonies, or specific knowledge of relations among tones within agiven melody such as its contour, for example. Since music unfolds intime and musical schemata represents various kinds of progression anddevelopment through time, they also serve to generate expectations byanticipation. Both the general and specific levels of schemata are moregeneral than the actual sound one hears in the act of listening and so thegenerated expectations of what is to follow would very rarely be sospecific or dominant as to lead us into perceptual errors (Dowling &Harwood, 1986, p. 125). They tend more to be indicators of the mostlikely unfolding of events. This is a general cognitive capacity that is not,or course, limited to music listening. Schemata are just as important forsleepily going through the motions of preparing one's morning tea asthey are for finding one's way to work whilst absorbed in thought or tothe toilet at night in an unlit apartment. They can also be useful indeciphering very complex structures since lower-level schemata canextract structures that are embedded in a larger-scale musical form. It iswith schemata that we pay attention to things in the world.

Perception takes place when appropriate schemata are actively and continuous-ly tuned to the temporally extended information that specifies an individualevent. Irrelevant events present information too, but remain unperceivedsimply because no such active tuning occurs with respect to it.

(Bahrick, Walker & Neisser, 1981 — cited inDowling & Harwood, 1986, p. 127)

This notion is important in explaining the inherent selectivity ofperception, i.e. we do not notice everything that is going on around us— we tend to select out of the environment things that have somemeaning to us individually. It is also important in explaining thedifference in effort required when listening to forms of music that arerelatively new to us compared with what we normally listen to, or whenlistening to those that are very familiar but also very long.

Musical schemata are not at all literal copies of sound events. They aremore abstract entities representing meaningful aspects of sound struc-tures. This aspect of schemata is apparent with respect to languagewhen people are asked to recall as precisely as possible a story they have



been read. None of them reproduces a verbatim account of the story.They elaborate upon the structure and the main elements of .theirunderstanding of the story; that is, they construct a linguistic utterancethat recommunicates the meaning. What is meaningful will, of course, bedifferent for each person, as the varied witness's accounts of the scene ofa crime reveal.

With this notion of the image as a reconstruction from conceptualstructures or schemata, it is easy to describe the relation betweenperceived and imagined or recalled images. Internal images have thesame nature as (though they are not identical to) sensory images, andconsciousness allows the brain to analyze and reinterpret an internalimage using the same perceptual mechanisms used for sensory input.

If we then consider the temporal nature of auditory perception, theseanticipatory schemata must have some kind of temporal ordering intheir structure which constrains the construction of auditory imagesfrom them, or the recognition of auditory patterns by them. Theimportant implication here is that these auditory images require astructural coherence. Since the schemata we are presuming to underliethem are ordered structures, perceptual grouping processes define theconstraints on these structured relations. It seems entirely reasonablethat the structure of these schemata (at least at the level of sound eventsthough not necessarily at the level even of a musical phrase) would berequired to reflect the reasonable behavior of physical sound sources.(See Shepard, 1984 for a fascinating discussion of the embodimentthrough evolution of physical constraints on internal representations ofvisual forms).

Let us examine the nature of memory for the different features ofmusical material in more detail. As one might expect, most of theresearch in this domain has focussed on pitch and, to a lesser extent,rhythm. Very little work has yet approached the problems of remember-ing or processing timbre structures.

Pitch

There are several features of pitch structures that are likely candidatesfor memorization. These include, at the lowest level of structure, pitchheight or value, chroma (or pitch class) and interval. At a higher level wehave contour, scale, key and harmonic progression. These more basicelements contribute, along with the elements for rhythm and timbre, tothe storing of more complicated structures such as phrases, sections andentire pieces. .

For harmonic sounds the mapping of frequency to pitch may beconsidered as a logarithmic psychophysical function. Constant ratiorelations in frequency give constant interval relations in pitch — at least"within the range of musical pitch. The mental representation of pitch isin terms of scales with discrete steps. This discretization of the pitchcontinuum into conventional scale degrees is found in all cultures of the


26 Stephen Me Adams

world that use pitch in their music. The steps in the scale function asperceptual categories that facilitate hearing, remembering and reproduc-ing a musical message. The dissolution of the relation of pitch classes toa scale is a relatively recent development in contemporary Westernmusic which can pose a certain number of problems of memorability.

There are a number of constraints on scales that one finds and whichseem to have important psychological consequences. The steps of a scaleare easily discriminable. There is a musical equivalence of two pitchesrelated by an octave. There are a moderate number of scale degreeswithin the octave. The aspects of discrete, distinguishable perceptualcategories and a small number of elements to be remembered areprobably very important for any perceptual dimension that is to carrysome aspect of form (McAdams & Saariaho, 1985). The fact of octaveequivalence enhances the usable range of pitches since a pattern ofintervals within an octave is considered perceptually equivalent withrespect to pitch structure in any other octave.

The feature of octave equivalence has another perceptual consequ-ence. It contributes to the multidimensional subjective experience ofpitch. In other words, pitch is not merely organized along a singledimension of higher-lower (called pitch height), but has, in addition,another dimension which is circular (called chroma). This subjectivecircularity is evident in the verbal labelling schemes used in manycultures to identify the pitches of a scale, e.g. do, re, mi, fa, sol, la, ti, doin Western solfege, or sa, re, ga, ma, pa, dha, ni, sa in North Indian music,where the names repeat in each octave. Thus, chroma is a subjectiveattribute of pitch that is independent of the octave in which it is found.In addition to this, another circular dimension of subjective experience isfound in relation to the Western tonal idiom where the interval of theperfect fifth has a strong structural importance. This dimension is thecircle of fifths which represents a distance, or degree of commonality ofpitch classes, between keys. A number of psychologists have investi-gated the mental structure of this subjective multidimensionality (with atechnique called "multidimensional scaling") and find that it variesaccording to the culture and musical sophistication of the listener. Thestructures of less experienced listeners have less dimensions that thoseof more experienced listeners, notably with respect to the circle of fifthsdimension, but the other dimensions are relatively similar in both typesof listeners. This would seem to imply that with greater experience, oneacquires increasingly rich mental structures with which to process themusic one hears (Shepard, 1964, 1982; Krumhansl, 1979; Krumhansl &Shephard, 1979; Krumhansl, Bharucha & Castellano, 1982; Krumhansl &Keil, 1982; Balzano, 1982). Another level of mental pitch structureassigns a hierarchy of organization to the pitches of a given scale system.This functional organization at, for example, the base of the Westerntonal system establishes dynamic patterns of tension as one learns toexpect that a certain scale degree will tend to move toward another. Oneof the more difficult tasks of serial composers and others composingoutside of the tonal system is to avoid at all costs any sequential or



chordal pitch combinations that might invoke these hierarchical struc-tures and generate expectancies that are foreign to the ordering schemethey are trying to establish.

With an understanding of these culturally acquired mental schematafor the organization of pitch material, it comes as no great surprise thatthe memory for single pitches, intervals and chords is strongly affectedby the musical context in which they are placed — as sparse as thiscontext may be in the many experimental situations where it was tested.In early studies by Krumhansl (1979), for example, additional informa-tion provided by the context helped in remembering a pitch that wasrelated to a tonal context or interfered with one that was unrelated. Thisdemonstrates that we remember pitches with respect to a referenceframe derived from the surrounding context in such a way that thecontext evokes a given mental structure within which the pitch to beremembered is interpreted. In a study by Shepard & Jordan (1982), thecontext was a major or minor scale in which all of the intervals were"stretched" such that the "octave" fell a semitone higher. Subjects werethen presented a pitch that came from the normal scale based on thestarting pitch, from the normal scale based on the final pitch, or from thestretched scale as actually presented. They were asked to decidewhether the pitch belonged to the scale they had just heard. Mostsubjects tended to choose pitches that belonged to the normal scalebased on the final pitch indicating that they have a kind of fixed patternor template representing a normal diatonic scale and that as they heardthe stretched scale this template was slid along the pitch continuum. Inthis case it would, of course, be anchored on the real final note whichwas a semitone higher, and when the test pitch was presented theywould make their comparison based on a shifted rather than stretchedscale pattern. Very few people actually chose pitches that correspondedto the actual pitch played, indicating that memory for pitches is notabsolute but occurs with respect to the structure that seems to beappropriate for a given context.

An important feature of sequential pitch patterns is their contour, orthe pattern of ups and downs in a melody. This feature seems primarilyto be used in STM. The use of contour for recognition of melody patternsseems to have importance where the relationship between the melodyand a scale schema are not yet established or where there is no scaleschema as in atonal music. It also seems to be a particularly strong factorin recognition for people with little musical training or listeningexperience (Dowling & Fujitani, 1971). This contour factor remainsconstant when one performs a tonal imitation of a melody (translation ofthe melody to another scale degree of the same scale where some majorintervals become minor and vice versa). In this case the pattern of upsand downs remains constant even though the actual intervals andchromas have changed. As a dimension of similarity, it may beimportant in the development of musical material. Bart6k uses thissimilarity as a cohesive force (wittingly or no) when he augments thesize of some or all of the intervals in a melodic theme: again there is a


28 Stephen McAdams

serious modification of chroma and interval size but the global contour ismaintained. Frances (1984) has shown the perceptual salience of thisfeature in experiments where atonal melodies were compared to eitherdirect transpositions or imitations of these transpositions which kept thesame contour but modified certain intervals. Subjects had difficultydistinguishing the transpositions from the imitations indicating thatcontour was the stronger feature used in the comparison. This wascorroborated by Dowling & Fujitani (1971) who showed that subjectscould readily distinguish similar melodies where the contours weremodified.

At a higher level of perceptual abstraction, studies that extractmultiple subjective dimensions from listeners' comparisons amongchords have shown that the intuitions of musicians on the importance ofkey distance have a psychological reality. In these kinds of studies(Krumhansl, Bharucha & Castellano, 1982) listeners with varyingdegrees of musical training are asked to judge the relative similarity ofchords derived from different keys. The quantitative result is a kind ofmap of keys in which one finds continuous paths of harmonicmodulation easily represented.

Pitch intervals and chroma are primarily used in long-term memory.Studies by Dowling & Bartlett (1981) suggest that interval recognition isimportant for melodic material stored in LTM which may mostly bematerial that we have become familiar with, since to get new materialinto LTM it must be repeated several times. However, if one hasschemata for more efficiently encoding new material, it can more easilybe stored in LTM. Both experienced and inexperienced listeners, forexample, store accurate interval information in LTM on first hearing anew melody when this melody is tonal (Dowling, 1982). With inexperi-enced listeners the information is simply stored as raw intervals, butwith experienced listeners the intervals seem to be stored as chromapatterns linked to tonal scale schemata. Dowling & Fujitani (1971) foundno difference between experienced and inexperienced listeners indiscrimination of imitations and transpositions of "atonal" sequences.

It should be mentioned that one often reads of pitch memory studiesin the music psychology literature contrasting "tonal" with "atonal"melodies. In most cases the "atonal" melodies are constructed byrandomly selecting pitches from the chromatic scale (though the workby Frances dating from the late 1950's was more carefully composed).Thus a structured tonal melody is compared to random atonal melody.This hardly does justice to the pitch structuring possibilities outside ofthe tonal system, and, in effect, does not test tonal versus atonal, butstructured versus unstructured material. What is sometimes demons-trated is mostly a markedly unscientific (as well as naive) aesthetic biason the part of the researcher who wants to "prove" the unreasonable lackof psychological reality in atonal music. We should take studies whereresearchers without appropriate musical culture place themselves in theshoes of composers with a large grain of salt (and swallow them withlots of mild to avoid the bad taste).



Rhythm

The analog of the pitch scale schema for rhythm is metric structure. Thegeneral consensus among rhythm researchers is that we use a metricbeat pattern as a cognitive framework with which to organize rhythmicpatterns in perception and production. Even in the case of very irregularrhythm sequences people attempt to infer a regular beat pattern(probably based on the generation of some kind of internal clock) andregular metric structure (Povel, 1984; Longuet-Higgins & Lee, 1984;Povel & Essens, 1985). This structure can be revealed by the waypeople's reproductions of heard rhythms deviate from the presentedpattern, presumably moving towards a simpler pattern representation.

Povel (1981) studied people's abilities to synchronize tapping with arepeated rhythmic pattern or to continue the pattern after it wasstopped. One result of the study was an apparent tendency to reducemore complicated patterns to binary rhythms with period subdivisionsthat had a duration ratio of 2:1. If a metric context could be provided thatmade it easier to encode the more complicated patterns, accuracy ofreproduction increased. Povel hypothesizes a two-level cognitive struc-ture including the determination of a regular beat pattern with amoderate tempo and then a simple (preferably binary) subdivision ofbeat intervals. The inferred metric structure that results from thisoperation then organizes the rhythmic patterns. According to Longuet-Higgins & Lee (1984), we still need a few more assumptions about thisprocess, since a given rhythmic pattern could be placed on a metricstructure starting on any of several different beats in the period of themeter. One such additional constraint would be that in the interpretiveprocess one tends to avoid as much as possible relations between metricframe and rhythmic pattern that give syncopations. This is, of course, avery strong culturally-based assumption that would not apply to manynon-Western rhythmic systems.

To draw an analogy with the relation between melodic pitch contoursand scale framework, Monahan (1984) suggests that patterns ofrhythmic subdivision are anchored to a beat framework, such that theencoding of subdivisions is of relative, rather than absolute, temporalrelations. As with melodic contours, rhythmic contours (or relativepatterns of long and short) can be expanded or contracted to fit differenttempo schemes or translated along a pattern of accents. Listeners tendto use these two dimensions together to judge the relative similarity ofrhythmic patterns. Different patterns played at different tempi might bejudged similar if the inferred relative rhythmic contours are similar inform. It remains to be shown if rhythmic contours behave similarly tomelodic contours with respect to their potential for memorization.

Cross-cultural studies show that the form of the inferred metricstructure depends entirely on cultural norms. A given rhythmic patternmight be anchored to a binary metric system with accents on the beat bya Westener and to a ternary metric system with accents just after the beatby a Central African, if we define the beat as being where one would


30 Stephen McAdams

clap in time, for example (cf. Arom, 1984, 1985). This implies that theschemata we develop for rhythmic interpretation are constrained by thenorms of our culture. As Simha Arom is fond of saying, even afterstudying this music for 20 years he will never be able to hear theserhythms as the Africans hear them! There is a latent assumption herethat there may be a limited critical period for the "natural" acquisition ofa musical system in childhood as is evident in the acquisition oflanguage. For further remarks on cross-cultural differences in rhythmicorganization in African, Indian and Indonesian music, see Dowling &Harwood (1986, chap. 7).

As with pitch, once again, rhythmic organization can be hierarchicallystructured. Much theory has been expounded on this subject and thereader is referred to work by Clarke (1985, this volume), Grisey (thisvolume) and by Lerdahl & Jackendoff (1983).

Timbre

Not much work has been done on memory for timbral identities andtimbral structures. It is clear that humans have a great capacity fortimbral discrimination and the use of timbral differences to conveyinformation. This is primarily found in our use of speech, where two ofthe main categories that carry meaning, i.e. vowels and consonants,have many physical similarities to the domain of sound structure wenormally call timbre with respect to musical sound.

The most notable research on mental structures for timbre has beendone by Grey (1977) and by Wessel (1979; Risset & Wessel, 1982). Inthese studies, subjects were asked to judge the similarity of various pairsof instrumental timbres and from these judgments a geometric "timbrespace" was derived in which distance corresponded to degree ofdissimilarity (Figure 3a). This notion corresponds psychologically to the"pitch space" represented by pitch height, the chroma circle and thecircle of fifths, and may serve as a kind of mental schema that organizestimbral sequences. We might expect to be able to extract from thisschema a certain quality in the relation between two points and judgeother relations as having a similar quality in other parts of the space justas we consider a pitch interval to be similar thrqugh all of the registers ofmusical pitch. Ehresman & Wessel (1978) tested this notion by askingsubjects to judge the similarity of relations between pairs of timbres inan analogy task (Figure 3b,c). A pair of instrument timbres, A and B,were presented sequentially. Then a third timbre C was presented and aseries of possible timbres D whose relation to C was to be judged.Listeners were to choose the D timbre whose relation to C was the mostlike the relation between A and B; a bit like the verbal analogy task,mother is to daughter as father is to.... The results indicate that if oneconsiders the relation between A and B to be represented by a vector inthe multidimensional timbre space, then the best completion of theanalogy would be that D which formed the end-point or a vector that



a) bright

sharp attack

Englishhorn

trumpet

bassoon

mutedtrombone

ce

flute

GGOG

Do

sax

iorn

aclarinet

clarinet

mutedcello

TWO-DIMENSIONAL

TIMBRE SPACE

soft attack

dull

b)TIMBRE ANALOGY

A is to Bas

C is to which D

A = English hornB = oboeC = cello

c)

BEST SOLUTIONIS D1

The timbre

interval English horn to oboe

is approximately analogous to

the interval cello to Eb clarinet

Figure 3 A geometric visualization of the mental representation of timbre canpredict certain kinds judged relations among timbres such as the timbre interval.The top square (a) shows 12 instruments from the 2-dimensional timbre space ofWessel (1979) for the dimensions "brightness" and "bite". The next square (b)shows the subject's task which is to complete the timbral analogy (by listening,of course), i.e. to find a timbre interval similar to A-B that starts on C. The lastsquare (c) shows that the best prediction based on the mental representation oftimbral relations is satisfied by subjects' reports. We can therefore infer thatthere is a mental representation of timbres whose form is similar to thisgeometric plot where distance corresponds to degree of similarity.


32 Stephen McAdams

was parallel to the vector AB and of the same length. Thus, a timbreinterval has not just distance in the timbre space, but orientation within itas well. To make a timbral transposition would involve sliding thetimbral vector within some plane containing that vector. Based uponpsychological considerations such as these, and others concerning thenature of potential form-bearing elements in music (cf. McAdams &Saariaho, 1985) or of general cognitive constraints on compositionalsystems (Lerdahl, in press), we should be able to develop ideas aboutthe structural possibilities of timbre as an ordered and composedelement in music (see chapters by Boulez, Bonnet, Lerdahl andSaariaho, in this volume).

Memory, musical structure and form

It seems almost banal to reiterate that there is an immense amount ofstructure in musical stimuli. For the psychologist struggling to makesense out of the human capacity to comprehend music, however, adescription of the stimulus is only the beginning.

The psychological questions start with the given structure and ask whataspects of that structure are relevant to cognition — which of the manydimensions of a musical stimulus structure have psychological reality, andhow fine a grain of discrimination on those dimensions is relevant.

Powling & Harwood, 1986, p. 163)

The pattern of memory process that evolves through the developmentof a piece include both specific memories for salient phrases or themesand less specific memories for more global invariants in the pattern.These two kinds of memory correspond more or less to Tulving's (1972)episodic and semantic memory, respectively. The time of an episodemay be related to the length of the "perceptual present" or the timewithin which ongoing experience is available to consciousness andduring which incoming information can be organized. We might thinkof it as a kind of sliding window, the width of which depends on theprocess of attention and on the density of new information coming in tobe processed. Estimates of its width tend to be on the order of 2-5seconds though some extend it to about 10 seconds.

The processing and organizing limits of this window have aconsiderable influence on the amount of information that could beintegrated into a conceptual unit to serve as a theme, for example. It israther unthinkable that a melody lasting 2 minutes could be held inmemory arid understood in various kinds of transformation as a wholeunit. The memory of a piece is organized in a series of episodes ofdifferent lengths that depend on the musical structure. Rhythm is a veryimportant element for episodic musical memory since it can help tohierarchically organize material for more efficient storage. It also tendsto help segment material into phrases or sub-phrases. With respect tothe entire structure of a piece an episode can correspond to the level of



the phrase. Phrases tend to have a kind of stability in memory. Memoryof sequences within a phrase structure has been shown to be better thanfor sequences that traverse a phrase (or episode) boundary. This mayimply that episodes are stored as structural units (Dowling, 1973). Theyare probably very closely linked to structural organization processessuch as auditory stream formation and segmentation, as well as toprocesses that infer harmonic and metric structure (see next section onorganizational processes). They are also probably linked to structuralschemata that give rise to expectancies and then evaluate whether themusic goes in the expected direction or not. What stands out asunexpected and unique in form is more likely to be remembered.

"Semantic" musical memory, or memory for global meaning orsignificance, is much less well understood, because less thoroughlyinvestigated. There is first of all a problem of defining what "meaning"or "significance" mean with respect to musical structures, though wemay be on the verge of being able to follow out some of the insightsgiven us by Leonard Meyer in the 1950s with the carrying over ofmethodologies from psycholinguistics into music psychology. This typeof memory is probably linked at a structural level to processesunderlying pattern perception, and at a contextual level to processes ofmental set and expectation based on wider cultural influences.

No discussion of memory and musical form would be completewithout some mention of the importance of hierarchical structuralorganization. This notion has already come up in relation to memory forpitch, rhythm and timbre, but all of these different elements mustconverge somewhere. The main question that considerations of hierar-chy attempt to address is how local and global features of structure arerelated? Concerns with "formal coherence" on the part of manycomposers have led to all kinds of theorizing about how far one canpush structural coherence between the micro-acoustic universe and thetotal structure of a composition. Without getting lost in such arcana, letus, anyway, look at a few basic points.

One advantage of hierarchical structures is that they are an efficientway to relate local and global features. Such structures have beenrecognized for some time as possible ways of representing manydifferent kinds of information in vision, audition, motor performanceand language (see Lerdahl & Jackendoff, 1983 for a hierarchical theory ofmusic structure organization; Lerdahl & Jackendoff, 1984 present aconcise overview of their theory, and Swain, 1986, discusses some of thepsychological limits that should be considered in any theory ofhierarchical structure). The existence has been demonstrated withgreater or lesser degrees of success for linear orderings of lists (Klahr,Chase & Lovelace, 1983; Johnson, 1970), for mental images (Reed, 1974;Reed & Johnsen, 1975), for propositional structures in language(Anderson, 1985, chap. 5), for mental schemata (Sowa, 1984, chap. 2),and for musical phrase structure (Stoffer, 1985).

Other evidence for the existence of a hierarchical representation oftemporal structure is drawn from research on visual or musical phrase


34 Stephen McAdams

structures. Phrasing is the introduction of various expressive variationsin a sequence that enhance the perception of the structure (at least in thecase oigood phrasing!). Good phrasing enhances memorability and badphrasing hinders it, presumably by strengthening or weakening thehierarchical sense of the structure. This has been shown by recognitionexperiments for temporal patterns of light flashes (Restle, 1972), bymusical dictation (Deutsch, 1980), and by the time it takes to indicate thepresence of a click superimposed on a musical phrase (Stoffer, 1985).What one does with phrasing in music is not only to show the basicstructure; it has a functional value of articulation as well. This is differentfrom merely revealing structure. It helps one hear something else in thestructure by, for example, resolving a structural ambiguity in one way oranother (Longuet-Higgins & Lee, 1984).

There is evidence that purely symmetrical tree structures may be toosimple for communication or aesthetic purposes. Interesting variationsusually take place along at least two dimensions in the development ofmusical material, e.g. rhythmic elaboration plus transposition in amelody (see chapter by Lerdahl on developing hierarchical structureswith timbre in this volume). In the sonata allegro form, for example, twocontrasting themes that are easily distinguished one from the othermake it easier for listeners to keep track of where they are in the piece.The assymetry provided by alternation of different materials also makesthe structure more easily remembered by breaking a structural homoge-neity (cf. Restle, 1970). Departures from structural homogeneity can beaccomplished by the elaboration of repetitions and by introducingdifferent material between repetitions as in the sonata allegro form. Thismay also be accomplished by embedding differing sub-phrase structureswithin a phrase in a hierarchical fashion (DowHng & Harwood, 1986,chap. 6).

There evidently are other kinds of structural relations in addition tohierarchies, which may be of a more associational and functional nature.A theme can be associated with any number of variations on it inmemory, the strength of association depending on the degree ofsimilarity and the clarity of the understanding of how one got to thattransformation in the music. Relations among elements in a musicalstructure at this level can be ones of transposition, harmonic modula-tion, rhythmic and melodic elaboration, repetition, timbral variation andso on. The perception of the well-formedness of a phrase or section maywell depend on some optimum balance between recognizable transfor-mation and the degree of difference between the original material andthe transformed version. Simplistically speaking, similarity amonglower level features such as melodic themes may be given relief ormusical interest by dissimilarity at a higher level such as harmonicmodulation or some structural transformation; or, to the contrary,greater variability of the material's features can be rendered coherentwith less change at higher levels of structure.

To dose this section on mental representation of musical structure Iwould like to summarize the three possible levels of knowledge that can



be represented by any structural description of a musical composition.The first is the musical structure itself which may be developedindependently of psychological considerations, purely on the basis ofthe material that is proper to the piece. This is classically the work of themusicologist and music analyst. The second level is one of describingknowledge the listener has and uses in understanding and producingmusic, what might be called musical competence in the terms used bythe linguist Noam Chomsky (1965). Here knowledge is described interms of invariants of a given piece and style, and it represents morethan merely the memory for particular instances of the piece as heard —it is rather an extracting of the grammar of an idiom. This level is verynicely exemplified by the work of Longuet-Higgins (1976) and of Lerdahl& Jackendoff (1983). The third level is one of description of the processesby which musical information is handled by the brain-mind. This levelmust represent the knowledge of the listener with the actual structuralproperties of mental (or physiological) representations and with theprocesses by which the knowledge is put to use by the listener,performer or creator. This level of description is the goal of theory andexperimentation in music psychology. The important element inevaluating a theory is whether a given structural description makes asuitable basis for an account of the processing that may be inferred to begoing on by experimentation. The form of representation is veryimportant for discerning the nature of the processing that is done,because it heavily affects how easy it is to do certain kinds of things withit.

Organizational processes in music listening

Music derives from a complex acoustic signal that is highly structuredand rich in intrinsic and extrinsic context. Intrinsic context consists ofthe relations among the elements within a given musical structure andextrinsic context consists of the relations of the piece to a musical styleand culture, including any extra-musical symbolic and codified emotio-nal associations. A psychological theory of musical organization aims todescribe the way a listener organizes this complex signal into a completeform. What will be described in this section is basically a beginningapproach to a theory of the processes underlying music listening and byno means claims to address all of the processes involved in musicalexperience. It will be limited to the perception and comprehension ofmusical structure and will not attempt at this point to respondspecifically to aesthetic, emotional and sociocultural aspects of musicalexperience that contribute significantly, nonetheless, to an appreciationof musical form.

The primary query of such a theory is "what can possibly be heard andwhat tends to be heard in a complex musical work!" The question implies thatthere are acoustic structures that one cannot apprehend as such, andthus proposes that there must be, after all, biological and psychological


36 Stephen McAdams

limits to the comprehension of musical form. The question also impliesthat there are default tendencies in hearing that one brings from theeveryday world or from specialized musical training into a musicallistening situation. These tendencies can place limits on the way inwhich certain musical structures are apprehended.

As an approach to understanding musical experience through theframework of cognitive science, the theory should address the mechan-isms of processing, the forms of representation (and storage) that theseprocesses operate upon, and the final comprehension of extendedtemporal structures. Some of the primary psychological interests hereinclude:

— the temporal and hierarchical ordering of processes of musicalorganization from psychological to cognitive levels,

— the development of the notion of psychological coherence withrespect to the cognitive processing of complex temporal structuresand in relation to the inherent coherence of the acoustic environ-ment,

— the nature and operation of preference rule systems in auditoryorganization that select one given possible organization of astructure over another.

— the psychological constraints on perceptual dimensions (such aspitch, rhythm and timbre) that are capable of carrying temporalform,

— the interaction between perceptual processing and memory struc-tures in the assembling of large-scale temporal forms,

— the construction and acquisition through learning of mentalschemata that permit a sense of development of tension andrelaxation, expectancy and directed motion in music, and

— the development of notions of lexicon and meaning in a non-verbalmedium without committing the error of expecting them to behavein the same manner as language.

The theory must ultimately describe the main functionalities of a"language" that is specifically musical and propose constraints on itsform-bearing elements as well as provide the structural and proceduralgrounds for eventually investigating aesthetic and emotional aspects ofmusical experience. We may consider five main areas that need to beincluded in the theory:

(1) reading the acoustic surface,(2) organization of acoustic information into coherent auditory "im-

ages",(3) segmentation and the extraction of the musical lexicon,(4) building structural relations, and(5) following a musical discourse.

All of these levels and kinds of processing create and operate upon themental representations of musical material and structure that werediscussed in the last section.



Reading the acoustic surface

This level of the theory must account for the type of information that ismade available to higherjevels of processing by the peripheral auditorysystem as it transduces the vibratory information into nerve impulses.This is classically considered as the domain of psychoacoustics. Withinthe limits of its accuracy, the ear encodes a sound in such a way as torepresent its temporal and spectral (frequency) characteristics. It doesthis by encoding the sound activity within small bands of frequencies inseparate nerve channels, each of which is capable of following the fine-grained time structure within that frequency band. This allows for theeventual extraction of periodic behavior as well as the tracking of overallintensity fluctuations. In this way the frequency domain is representedacross the array of some 80,000 nerve fibers that enter the brain fromeach ear, and the time domain is represented within the pattern ofactivity in each nerve fibre. This parallel representation of reciprocalproperties (periodicity is the mathematical inverse of frequency) isimportant since various characteristics of a sound-producing body are attimes more accurately represented in one domain than in the other aswe discussed at the beginning of the last section.

At this level also, one takes into account the important effects due tomasking, i.e. the partial or complete covering of one sound by others.Masking, as well as other limits of auditory discrimination, areconsidered to be constrained by the temporal and spectral resolvingpower of the ear, that is, by the degree to which the ear is capable ofencoding very fine-grained differences in frequency and time. Forexample, the information from two sound events that are similar incharacter and arrive within about 25-50 milliseconds of each other willtend to be integrated over time and perceptually fused into a singleevent; whereas those separated by more than 50 milliseconds are easilyheard as separate events. Similarly, two frequencies that fall within acertain range (within about a minor third for middle and upper registers,and as much as a fourth or more at the bottom of the piano) tend tointeract with one another, creating beats and a rough sound quality,among other effects. This range, called the critical bandwidth, representsthe limit within which separate frequencies tend more to be fusedtogether than separated perceptually. These two kinds of resolutionlimit, temporal and spectral, depend a great deal, of course, on theconstitution and behavior of the sound events.

An understanding of the nature of this parallel, low-level representa-tion of the acoustic environment becomes important as an explanatorytool for the mechanisms underlying the organization of the soundworld. Attentional processes guided either by listening or by visualinformation in a musical score, for example, may modify this phase ofanalysis and encoding so that the detection of "primitive" features, suchas the beginning of a new sound event, may not be totally automatic,but, to some extent, under the influence of activity at more cognitivelevels that "scan" the information array for certain objects or events that


38 Stephen Me Adams

may be "expected" to occur given the preceding context.Research on the perception of sound qualities such as pitch,

roughness, loudness, brightness and so on has classically been treatedat this level, considering such perception to precede most other levels ofauditory organization. However, recent work strongly suggests thatthe perception of qualities even as "primitive" as these is dependent onthe result of sound source organization, which is discussed at lengthbelow. The data from experiments on sound qualities may still beconsidered to be relevant to our present concerns with a couple ofcaveats. The results are almost always obtained for single soundcomplexes presented in isolation to listeners. There is, therefore, verylittle context to conflict with an interpretation of the sound complex as asingle source. One might presume, in this case, that the results indicatewhat one perceives after the acoustic elements have been organized intosound sources. However, it should be understood that these results areoften obtained under conditions of extremely prolonged listening byhighly trained and analytic listeners. They represent refined listeningcapacities under circumstances that are seldom found in a musicalsituation where one does not hear the same family of sounds presentedthousands of times over headphones in a sound-proofed listeningchamber. This is not to degrade the importance of these results forunderstanding the limits and certain aspects of the mechanismsunderlying the perception of sound qualities. I want merely to signal thefact that the results of controlled laboratory listening can be quitedifferent than would be the case in a concert hall, where in the samehour's time a myriad different sounds in a rich context go flying by. It ismore and more clear that context affects considerably the perceptual endresult of even very "low-level" qualities which may be considered asproperties that emerge from a given organization of the soundenvironment.

Organization of acoustic information into coherent auditory images

Most of my own experimental work to date falls into this phase of thehearing process and has been concerned with elaborating a metaphor,the auditory image, that informs both compositional and experimentalparadigms (McAdams, 1984a,b). A good metaphor in any domain ofsystematic thought should both provide a generality of description ofexisting knowledge and have predictive capabilities that generate newideas, experiments and structures of thought.

The auditory image is defined as a psychological representation of asound entity that exhibits an internal consistency (or coherence) in itsacoustic behavior. For this metaphor to be of any use in illuminatingmusical experience, it is necessary that the image, as a mental projectioninto consciousness, be derivable from perception, memory and imagina-tion and that there be a commonality of mental representation thatallows for this. It is also important that the metaphor allow for multi-


Music': a science of the mind? 39

leveled (or hierarchical) representations since a sound "entity", such as achord or orchestral texture in polyphonic music, may also be a collectionof other sound entities. In this way, the metaphor allows for therepresentation of more complex perceptual structures that result fromthe grouping of less complex structures. Two of the key concepts hereare the notion of coherent behavior of a sound entity and the notion ofgrouping processes that operate upon or seek this coherence in theirattempt to form an accurate and plausible representation of the world'sbehavior.

There are several levels of coherence that might be considered. Thefirst is the coherent behavior of physical sound-producing objects thatobey the laws of physics. It seems reasonable to presume, since thehuman perceptual system evolved in relation to the physical world, thatthis level of coherence should be reflected to a large extent in perceptionitself (Shepard, 1984). Most musical instruments are physical vibratingsystems, with the exception of electronic sound synthesizers andprocessors whose complex behavior (such as found in additive synthesisor FM synthesis) maintains a mathematical consistency, but does notnecessarily correspond to that of a vibrating physical system. Soundstructures produced with these instruments can often present illusorysituations to the hearing system, since many of the acoustic dimensionsare not always correlated among themselves as they are in a physicalsystem.

Another level of coherence is psychological, and relates to thetendencies and limits of processing complex sound structures. It ispossible, for example, to create sound structures with an acousticmusical instrument that are not assimilable nor comprehended as such asmusical form, and there are several examples of 20th Century musicfrom the integral serialism and stochastic process idioms that illustratethis. This does not mean that no form is heard in these works nor thatthey do not have aesthetic value based on something else. It ratherpoints to the gap that is possible between the intended structure notatedon the page and produced in the air by instruments and the received formthat is accumulated "in the head." These are examples of what Lerdahl(in press) calls "cognitively opaque" musical structures where there is asignificant discrepancy between the compositional grammar plus someintuitive constraints used by the composer to create the piece, and thecognitive grammar possessed by the listener who tries to understand thework.

Part of this reflects the fact that psychological coherence depends tosome extent on the interpretive schemata of everyday perceptualprocesses and past experience. This implies that what is coherentpsychologically may evolve with specialized musical experience.However, this most likely also reflects some biological limits instructural processing. That is, we have a biological predisposition toprocess certain information structures and not others. (Cf. Gould &Marler, 1987, for a more general consideration of the biological limits oflearning.) The third level of coherence would be that of the musical


40 Stephen McAdams

. , . grouping rulessequential | 6 K *grouping (generation ofprocesses J preferences)

simultaneousgroupingprocesses

imageformationprocesses

of attentionand selection (evaluation of

preferences)

acquiredmental schemataand the accumulation

musical knowledge J T emergence of imageI qualities et properties

processesof segmentation Iand higher-level |

grouping

building ofstructures

followingmusical discourse

- "musical surface"

"musical" perception

Figure 4 Schematic diagram showing the flow of information through thevarious levels of organizational processing in music listening, the building up ofa description of the current musical world, and the accumulation of musicalknowledge.



structure developed by a composer, which may or may not correspondto a psychological coherence. A theory of musical organization proces-ses should address itself mainly to the notion of psychologicalcoherence, but should consider physical coherence as a limiting case. Amore difficult task would be to investigate musical coherence in terms ofthe larger aesthetic and cultural issue of what a composer might possiblyexpect an audience to assimilate psychologically.

In order to understand the emanations of a sound entity, the auditorysystem must be able to extract the properties that derive from theensemble of its elements. To do this it must know which ones to consideras an ensemble. The process of grouping is an attempt on the part of thehearing system to decide which acoustic elements "belong" to a givensound entity. This proposes that music listening is an informationprocessing network wherein the grouping processes form a model ordescription of the current state of the world based on past experienceand on incoming sensory data (Figure 4). This model also embodiesexpectations of the most probable continuation of the current situation,which constrains possible perceptual and musical interpretations of thebehavior of the sources. A re-evaluation of the grouping decisions maybe triggered by the detection of discrepancies between the currentmodel and incoming sensory data, indicating the arrival of a new eventand, potentially, a new source.

The theory postulates two sets of rules for grouping concurrent andsuccessive acoustic elements. These rules define psychological coher-ence and many of them reflect very strongly the relation betweenphysical and psychological coherence at this level of organization. Theserules serve to form a description of source events and to group them intosource streams (Bregman, 1981; McAdams & Bregman, 1979; McAdams,1984a,b). This is an important step for auditory theory in general sincethe form the rules take bridges what may be considered a long-standinggap between studies of higher level cognitive processes of hearing andthe lower level processes of auditory sensation and perception that areclassically studied with the quantifying methods of psychophysics. Thisform refines the classic Gestalt principles of organization for audition byconnecting them to their underlying psychoacoustic processes. This isequally an important step for music theory as it proposes a frameworkwithin which to evaluate realistically the relation between the limits onthe processing of musical structure and the assimilation and apprecia-tion of a musical form.

There are several criteria for the grouping of concurrent or simulta-neous elements into source events. The acoustic elements from a singlesource tend to have a common spatial origin, coherent frequencybehavior (McAdams, 1984b), coherent amplitude behavior (Dannen-bring & Bregman, 1978; Hall, Haggard & Fernandes, 1984; Bregman,Abramson, Doehring & Darwin, 1985), harmonicity or commonperiodicity of components (Darwin, 1981; McAdams, 1982; Scheffers,1983; Hartman, McAdams & Smith, 1986; Moore, Glasberg & Peters,1986) and coherent behavior of resonance structure (McAdams, 1984b;


42 Stephen McAdams

Darwin, 1984). This set of rules assembles acoustic elements thattogether constitute an event emanating from a sound source. Theyoperate mainly on the way the various acoustic dimensions change oversmall periods of time and attempt to track these changes and determinewhich elements are changing in a similar fashion. This set may thereforebe paying attention primarily to the temporal encoding of informationacross the nerve fiber array.

The criteria for grouping sequential elements include continuity ofspectral content (related to changes in pitch and timbre) and continuityof intensity (Bregman & Campbell, 1971; van Noorden, 1977; Bregman &Pinker, 1978; McAdams & Bregman, 1979; Bregman, 1981). This set ofrules connects events across time that come from the same soundsource. They appear to operate primarily on average estimates ofspectral activity within events and thus over a slightly larger time framethan concurrent grouping. This set would appear to pay more attentionto the general form of activity across the nerve fiber array and detectsudden changes in this activity indicating a new sound source entry. Itwould also need to invoke higher level processes to connect similarevents across time. Both sets are, of course, limited by the spectral andtemporal resolving power of the auditory system, i.e. perceptualintegration into "texture" or a complex timbre may occur for hightemporal and spectral densities.

All of these criteria are graded conditions for grouping, meaning thattheir presence may influence, but does not necessarily guarantee,grouping. The degree to which they induce grouping depends on theirstrength. For example, the fusion of frequency components into a singleimage due to vibrato can be stronger than their separation due to the factthat the components come from two different spatial locations, i.e.frequency components split between two loudspeakers can be made tofuse into a single image if they have a common vibrato (McAdams,1984a). The rules establish preferences among a set of logically possibleauditory organizations, and a projected form is imposed on the acousticsurface that represents the evaluated highest degree of overall prefer-ence. Indeed this form is the one judged to be the most coherent orstable and results in the image formed in consciousness (Figure 4).

Considering these criteria in this way leaves open the possibility thatthey may interact with one another, which is clearly the case. Thevarious criteria may converge on a similar structure, leading to a clarityof perception, or they may diverge, proposing different structures, inwhich case structural ambiguity results. Thus, the grouping rules forman interactive system that operates according to principles of reinforce-ment and conflict. These principles operate at many different levels ofmusical organization which is one of the properties of musical structuresthat gives them their vast richness. We should be careful to distinguishambiguity from indeterminancy here. I refer specifically to a desiredstructural ambiguity that gives rise to a polyvalence of form. This raisesthe aesthetic question of what kinds of ambiguity are relevant andvalued, and in what kinds of contexts.



Higher level grouping may be conjectured to involve criteria of"common fate" behavior-such as similar trajectories along a givenmusical dimension or even higher level musical coherence such as thedegree to which elements belong to or contribute to a similar structuralframe (harmonic, metric, textural, etc.)- It would appear that what isderived as a perceptual quality after one level of auditory organizationmay become an "element" contributing to grouping decisions at a higherlevel of organization. For example, the separation of several series ofharmonic frequencies and the subsequent perceptual fusion of each onegives rise to a distinct pitch for each series. At the next level oforganization, some of these pitches may be grouped according to themusical context as members of a common chord whose quality of beingconsonant or dissonant arises from their being considered as a group.Melodic context may, however, pull one of these pitches into asequential organization and its perceptual contribution to the disso-nance of the chord, for example, may be diminished. Harmonicconsonance and dissonance can thus be influenced by polyphonicorganization (see Wright & Bregman, this volume).

With most of the sound sources encountered in everyday life, thesimultaneous criteria at the lowest level of grouping assemble acousticelements into source events and the sequential criteria connect eventsappropriately into meaningful source streams. In these cases, allgrouping criteria mutually reinforce one another and what emerges maybe considered as a stereotypical (or prototypical) grouping (Jackendoff,1983, chap. 8). However, some work has shown that with certainstimulus configurations, even simultaneous and sequential groupingprocesses can conflict with one another creating situations with multipleperceptual interpretations. The resulting perceived qualities of thesources depend on the way the conflict is resolved (Bregman & Pinker,1978). It is clear that active and passive attentional processes can play astrong role in the resolution of these conflicts, particularly at higherlevels of grouping where functional ambiguities resulting from conflict-ing "vertical" (harmonic) and "horizontal" (melodic) organizations canbe of great musical value.

One of the more compelling conclusions from work on auditoryorganization, as mentioned in the previous section, is that the groupingof elements into auditory images occurs before the perceptual qualities ofthe source image are evaluated (at least within a relatively small timeframe, probably less than a second; see Figure 4). This is evidenced bythe fact that with a same number of harmonically related frequencycomponents, for example, one can obtain either a single pitch or twopitches an octave apart simply by introducing independent vibratosonto each subset, thus inducing the auditory system to split the evenand odd components into separate images. This has important consequ-ences for music composition with electronic or computer soundsynthesis and for orchestration technique since it implies that arriving ata given musical quality depends on arranging one's sound elements sothat they are organized appropriately by the auditory system. In


44 Stephen McAdams

essence, these principles of auditory organization give composersadditional conceptual tools with which to ply their trade.

The important psychological notion here is that of emergent propertiesof a perceptual organization. Qualities emerge from perceptual group-ings or, as Bregman claims, are assigned to sources. Thus, the timbre orpitch of a sound depends in part on the relations of the amplitudes andfrequencies of the components that constitute the auditory object. Ifsome of them are extracted from an object and associated with another,the qualities of the first one would change. At a higher level, thedissonance of a chord depends upon which of a group of recurrent notesare grouped together as the chord. If a more dissonant component ispulled into a separate melodic organization, its departure from theconcurrent organization may diminish the perceived dissonance of thechord.

Much of the composition of what is called texture in the music of thiscentury, in effect, plays upon the strength of grouping of elementsextended across pitch and time into large-scale groupings. It depends ona kind of balancing between fusion of the elements into a coherentorganization and their fission into separate organizations. We perceiveon the one hand its global texture (or the group as a whole) and on theother its grain (or a partial distinguishing of the individual elements).This play between what Boulez calls fusion and articulation is animportant aspect of orchestration in the composition of musical timbre(see his chapter and that of Bonnet, this volume, as well as Erikson,1975, chap. 6, and 1982). This suggests a continuum between fusionand fission that allows the composer to move between the perception ofunity and multiplicity in an agglomeration of sounds, and thus to movebetween timbre and harmony (see Saariaho, this volume). The notionthat fusion and fission are not two points of a binary relation but ratheropposite poles of a continuum means that the strength of presence ofemergent qualities can be modulated by the strength of the grouping:the stronger the tendency for a given organization, the stronger thepresence of the quality. It seems clear that in intermediate cases theattentional processes of the listener can strongly influence where on thecontinuum the perception falls.

To conclude this section, let us say that what emerges from this stageof perceptual processing is what Lerdahl & Jackendoff (1984) have calledthe "musical surface" (see Figure 4), though I depart significantly fromthem since they consider the surface to be the "physical signal of a piecewhen it is played," and I consider it to exist at a certain level ofperceptual abstraction, i.e. after the detection of events, their organiza-tion and the assignment of their qualities. The acoustic environment hasbeen organized into objects and one may now pay attention to theevents, their qualities, their relations to other events and the structurethat they carry.



Segmentation and the extraction of the musical lexicon

It is at this next level that truly musical organization begins to take place.The theoretical concern is how the organized stream of musical events isbroken into meaningful chunks at various levels of hierarchical organi-zation and represented in memory so that recurring patterns (withvariations) can be recognized later in a piece. Essentially, this aspect of atheory should describe the auditory analysis of the thematic musicalmaterial used by a composer that is subsequently re-presented in variedforms in the development of a composition. The thematic material maybe considered as the musical lexon used by the composer within theconfines of the grammar of his musical language to construct a musicaldiscourse. A great deal of description of how variation and transforma-tion is effected in a musical discourse has been offered by musictheorists, but what music theory itself has not addressed until recently isthe psychological nature of the listening activity and its tendencies andlimits of analysis and reconstruction of musical form as the sound eventscome incessantly through time to the ears of the listener.

A laudable attempt at a more listener-oriented music theory has beenrecently published by Lerdahl & Jackendoff (1983) who develop acognitive theory of musical competence for listeners experienced in theclassical Western tonal idiom (e.g. the music of Bach, Haydn, Mozartand Beethoven). The major achievement of this theory is its formalexpression which should in practice lend itself to scientific verification.A primary focus of the theory is how a listener hierarchically segments astream of musical events (the musical surface structure) into motives,phrases and sections and derives a metric structure from the patterns ofstrong and weak events. The authors rightly intuit that a music theorywhich is "generative" (in the sense of Chomsky, 1965) must not onlyassign structural descriptions to a piece (of which many are logicallypossible), but must, as well, describe how a listener "selects" among thepossible and arrives at a single heard structure at a given listening. Toaccomplish this, two types of structuring rules are postulated: the well-formedness rules which specify the possible structural descriptions (theensemble of rules corresponding more or less to a mathematicaldefinition of hierarchy) and the preference rules which designate thosedescriptions which correspond to a listener's preferred hearing of thepiece. The well-formedness rules put limits on what constitutes anallowable group structure and represent the rules of grammar, while thepreference rules correspond more to perceptual laws of organization ofform. The relative importance of the preference rules is a reflection ofthe sense of music as more or less pure structure to be manipulatedpsychologically, within certain bounds, by the listener. It is in relation tothe nature and organization of preference rules, which may beconsidered as laws of aesthetic preference modulated by culturalconvention and personal experience, that one can eventually beginseriously to entertain questions about aesthetic aspects of musicalexperience.


46 Stephen McAdams

Experimental work suggests that Lerdahl & Jackendoff's conception ofthe ensemble of preference rules for grouping structure is reasonablysatisfactory for predicting the responses of listeners. Deliege (1985, inpress) proposes a repartitioning of the rules based on perceptualmechanisms, as opposed to music theoretic dimensions. The two kindsof mechanisms that appear to effect segmentation operate upon 1)spectral continuity which is directly related to changes in pitch, timbreand dynamics (as with sequential grouping processes), and 2) a numberof temporal factors such as change in duration, articulation and restsbetween groups of notes. This repartitioning may both simplify the ruleset and better predict the nature of structural ambiguities that do resultwhen rules enter into conflict with one another. Also, these perceptualmechanisms appear to be closely related to those responsible forsequential auditory organization, indicating that segmentation groupingcould influence event stream formation, i.e. influence the constructionof the "musical surface" which should ostensibly already be organizedinto source streams. The construction of the "musical surface" isaccepted as a given that is not even conceivably operated upon byhigher level processes in Lerdahl & Jackendoff's theory. This shows thebias of the theory as one of musical structure representation more thanof structural processing. These issues will eventually need to beaddressed in further refinements of the theory.

Another area not considered by Lerdahl & Jackendoff which a morecomplete psychological theory would need to address is the involve-ment of memory processes in the storage, categorization and laterrecognition of musical motives and themes. Segmentational grouping isoften referred to as "chunking" in memory research. A great deal ofwork has shown that there are severe limits on how much informationcan be held in short-term memory (STM) which is a kind of temporarystorage for information that has recently arrived through the senses orhas been retrieved from long-term memory (LTM). Remember thatabout 5-9 organized chunks of information can be held in STM. A chunkcan be as simple as a single note, and as complicated as several notesarranged in a small hierarchical organization, a phrase for instance. Themore organized the information is, the more of it one can hold on to. It isconvenient to consider these organized chunks as musical fragments.

What this research implies for musical organization is that segmentsthat are to serve as musical material for further development must beeither very simple or well-organized, making them unified enough to bea chunk in STM, and small enough to fit into a limited time frame, whatmight be called the window of the perceptual present. A grouped fragmentwill only be successful as musical material insofar as it can be stored andcan serve as a pattern against which later variations or related structuresare recognized or compared. It is certain that the nature of memoryencoding places certain limits on the form that groups can assume andstill be easily recognized in order to "tie together" similar groups at moredistant points in a composition. For a segment to contribute to theperception of relatedness with variations or transformations of the



material, it must be striking enough to have been presented enoughtimes to be stored in LTM. This may occur through repetition within agiven piece or across several hearings of a piece. This shows the primaryimportance for comprehension of redundancy and repetition in music.To comprehend large-scale' temporal structures, their building blocksmust be storable in LTM in order to contribute to the building ofstructural relations. What are equally important are the kinds of mentalschemata for music one has acquired, since the more familiar thematerial is stylistically, the easier it will be to store in LTM as well.

Considering the segmentation of the musical surface into groups as acrucial step in the extraction of the musical vocabulary of a piece raisessome interesting issues about the possible nature of the material ofmusic and the structural limits of a musical "object." This kind offormalization of the problem should eventually allow experimentationon the degree to which the segmentation criteria are universal orculture-specific. (See Reynolds, this volume for a consideration of theimportance of recognition and identifiability of musical "cells" in theconstruction of musical form.)

Building structural relations10

This level of the theory would describe the way in which relationsamong musical elements assembled at lower levels begin to give rise tomental structures (Figure 4). Part of this process involves deriving ahierarchy of structurally important events or motives with respect totheir position in the grouping organization determined in the previousstage. As such it corresponds partially with the "metrical structure" and"time-span reduction" rule sets of Lerdahl & Jackendoff (1983). Themetrical structure rule set derives a hierarchical ordering or ranking bydegree of importance of the beats that a listener attributes to a piece. Thetime-span reduction rule set determines the relative structural import-ance of pitch events within the heard rhythmic units of the piece. In fact,Lerdahl & Jackendoff's theory is admitted to be purely hierarchical, nottaking into account associative relations in the description of musicalstructure.

The main issues at this level should include the categorization ofmusical material, the recognition of similarity or relatedness of varioustransformed versions of the material, the encoding in memory ofassociative relations and of hierarchical relations, and the emergence ofvarious structural properties. It is not yet clear to what extent musicalqualities such as timbre and texture lend themselves to hierarchicalorganization, though serious attempts are being made to move in thisdirection (see papers by Lerdahl and Saariaho, this volume). It is quiteclear, however, that timbre can play a strong role in one's perception ofthe structure of the piece. It seems possible that building timbreorganizations on the basis of consonance and dissonance in a muchlarger sense of these words could give rise to a sense of directed motion


48 Stephen Me Adams •

and thus to the kinds of anticipatory mental schemata that are therequisite for a hierarchical form-bearing dimension.

The apprehension of form in music obviously depends on theperception of the structure of events across time (see papers by Clarkeand Grisey, this volume). We need to investigate the extent to whichand the manner in which this structure can be apprehended by alistener. The extent partially depends on morphophoric, or form-bearing,capacities of the sound medium with respect to the perceivingindividual. These capacities represent basic biopsychological constraintson the apprehension of form. The manner depends on the degree andtype of musical experience and training of the listener.

An important question is the way in which and the degree to whichmusical material can be transformed or varied and still be perceived asrelated to previously presented material. The perception of the related-ness of musical elements separated in time and the perception ofredundancy partially determine the perception of temporal form.Redundancy results from a strong perception of relatedness. The notionof relatedness bears very hevaily on the nature of the constraints onform-bearing elements in music, i.e. one cannot simply choose anyarbitrary combination of notes or organize variations of that combinationand expect in all cases that it will give rise to a strong or perceivablemusical form. Large-scale form is in some sense limited by the nature ofthe material that is structured to give rise to that form, and the task ofpsychological theory is to understand the limits of the cognitivecapacities of a listener to process such material. The perception ofrelatedness and the degree of relatedness depend on the perception ofsimilarity under various classes of transformation of musical materialand the nature of morphophoric capacities determines the kinds ofpossible transformation. If some musical element is easily remembered,and its transformations easily recognized, its potential contribution to aform will be greater than another element which is very difficult toremember.

The relation between the mechanisms of grouping and the perceptualdimensions along which groups are organized is of paramount import-ance for considerations of which perceptual dimensions are capable ofcarrying musical form, such as pitch, rhythm, and some dimensions oftimbre. One can easily imagine the formation of groups with similarpitch, timbre and rhythmic content, but it is less likely that one wouldgroup musical notes on the basis of the amount of vibrato present, forexample. From recent investigations in the realm of musical timbre haveemerged several criteria for musical form-bearing elements. Theseinclude the following six criteria (from McAdams & Saariaho, 1985).

Form-bearing elements are perceptually differentiated into discrete categories.This means they are either artificially restricted to differentiablecategories, as is the case with musical pitch, or a continuous acousticdimension is broken into categories by the auditory system, as is thecase with speech phenomes. We remember discrete dimensions muchbetter than continuous ones. This is not to say that in the case of musical



pitch, for instance, we cannot appreciate whether a note is out of tune ornot: of course,-most of us can; but if a melody is played that continuallychanges in pitch and only passes through the appropriate pitch at theright time, one generally has a difficult time remembering this form inorder to recognize it later or to compare it with another similar form.Where a categorized dimension may serve the building of structure,continuous variation along this dimension may serve for expressivetransformations of the "ideal" categories as we find in expressiveintonation and in micro-tonal ornamentation in Indian and Arabicmusic, or in micro-temporal variations in expressive rhythmic perform-ance (Clarke, 1985). The classification of points along a single dimensionor several correlated dimensions helps also with the characterization ofthe properties of musical entities. It is with the contrasts between theseentities that musical structure is built.

The perceptual categories are ordered so that the relations among them arefunctional. This relates to the syntactic characterization of the propertiesof musical entitites, and is a very strong characteristic of the pitches inmodal and tonal music systems, e.g. the functional relationship betweenthe seventh degree of the scale and the tonic is different than thatbetween the seventh and the sixth. These relations are not, of course,entirely independent of the context, but they certainly have strongtendencies of their own due to their position within the system. Thisordering may be of different types, e.g. the hierarchical ordering oftonality or the serial ordering of dodecaphonic music. The way ofordering often places severe constraints on the possibilities of musicalform that can successfully be realized within such a system. Theclassification schemes and their ordering must therefore reflect psycho-logical possibilities or these structures will not be decodable by thelistener and thus not contribute in themselves to an appreciation ofmusical form. One of the contemporary compositional problems is theinvention of psychologically relevant functionalities within the increas-ing number of musical dimensions of compositional interest.

The functional relations are of varied strengths and types which allow for thebuilding and release of tension. In tonality, for example, the strength of therelation between the first and fifth degrees of the scale is much greaterthan that between the third and the fifth, and indeed such a movementis structurally less important for tension than in the former case. Thereare many different ways to describe the function and strength ofrelations among entities which include similarity and dissimilarity,consonance and dissonance, dominance and submission, and so on. It isimportant that the nature of the relations be relatively independent ofthe local context of the piece and yet closely bound to the "context" ofthe ordering system itself in order to have a generality of application.

Attention can be paid directly to qualities of a category, to qualities of relationsamong categories, or to combinations of relations. With pitch, for instance,one can focus on the pitch itself, on the quality of an interval betweentwo pitches or on the quality of a chord as an assembly of intervals. Indifferent musical contexts, one or the other of these levels of attention


50 Stephen McAdams

are more important for the extraction of musical structure. Thisdemonstrates the importance of emergent properties of perceivedrelations or groupings, a realm where the psychological processes ofauditory organization play a strong role in what is available as musicalmaterial emerging from the acoustic structure at a given moment. Thereis obviously an interest in combining qualities as well as their emergentrelations. This raises compositional problems concerning the interde-pendence of manipulating both musical entities and the properties thatemerge from their interactions.

The categories, functional relations and ordering within a classification systemmust either reflect the existing structure of the mind and world or be susceptibleto learning by listeners if the structures are to be apprehended as musical form.This is a criterion that evokes the role of mental schemata in theorganization of perceptual experience. Obviously, one of the mostimportant elements here is the categorization, structuring and thencommitting to memory of experience. Important work in the develop-ment of perceptual skill has shown that the basic machinery forinterpreting the world exists at birth and that perceptual learning is a •process of progressive discrimination, or reduction of discrimination,i.e. the elaboration of innate schemata and the further division ofexisting categories, or the removal of some pre-existing boundariesbetween categories that have no functional value.

Relations among categories must be able to maintain a certain degree ofinvariance under various classes of transformation. If patterns composedwithin a set of dimensions are not susceptible to being varied and stillbeing perceived as similar, then the set of dimensions cannot be str6ngcontributors to musical form. After all, variation and transformation aresome of the main features of music throughout the world. The limits ofperceiving a transformed pattern as somehow related to an original onepoint to psychological limits of viable processes of musical transforma-tion. Pitch patterns, for example, can be transposed, played in a relatedkey, expanded or contracted, and still be perceived as related to theoriginal motif. Psychologically this implies a mental representation ofthe original concept or model (motif) which maintains certain propertiesin its structure during these transformations and which are perceivableas such (similar contour, for example). An exploration of this domaincould ultimately open up a wide territory of rich functional substitutesfor the process of variation. However, it also implies limits to thepossibilities of transformation through the many quality dimensions ofmusical entities, and limits to the possibilities of significant coherentstructuring of musical material that is to be transformed.

More concerted theoretical and experimental efforts are needed tomap out the kinds of transformations that are possible for each potentialmorphophoric dimension, such as pitch, rhythm and timbre. It remainsto be seen to what extent the various "dimensions" of timbre (such asbrightness, roughness, harmonicity, attack quality, etc.) are capable ofcarrying structural relations.

The psychological processes that underly the perception of related-



ness would include processes of comparison for similarity of informa-tion just received in short-term memory and structures previouslystored in long-term memory. This comparison can be conceived of as anactivation of associations between the segment in STM with the materialin LTM. What is of primary importance here is the process of activationthat allows us to recognize the previously presented material. Theremust be a relation of the degree of similarity to the strength of thisrecognition and thus to the strength of the contribution that the currentsegment makes to the apprehension of formal relations. One wouldexpect, for example, that an identical re-presentation of a theme wouldhave a very strong association with its earlier presentation; whereas aretrograde-inversion of the theme would have a very weak associationor none at all. In the former case a stronger structural tie would resultthan in the latter case. Though hierarchical rather than assodational, theprolongational reduction in Lerdahl & Jackendoff's theory shows acertain commonality with these aspects of the representation of musicalform. (See also paper by Reynolds, this volume, on variation andtransformation of musical material.)

It should be emphasized again that this process is limited by theoperations of the organizational processes. A transformed segmentcannot be too long in time or be too complicated or it would not be easilystored in short-term memory for comparison. It cannot have its pitchintervals stretched too much at a faster tempo, for example, because thesegment would tend to segregate into other segments and, first of all, beno longer processed as a whole segment and, consequently, not berecognized as related to its untransformed predecessor.

Another important psychological issue is the way in which thestructuring of musical material gives rise to the perception of moreglobal properties of structure such as tempo, meter, key, interval, set,harmonic progression, textural evolution and large scale control ofspectral/registral form. These last two properties are masterfully used bythe composer Ligeti who sculpts massive trajectories of timbre andtexture through time. We urgently need ideas of how to proceedexperimentally in the understanding of the contribution of organizatio-nal processes to perception of global properties.

Following a musical discourse

At this level the psychological issues include questions of how oneperceives and represents the flow of musical meaning: the developmentof tension and relaxation, a sense of continuity and progression, thecreation of expectation and a sense of directed motion, and the natureand limits of auditory attention as a listener "participates" (perhaps byconscious introspection) in the re-creation of a musical form. It partiallycorresponds to the "prolongational reduction" rule set of Lerdahl &Jackendoff. This rule set derives a hierarchy of the stability of pitchevents in terms of perceived patterns of tension and relaxation.


52 Stephen McAdams

The major psychological question concerns the nature of the processesthat create models or schemata for what the listener believes the musicto be doing and where he believes it to be going (either tacitly orconsciously). This kind of mental model, in a sense, inspires expectationif the musical language is accessible to the listener. Anomalies in whatactually takes place with respect to the mental model can introducestrains in the expected structural completion and give rise to musicaltension. It is probably at this level that one can begin to considerseriously the structural source of emotional experience in relation tomusic.

The temporal perspective necessary in music is a kind of veil thatrequires we make decisions about our directions, and the directing ofattention serves to define immediate or local structural disposition. Thisis one of the areas that allows the richness of musical experience in thatthere are always several paths to be taken in the accumulation of amusical form. Another psychological issue concerns how the assemblyof mental schemata that create expectations feeds back to the lower levelorganizational processes and influences what is actually constructed andexpected as a musical form.

It is at this level that one can really begin to deal with the notion ofmusical grammar. There are clearly similarities and differences betweenmusical and language grammars, but not a great deal of clarity about thenature of these differences. The problem is badly in need of beingproperly situated in the framework of cognitive science. Sloboda (1985,chap. 2) poses the following question with respect to musical grammar:what does the structure of music tell us about the structure of mind?One view is that the rules of grammar are psychological procedures thatare used to organize music. The problem with this is that there is nosuch thing as a unique grammar for any body of data on listeners'responses to music. Humans tend to violate the rules that seem toaccount for their behavior which means the grammars are wrong andthat the rules are only a rough approximation. Another view is thatsome features of grammar have implications for the general way inwhich we think about psychological principles: a) we decide what toemit by applying a finite set of rules to some unspoken thought we wishto utter, whether this utterance be a musical phrase or a sentence inlanguage, b) listeners can sort utterances into acceptable and unaccept-able, c) structures embodied in generative grammars have psychologicalreality: elements closely linked in a grammar are closely linkedpsychologically, whatever the form of cognitive representation. The firstof these features applies more to composition and improvisation than tolistening. As with language, we might imagine that from the flash ofmusical insight a composer has, there are any number of ways of"uttering" a musical sequence that realizes this insight structurally. Theschemata (or conceptual tools and technical training) that a composer orimprovisor has at his or her disposal can be a limiting factor in how theinsight is made concrete. The second feature has some rather heavyimplications with respect to what is conventional and may be too



restrictive of a concept for many composers' tastes. What is acceptablerelates, of course, to an acquired sense of what conforms to theconvention of the grammar in question. Since the possible grammars ofmusic have multiplied significantly in the latter part of our century, it ishard to imagine what acceptable or unacceptable means in any generalsense, even if one were to limit the population of people to which thegrammars apply to those highly experienced in listening to and studyingcontemporary music.

There have actually been some reasonably successful attempts atexperimentation on the psychological reality of conventional musicalgrammars. Stoffer (1985), for example, employed techniques drawnfrom psycholinguistics to study the representation of phrase structure inmusic perception. He found evidence that listeners use tacit (orunconscious) knowledge about phrase structures expected in musicalforms taken from the Western tonal/metric idiom and that thisknowledge structure is organized in the form of hierarchies of mentalrepresentations of musical concepts. Of interest to our consideration inthe processing of musical discourse is his claim that the act of "listeningattentively to music may be characterized by an interaction betweenseveral cognitive schemata that form a hierarchy of musical conceptsaccording to abstractness or generality" (p. 217). The results demons-trate the value of his syntax model for modelling structural aspects of theprocessing of perceived music.

Nonetheless, there are restrictions on the way we might postulate agrammar to be used either for production or for comprehension. It isclear that in music, the grammar is conformed to in a rather loose way.In composing, one may know a grammar and be able to adhere to it.However, the composer also insists on moving outside the grammar.Having a global view of it, one can anticipate the listening strategies thatare derived from our own culture and which are used to structure ourmusical experience, and seek ways to thwart these strategies ininteresting ways. What constitutes an interesting way that is stillsomehow understandable is an important question in the realm ofaesthetics. Grammars that are shared among a community of musiclisteners, however, do not generate musical compositions. Theirexistence is a major consideration in determining the nature and degreeof freedom a composer can exercise to transform and extend a musicalstyle (Sloboda, 1985, chap. 2). It seems, though, that there is someconfusion in 20th Century music between stepping outside a consensualgrammar and creating a set of arbitrary methods of composition andstepping outside of those. While the composer may think this createstension, it presumes the listener "feels" the method in order to be able toexperience a departure from it as a violation or deviation of expectation.

The important element here is the necessity of a commonly shared setof mental schemata that allows the listener to sense the music as goingin the way the composer thinks it is going. The primal importance ofdirected motion in music and of the psychological constraints on itsexperience comes very much to the fore. It is here as well that the great


54 Siephen McAdams

importance of the nature of underlying mental representations for theprocesses of organization becomes apparent. Directed motion dependson several things. First of all, motion of any kind is perceived withrespect to a stable ground. In the case of music, we might imagine thatthe stable ground is provided by processes that seek stable mentalstructures through which the incoming information is organized onmany levels (see Povel & Essens, 1985, on the perception of temporalpatterns in this regard). This bias toward regularity instantiates a kind ofperceptual inertia. To move away from stability is to induce a sense ofmotion in a psychological sense. But for the motion to appear directed,some kind of trajectory must be inferred, some path of transformationhas to be understood, and mental schemata must have been acquiredthat anticipate an end point of this trajectory. Thus, one must have asense not only that something changes but must sense as well the natureof the process of change and the direction implied by this process. Thesekinds of anticipatory mental schemata direct our attentional processes tocertain aspects of the musical structure, prime us for the arrival ofcertain things. It is in the time course of the trajectory that projectstoward the anticipated event's arriving that, according to Meyer (1956),lie the seeds of affect and meaning in musical experience. There are,however, many unanswered questions from a psychological standpointabout the nature of directed motion and the understanding of the flowof musical discourse.

This final level of the theory must integrate all the aspects of internalrepresentation of musical elements, from the smallest to the largestlevels of structure, and the processes by which these are accumulated inmemory and give rise to our experience of form. This addresses againthe major question of assembling a received structure into an experi-enced form.

The apprehension of musical form

Structure is in the world, either notated on paper, stored in computermemory, on magnetic audio tape, impressed on vinyl, or present asvibrations in the air. Form is in the mind and is thus limited by thepossibilities of mind. In the preceding pages we have rapidly scannedsome of the psychological considerations of what underlies the derivingof musical form from the acoustic structures presented to our ears. Themind seeks a coherence of organization at various levels of structure. Assuch, comprehension implies the presence of internal representationsand of organizational processes that operate upon these mentalstructures.

Langer (1942) proposes the notion of structural resonance to approachthe problem of the origins of meaning and emotional response to music.The structural resonance hypothesis can be viewed as a body ofpsychological constraints. There must be a logical resemblance of thestructure of music and experience. When these symbolic structures (or



mental schemata) are similar, an important requirement for a connota-tive relationship is- satisfied and musical structures can have meaning.The problem of the psychologist is to show that this resemblance of formexists at all of the levels believed to be involved in music comprehensionand to demonstrate the resonance that is believed to take place. Thistask is most likely far beyond our scientific capabilities at present, butthere are several notions here that recall things we have alreadydiscussed. A theory of musical organization based on the assembly ofmental schemata, including their anticipatory nature, their ability to beassembled into hierarchies and our tendency to use them from higherlevels to lower ones in the recognition and processing of complexstructures, lends itself very well to a move in this direction.

There are several properties of musical structures that recommendthem for symbolic use and for the eventual extraction of musical"meaning." They are composed of separable items that are easilydistinguishable, easily produced and easily combined in a great varietyof ways. These are, in effect, psychological constraints on form-bearingelements in general. The criterion of categorization implies also theimportance of having prototypes of categories that are the more stableand easily recognized Gestalt forms of which well-formed musicalmotivic material is made. These discrete identities are essential for thebuilding of complex syntax and structure. These forms must not only bereadily distinguished one from another, but must also be readilyremembered and repeated. However, as Boulez (this volume) remarks,the building blocks of musical structure must also have a certainneutrality of identity if they are not to invoke too strong of a centrifugalforce that thrusts them into conflict with the structure of which theyshould be a part. They must not play too strong of a role on their ownwhich overshadows their structural and semantic function in the music.A relative neutrality of identity allows a certain degree of blending and acertain malleability of function by which each element has a tendency tomodify each other's character when they are put into combination. Thisis a crucial factor for a semantic network, that each element be able toserve each other element as context.

One of the problems with musique concrete is that the sound elementsselected as a vocabulary for a given piece often have such strongreferences to everyday life that they are made to cohere with anoverriding structure only with great difficulty. They tend to stand out ontheir own and thus do not contribute to the more global structure. In asense, the material is not only too identifiable but is also too discon-tinuous or categorized to be assimilable into a form that is foreign to itsalready strong semantic function.

Deriving a form from a structure requires, in addition to categorizationof the material, a comprehension of its transformation and elaboration.The process of transformation must be sensed, if not completelyunderstood. It is, of course, not very interesting if everything thathappens in a composition is absolutely transparent. One is bored veryquickly of this type of music. However, some impression of the nature


56 Stephen McAdams

and logic of transformation is essential for one to appreciate thedirectionality of music. Boulez (1986) states that there is the necessity ofa heavy directionality in the nature of a device or method in order for itto be useful musically. This must also be in the direction of the musicalnecessity. Transformations must be non-neutral.

The question concerning the relation between structure and form inmusic in some broad sense asks what is the nature and, above all, whatare the capacities, tendencies and limits of the comprehension of music?What does it mean to "comprehend" or "apprehend" music? Whatmight the sciences of human cognition bring to the search forunderstanding? Are there universal and culturally bound aspects tomodern comprehension? How is music comprehension affected byincidental experience with music and by musical training? To whatextent is comprehension tied to conscious awareness of what one isexperiencing? In this sense we speak often of "feeling" more than"understanding" musical form, of "apprehending" more than "perceiv-ing." Rosen (1971, p. 35) remarks on the psychological reality ofstructural hearing approaches to music analysis and states that thenotion that these structures are consciously perceived is unsatisfactory.It "reduces" a piece too far with respect to important heard elements andneeds to be further developed with a consideration of perceived andstored motivic elements and their relation to an inferred structure. Thestructure is extended over a much greater expanse of time than we canconsciously perceive. Our limits of appreciation of musical form are thusdetermined by the limits of what we can represent in memory. Theextent to which we can follow a musical discourse depends on ourhaving previously acquired the necessary grammatical structures or onour being able to infer/reconstruct these structures from the musicalmaterial itself. This is the process of accumulating and experiencing amusical form. This process is an essential substance of music itself. Theexperience of music involves acts of the mind and the study of thisexperience and of the human knowledge embodied in musical culturewarrants the attention of the sciences of mind. In reading the reflectionsof the musicians and psychologists in the following articles we shouldkeep the reciprocal question in view: how well does this direction ofthought correspond with and nourish musical thought itself?

Acknowledgements

I am deeply indebted to four recent works that were the source of muchinspiration for this article, namely, Spender (1980), Lerdahl & Jackendoff(1983), Sloboda (1985) and Dowling & Harwood (1986). I would like toexpress my gratitute to Deke Dusinberre, Dick Carter and MarcoStroppa for thoughtful and thorough critiques of the manuscript. Iwould also like to thank Nigel Osborne for the possibility of realizingthis ambitious project. Finally I would like to thank my musical andscientific colleagues at IRCAM for inspiration and for keeping me honest



and more or less on the track. Of course, any detours that are less thanmusical are no fault of theirs.

Notes

1. Much of the material in this section was derived from a thorough review of the historyof music psychology in Spender (1980, section I).

2. This summary of Sloboda's (1986) paper is derived from a recording of an oralpresentation of the material given to the Belgian Psychological Society seminar on thePsychology of Music in Brussels in December 1985. There may have been changes inthe printed version to which the reader is referred for a more thorough treatment ofthe subject.

3. The readers are invited to verify for themselves whether this coherence is actuallytaking form by consulting the several new books and journals on music psychologythat are listed in the references.

4. Some people question whether we can even still speak of "language" anymore withrespect to Western contemporary music. We would seem to be in a period after the fallof the tonal Tower of Babel, where there appear to be as many "languages" as thereare composers. Can we still call it "language" if it is not "conventional" in the sense ofbeing shared by a population of people? Or does a global look at these musics reveal acertain number of common characteristics that will eventually be absorbed by theculture?

5. It should be understood that the notions of representation and computation are termsthat we use in order to express certain generalizations about what we believe to behappening in the human mind (Pylyshyn, 1985, p. 1). We use the concepts to explainthe way the mind functions. In order to explain things, we need a vocabulary thatallows the concepts to be communicated.

6. Much of the material briefly reviewed in this section was drawn from two importantrecent books on music cognition: The Musical Mind by John Sloboda (1985) and MusicCognition by Jay Dowling & Dane Harwood (1986).

7. There is a vast amount of research on memory processes. For a fairly completeintroduction to current thought in this field refer to chapters 8-11 in Lindsay &Norman (1977) and chapters 6-7 in Anderson (1985).

8. See the special issue of The Behavioral and Brain Sciences, vol. 3, no 1, March 1980, onthe foundations of cognitive science.

9. For more in-depth reading on this level of perception refer to Moore (1982) and Plomp(1976).

10. The discussion of the next two levels of musical organization will necessarily be verygeneral and speculative as there has not been much research on the question.

References

J.R. Anderson, Cognitive Psychology and its Implications, 2nd ed. (W.H. Freeman, New York,1985).

S. Arom, Structuration du temps dans les musiques d'Afrique centrale: périodicité, mètre,rhythmique et polyrhythmie, Revue de Musicologie, 70, 5-36 (1984).

S. Arom, Polyphonies et polyrythmies d'Afrique centrale: Structure et méthodologie,Doctorat d'Etat thesis, University of Paris — Sorbonne (1985).

L.E. Bahrick, A.S. Walker & U. Neisser, Selective looking by infants, Cognitive Psychology,13, 377-390 (1981).

G.J. Balzano, The pitch set as a level of description for studying musical pitch perception,in M. dynes (ed.), Music, Mind and Brain: The Neuropsychology of Music, pp. 321-351(Plenum Press, New York, 1982).


58 Stephen McAdams

F.C Bartlett, Remembering: A Study in Experimental and Social Psychology (CambridgeUniversity Press, London, 1932).

D. Berlyne, Æsthetics and Psychobiology (Appleton-Century-Crofts, New York, 1971).J. Bloch & R.B. Dannenberg, Real-time computer accompaniment of keyboard perform-

ance, Proceedings of the 1985 International Computer Music Conference, Vancouver, BritishColumbia, Canada, pp. 279-290 (Computer Music Association, Berkeley, 1985).

A.S. Bregman, Asking the "what for" question in auditory perception, in: M. Kubovy & J.Pomerantz (eds.), Perceptual Organization, pp. 99-118 (Erlbaum, Hillsdale, N.J., 1981).

A.S. Bregman, J. Abramson, P. Doehring & C. Darwin, Spectral integration based oncommon'amplitude modulation. Perception and Psychophysics, 37, 483-492 (1985).

A.S. Bregman & J. Campbell, Primary auditory stream segregation and the perception oforder in rapid sequences of tones, Journal of Experimental Psychology, 89, 244-249(1971).

A.S. Bregman & S. Pinker, Auditory streaming and the building of timbre, Canadian Journalof Psychology/Revue Canadienne de Psychologie, 32, 19-31 (1978).

P. Boulez, Composition et technologie, conference presented in the seminar NouvellesTechnologies et Mutation des Savoirs, IRCAM, Paris, France (October, 1986).

X. Chabot, R.B. Dannenberg & G. Bloch, A workstation in live performance: Composedimprovisation, Proceedings of the 1986 International Computer Music Conference, DenHaag, The Netherlands, pp. 57-60 (Computer Music Association, Berkeley, 1986).

N. Chomsky, Aspects of the Theory of Syntax (MIT Press, Cambridge, Mass., 1965)E.F. Clarke, Structure and expression in rhythmic performance, in: P. Howell, I. Cross &

R. West (eds.), Musical Structure and Cognition, pp. 209-236 (Academic Press, London,1985).

G.L. Dannenbring & A.S. Bregman, Stream segregation and the illusion of overlap, Journalof Experimental Psychology/Human Perception and Performance, 2, 544-555 (1978).

C.J. Darwin, Perceptual grouping of speech components differing in fundamentalfrequency and onset-time, Quarterly Journal of Experimental Psychology, 33A, 185-207(1981).

C.J. Darwin, Perceiving vowels in the presence of another sound: Constraints on formantperception, Journal of the Acoustical Society of America, 76, 1636-1647 (1984).

C. Delezenne, Mémoires sur les valeurs numériques des notes de la gamme, Recueil destravaux de la Société des sciences, de l'agriculture et des arts, de Lille, 1, 1-56 (1826-7).

I. Deliège, Les règles préférentielles de groupement dans la perception musicale, thesis,Université Libre de Bruxelles, Brussels, Belgium (1985).

I. Deliège, Grouping conditions in music listening: An approach to Lerdahl & Jackendoff'sgrouping preference rules, Music Perception, (in press).

D. Deutsch, The processing of structured and unstructured tonal sequences, Perception andPsychophysks, 28, 381-389 (1980).

W.J. Dowling, Rhythmic groups and subjective chunks in memory for melodies. Perceptionand Psychophysks, 14, 37-40 (1973).

W.J. Dowling, Melody information processing and its development, in: D. Deutsch (ed.),The Psychology of Music, pp. 413-429 (Academic Press, New York, 1982).

W.J. Dowling & J.C. Bartlett, The importance of interval information in long-term memoryfor melodies, Psychomusicology, 1, 30-49 (1981).

W.J. Dowling & D.S. Fujitani, Contour, interval, and pitch recognition in memory formelodies, Journal of the Acoustical Society of America, 49, 524-531 (1971).

W.J. Dowling & D. Harwood, Music Cognition (Academic Press, New York, 1986).D. Ehresman & D. Wessel, Perception of timbrai analogies, Rapports IRCAM, no 13

(IRCAM, Paris, 1978).R. Erickson, Sound Structure in Musk (University of California Press, Berkeley, 1975).R. Erickson, New music and psychology, in D. Deutsch (ed.). The Psychology of Musk, pp.

517-536 (Academic Press, New York, 1982).G. Fechner, Elemente der Psychophysik (Breitkopf und Härtel, Leipzig, 1860).G. Fechner, Vorschule der Æsthetik (Breitkopf und Härtel, Leipzig, 1876).P. Fraisse, Time and rhythm perception, in: E.C. Carterette & M.P. Friedman (eds.).

Handbook of Perception, vol. 8: Perceptual Coding, pp. 203-254 (Academic Press, NewYork, 1978).



P. Fraisse, Rhythm and tempo, in: D. Deutsch (ed.), The Psychology of Music, pp. 149-180(Academic Press, New York, 1982).

R. Francès, La perception de la musique, 2nd ed. (J. Vrin, Paris, 1984).J.L. Gould & P. Marier, Learning by instinct, Scientific American, 256(1), 62-73 (1987).J.M. Grey, Multidimensional perceptual scaling of musical timbres, Journal of the Acoustical

Society of America, 61, 1270-1277 (1977).J.W. Hall, M.P. Haggard & M.A. Fernandes, Detection in noise by spectro-temporal

pattern analysis, Journal of the Acoustical Society of America, 76, 50-56 (1984).L. Harrison, Lou Harrison's Music Primer: Various Items About Music to 1970 (C.F. Peters,

New York, 1971).W.M. Hartmann, S. McAdams & B.K. Smith, Matching the pitch of a mistuned harmonic

in an otherwise periodic complex tone, Journal of the Acoustical Society of America,80,S93(A) (1986).

H. von Helmholtz, On the Sensations of Tone as a Physiological Basis for the Theory of Musk,2nd English ed. trans. in 1885 by A.J. Ellis from the 4th German ed. 1877 (reprintedDover, New York, 1954).

J. Hochberg, Organization and the Gestalt tradition, in: E.C. Carterette & M.P. Friedman(eds.), Handbook of Perception, vol. 1: Historical and Philosophical Roots of Perception, pp.180-211 (Academic Press, New York, 1974).

R.S. Jackendoff, Semantics and Cognition, (MIT Press, Cambridge, Mass., 1983).N.F. Johnson, The role of chunking and organization in process of recall, in: G. Bower

(ed.) Psychology of Language and Motivation, vol. 4 (Academic Press, New York, 1970).D. Klahr, W.G. Chase & E.A. Lovelace, Structure and process in alphabetic retrieval,

Journal of Experimental Psychology/Learning, Memory and Cognition, 9, 462-477 (1983).W. Köhler, Gestalt Psychology (Horace Liveright, New York, 1929).S. Kosslyn, Image and Mind (Harvard University Press, Cambridge, Mass., 1980).C.L. Krumhansl, The psychological representation of musical pitch in a tonal context,

Cognitive Psychology, 11, 346-374 (1979).C.L. Krumhansl, J. Bharucha & M.A. Castellano, Key distance effects on perceived

harmonic structure in music, Perception and Psychophysics, 31, 75-85 (1982).C.L. Krumhansl & F.C. Keil, Acquisition of the hierarchy of tonal functions in music,

Memory and Cognition, 10, 243-251 (1982).C.L. Krumhansl & R.N. Shepard, Quantification of the hierarchy of tonal functions within

a diatonic context, Journal of Experimental Psychology/Human Perception and Performance,5, 579-594 (1979).

T.S. Kuhn, The Structure of Scientific Revolutions, 2nd ed. (University of Chicago Press,Chicago, 1972).

S. Langer, Philosophy in a New Key (Harvard University Press, Cambridge, Mass., 1942).F. Lerdahl, Cognitive constraints on compositional systems, in: J. Sloboda (ed.). Generative

Processes in Musk (Oxford University Press, Oxford, in press).F. Lerdahl & R.S. Jackendoff, A Generative Theory of Tonal Music (MIT Press, Cambridge,

Mass., 1983).F. Lerdahl & R.S. Jackendoff, An overview of hierarchical structure in music, Musk

Perception, 2, 229-252 (1984).P.H. Lindsay & D.A. Norman, Human Information Processing: An Introduction to Psychology,

2nd ed. (Academic Press, New York, 1977).H.C. Longuet-Higgins, Perception of melodies, Nature, 263, 646-653 (1976).H.C. Longuet-Higgins & C.S. Lee, The rhythmic interpretation of monophonic music,

Music Perception, 1, 424-411 (1984).R.W. Lundin, An Objective Psychology of Music, 2nd ed. (Ronald Press, New York, 1967).E. Mach, Beiträge zur Analyse der Empfindungen (Jena, 1886) [cited in Spender (1980)].S. Makeig, Affective versus analytic perception of musical intervals, in: M. Clynes (ed.),

Music, Mind and Brain: The Neuropsychology of Musk, pp. 227-250 (Plenum Press, NewYork, 1982).

D. Marr, Vision (W.H. Freeman, New York, 1982).S. McAdams, Spectral fusion and the creation of auditory images, in: M. Clynes (ed.)

Music, Mind and Brain: The Neuropsychology of Music, pp. 279-298 (Plenum Press, NewYork, 1982).


60 Stephen McAdams

S. McAdams, The auditory image: A metaphor for musical and psychological research onauditory organization, in: W.R. Crozier & A.J. Chapman (eds.), Cognitive Processes inthe Perception of Art, pp. 289-324 (North-Holland, Amsterdam, 1984). (a)

S. McAdams, Spectral fusion, spectral parsing and the formation of auditory images, PhD.dissertation, Stanford University (available as Dept. of Music Report STAN-M-22),Stanford, California (1984). (b)

S. McAdams & A.S. Bregman, Hearing musical streams, Computer Music Journal, 3(4), 26-43 (1979).

S. McAdams & K. Saariaho, Qualities and functions of musical timbre, Proceedings of the1985 International Computer Music Conference, Vancouver, British Columbia, Canada,pp. 367-374 (Computer Music Association, Berkeley, 1985).

L. Meyer, Emotion and Meaning in Music, (University of Chicago Press, Chicago, 1956).G.A. Miller, The magical number seven, plus or minus two: Some limits on our capacity

for processing information, Psychological Review, 63, 81-97 (1956).C. Monahan, Parallels between pitch and time: The determinants of musical space, PhD.

dissertation, University of California, Los Angeles (1984).B.C.J. Moore, An Introduction to the Psychology of Hearing, 2nd ed. (Academic Press,

London, 1982).B.C.J. Moore, B.R. Glasberg & R.W. Peters, Thresholds for hearing mistuned partials as

separate tones in harmonic complexes, Journal of the Acoustical Society of America, 80,479-483 (1986).

J.L. Mursell, The Psychology of Music (W.W. Norton, New York, 1937).U. Neisser, Cognition and Reality (W.H. Freeman, New York, 1976).L.P.A.S. van Noorden, Minimum differences of level and frequency for perceptual fission

of tone sequences ABAB, Journal of the Acoustical Society of America, 61, 1041-1045(1977).

C.F.A. Pantin, The Relations between the Sciences (Cambridge University Press, London,1968).

H. Partch, The Genesis of a Music, 2nd ed. (DaCapo Press, New York, 1974).S. Pinker, Visual cognition: An introduction, in: S. Pinker (ed.), Visual Cognition, pp. 1-64

(MIT Press, Cambridge, Mass., 1984).R. Plomp, Aspects of Tone Sensation: A Psychophysical Study (Academic Press, London, 1976).DJ. Povel, Internal representation of simple temporal patterns, Journal of Experimental

Psychology/Human Perception and Performance, 7, 3-18 (1981).DJ. Povel, A theoretical framework for rhythm perception, Psychological Research, 45, 315-

337 (1984).D.J. Povel & P. Essens, The perception of temporal patterns, Music Perception, 2, 411-440

(1985).Z. Pylyshyn, Computation and Cognition (MIT Press, Cambridge, Mass., 1985).S.K. Reed, Structural descriptions and the limitations of visual images, Memory and

Cognition, 2, 329-336 (1974).S.K. Reed & J.A. Johnsen, Detection of parts in patterns and images, Memory and

Cognition, 3, 569-575 (1975).F. Restle, Theories of serial pattern learning: Structural trees, Psychological Review, 77, 481-

495 (1970).F. Restle, Serial patterns: The role of phrasing, Journal of Experimental Psychology, 92, 385-

390 (1972).J.C. Risset & D.L. Wessel, Exploration of timbre by analysis and synthesis, in: D. Deutsch

(ed.), The Psychology of Music, pp. 26-58 (Academic Press, New York, 1982).C. Rosen, The Classical Style: Haydn, Mozart, Beethoven (W.W. Norton, New York, 1971).M.T.M. Scheffers, Sifting vowels: Auditory pitch analysis and sound segregation, Doctoral

dissertation, University of Groningen, The Netherlands (1983).M. Schoen, The Psychology of Music (Ronald Press, New York, 1940).C. Seashore, Psychology of Music (McGraw-Hill, New York, 1938).R.N. Shepard, Circulatory in judgments of relative pitch, Journal of the Acoustical Society of

America, 36, 2346-2353 (1964).R.N. Shepard, Structural approximations of musical pitch, in: D. Deutsch (ed.), The

Psychology of Music, pp. 343-390 (1982).



R.N. Shepard, Ecological constraints on internal representation: Resonant kinematics ofperceiving, imagining, thinking and dreaming, Psychological Review, 91, 417-447(1984).

R.N. Shepard & L.A. Cooper, Mental Images and their Transformations (MIT Press,Cambridge, Mass., 1982).

R.N. Shepard & D. Jordan, Auditory illusions demonstrating that pitches are assimilatedto an internalized musical scale, paper presented at the Psychonomic Society meeting,Minneapolis, Minnesota (November, 1982).

J. Sloboda, The Musical Mind: The Cognitive Psychology of Music (Clarendon Press, Oxford,1985).

J. Sloboda, Cognition and real music: The psychology of music comes of age, PsychologicaBelgica, 26, 199-219 (1986).

J.F. Sowa, Conceptual Structures: Information Processing in Mind and Machine (Addison-Wesley, Reading, Mass., 1984).

N. Spender, Psychology of music (I-III), in: S. Sadie (ed.), The New Grove's Dictionary ofMusic and Musicians, vol. 15, pp. 388-427 (1980).

D. Sperber & D. Wilson, Relevance: Communication and Cognition (Basil Blackwell, Oxford,1986).

T.S. Stoffer, Representation of phrase structure in the perception of music, MusicPerception, 3, 191-220 (1985).

C. Stumpf, Tonpsychologie (Herzel, Leipzig, 1883).J. Sundberg, A. Askenfelt & L. Frydén, Musical performance: A synthesis-by-rule

approach, Computer Music Journal, 7(1), 37-43 (1983).J.P. Swain, The need for limits in hierarchical theories of music, Music Perception, 4, 121-

148 (1986).E.B. Titchener, A Text-book of Psychology (Macmillan, New York, 1909).E. Tulving, Episodic and semantic memory, in: E. Tulving & W. Donaldson (eds.),

Organization of Memory, pp. 381-403 (Academic Press, New York, 1972).B. Vercoe, The synthetic performer in the context of live performance, Proceedings of the

1984 International Computer Music Conference, Paris, France, pp. 199-200 (ComputerMusic Association, Berkeley, 1985).

D. Wessel, Timbre space as a musical control structure, Computer Music Journal, 3(2), 45-52(1979).

W. Wundt, Grundriss der Psychologie (Engelmann, Leipzig, 1896).


music: a science of the mind? - mcgill university...music: a science of the mind? 3 the...

Documents