congruence-association model and experiments in film music - toward interdisciplinary collaboration...

Congruence-Association Model and Experiments in FilmMusic: Toward Interdisciplinary Collaboration

Annabel J. Cohen

Music and the Moving Image, Volume 8, Number 2, Summer 2015,pp. 5-24 (Article)

Published by University of Illinois Press

For additional information about this article

Access provided by University of California @ Berkeley (20 Oct 2015 03:43 GMT)

http://muse.jhu.edu/journals/mmi/summary/v008/8.2.cohen.html

http://muse.jhu.edu/journals/mmi/summary/v008/8.2.cohen.html

5music and the moving image 8.2 / summer 2015©2015 by the board of trustees of the university of illinois

Scholarship in film and film music typically entails insights of researchers who often have dedicated their lives to the pursuit of knowledge in these fields. The same is true of scholarship in aesthetic and literary theory. The hard-earned opinions matter, and through them understanding grows. And yet, bias can arise in ideas based on subjec-tive experience.The problems of subjective bias and fallibility of the senses motivated the development of the scientific method in the seventeenth cen-tury and led to progress in the physical sci-ences in the eighteenth century. The earliest days of experimental psychology in the late nineteenth and early twentieth century took the experimental method as its creed, even though introspection seemed the best way to know about the mind. From the late nine-teenth to early twentieth century, students working in the first experimental psychology laboratory of Wilhelm Wundt in Leipzig attempted to gather introspective evidence under controlled conditions in order to gain

insights into the mind. Around the same time in Russia, however, physiologist Ivan Pavlov was conducting studies of digestion that had important implications for under-standing basic mental operations underlying learning. Observation of dogs whose food was paired with sounds or other stimuli led to the laws of classical conditioning. In classical conditioning,1 an arbitrarily chosen perceptible object (e.g., a tone) is paired with another, such as a puff of air, that routinely elicits a reflexive response (an eye-blink in this example). After a sufficient number of pairings, the originally neutral object, in this case, the tone (called the con-ditioned stimulus) comes to mean much the same as the reflex-inducing unconditioned stimulus (the air puff) and will elicit the reflex in the absence of the unconditioned stimulus. In the example, a tone can cause an eye to blink if it has been paired with a puff of air that causes a blink reflex. In the original Pavlovian study, dogs salivated when they heard a tone that had originally been

Congruence-Association Model and Experiments in Film Music: Toward Interdisciplinary Collaboration

Annabel J. Cohen

Abstract. The article draws attention to the value of objective measurement of both human responses and soundtracks for validating conjectures about the role of film music. Based on empirical research on film music as well as ideas from film theory and cognitive science, the author’s Congruence-Association Model with Working Narrative (CAM-WN) is introduced as a means for organizing ideas about the role of music in film. The application of CAM-WN to the problem of unattended film music is reviewed, and the notion of subjective silence is exami-ned in the context of spectrographic representation of film music. While empirical research with human participants is also encouraged, the complexity of the enter-prise is highlighted by a list of questions to consider in advance. Interdisciplinary collaboration is suggested as a means for overcoming the complexity of film-music behavioral research, though it is not a panacea. The Lund University Film Studies Department offers an important example in this direction.

MMI 8_2 text.indd 5 5/28/15 10:38 AM

6 music and the moving image 8.2 / summer 2015

presented with their food. This discovery has many applications, including the under-standing of how film music works. The success of the objective experimental approach of Pavlov in measuring behavior (which could also be referred to as a behav-ioral approach) contrasted with the absence of tangible results arising from introspective studies in Western Europe. Impressed by Pavlov, psychologist John Watson at Johns Hopkins University in America espoused the importance of the behaviorist approach as the basis for psychological study and understanding of human behavior.2 His manifesto entitled “Psychology as the Behav-iorist Views It” proclaimed the importance of measuring only observables that could be quantified (like numbers of drops of saliva produced when a dog heard a bell that had been paired with food, the number of times a rat pressed a bar, or that a human blinked an eye). The ensuing Behaviorist era, which dominated psychological research well into the 1970’s, provided a poor context for stud-ies of mental imagery, consciousness, and thought. The experience of film fell into this chilly climate. Decades of behaviorism did much to advance learning theory and develop methodological and statistical rigor but failed to account for human abilities such as reading, speaking, pattern recognition, perception, and aesthetic appreciation. In the early 1970’s, behaviorism began to give way to the new paradigm of cognitive sci-ence, which opened the doors again to the study of the mind. The cognitive era adopted the rigor of behaviorism and added a host of experimental paradigms assisted now by computer technology.3 The wealth of data collected in studies of visual, and, more re-cently auditory, perception encouraged the development of models of human memory, perception, and cognition, and also the study of topics such as meaning and aesthetics. Thus, in the present day when film theorists conjecture that a film director’s choice influ-

ences an audience in a certain way, in the context of cognitive science, it should be possible to conduct an experiment to test the conjecture. Although the possibility exists for testing theoretical ideas, in reality this has yet to be commonplace. The present ar-ticle is directed at encouraging the empirical study of film-music perception within the broader domain of empirical film-theoretic studies. The current cognitive era welcomes em-pirical research in film, and film music pro-vides a particularly fertile ground for testing speculations and for developing theories.4 Part of the reason for this is that unlike other media contributing to a film, music is typi-cally outside the diegesis. For the most part, music is there for the benefit of the audience rather than for the protagonists. A complete theory of film needs an account of diegetic and nondiegetic worlds. Film music opens a window to research in both worlds as well as to their shared territory. Moreover, the topic of film music lends itself to psychological inquiry in the present day because technol-ogy makes it increasingly feasible to alter the music background of a film. The abil-ity to alter a film score makes it possible to compare the effects of different music on the perception and interpretation of a particular film clip that remains unchanged in all other respects. It is often easier to edit the music aspect of a film as compared to the visual as-pect, which might entail either bringing back the actors to redo the scene in a particular way, or modifying the set. Changing sound effects can also be straightforward, while changing the dialogue creates challenges due to the necessary coordination with visual facial articulatory cues. The ease of manipu-lating the music track is nevertheless a recent development. The art of film is only ap-proximately a century old, and the linking of sound to a film reel was a technical discovery that became common only in the late 1930’s. The ability to control the music track inde-

MMI 8_2 text.indd 6 5/28/15 10:38 AM

Cohen : Interdisciplinary Collaboration in Film Music 7

pendently followed sometime after. What was once one of the most challenging aspects of the film to manipulate has become one of the easiest. The following article reviews some aspects of the progress in the empirical study of the role of music in perception of the moving image and offers suggestions for the accelera-tion of future progress. The article has three parts. The first considers film music from the perspective of empirical methodology, his-torical context, and actual studies. The sec-ond part focuses on a theoretical approach to understanding film music called the Con-gruence-Associationist Model that I have been developing over the last twenty-five years, inspired by the results of experiments.5 I will focus on the two concepts in the title of the model, Congruence and Association, and will also develop the concept of the Work-ing Narrative, the conscious representation of the film, or the moment-to-moment film experience. The final part of the article fo-cuses on the future—what experiments and human motivation are needed to advance an empirically based psychological theory of film and film music.

Part 1. Experiments on Film Music

Experimental Method

Film theory is characterized by the insights of scholars of film. Sometimes their insights conflict, and sometimes insights of an indi-vidual scholar differ from one time to the next. Insights may concern how a director’s choice influences the audience’s understand-ing and interpretation of the film. The film theorist makes assumptions based on his or her experience of the film. This experience of a film is informed by both a professional lifetime of film viewing and natural sensitivi-ties to aesthetic objects and narrative that probably led the theorist to this field in the first place. As well, the theorist will be versed in the ideas of other theorists, which may

also have an influence on the experience of the film. The perspective of the present article is that many of the insights of film theorists can inspire hypotheses that can be tested em-pirically in psychological experiments. Even what appear to be simple assumptions, such as “music has meaning” or “music influences the meaning of a film,” can benefit from scientific experimentation. What does it mean “to influence interpretation of a film”? Experimental psychologists would say that it is not self-evident. Experimental psycholo-gists would create “operational definitions” that describe precisely how a construct can be measured within the context of an ex-periment. For example, one could measure the meaning of a word or concept by asking people to draw a representation of the word or concept. With “the drawn representation” as the operational definition of meaning, the role of context on the meaning of a concept can be determined. For example, consider the concept of the “X-like” symbol in the center of figure 1 below. Imagine that it is shown to two groups of people, after which one group is given a list of words that has the word hourglass in it, while the other group has a list of words that includes the word table. After a while, members of each of the two groups receive a blank sheet of paper and are asked to draw the symbols they have been previously presented (only one of which has

Figure 1. In the center is an ambiguous symbol, and on the left and right, concepts that might influence the meaning of the symbol. (From Carmichael, Hogan, and Walter, “An Experi-mental Study of the Effect of Language on the Reproduction of Visually Perceived Form,” Journal of Experimental Psychology 15 [1932]: 75, used with permission of the American Psycho-logical Association.)

MMI 8_2 text.indd 7 5/28/15 10:38 AM


been shown above). Although both groups saw the identical symbol, the members of the two groups will likely differ in the tenden-cies of their drawings. Those whose thoughts were directed by the concept of hour glass might draw representations that more closely resembled an hour glass than a table, and vice versa. This was the observation of Car-michael, Hogan, and Walter in their classic study using two lists of twelve words and one corresponding list of twelve symbols.6 The point is that with effort and ingenu-ity, it is possible to invent ways of measur-ing constructs of interest. While it might seem a waste of time to discover measures of meaning, such as rating scales, because they clearly do not capture the whole picture, the collection of data using these measures can lead to reliable results that must reflect how the mind works. Experimental psy-chologists believe that to support or refute a theory, data from human participants are needed, collected under controlled experi-mental conditions. And of course, the data by themselves become useful when submit-ted to analytical methods recognized by the discipline. Some psychologists believe there is no other way to advance knowl-edge, and some scholars in the humanities may also embrace the principles. Consider for example, musicologist and semiotician Philip Tagg. In response to challenges of his semiotic proposals in such work as his more

than four-hundred-page analysis of the first fifty seconds of the Kojak theme, specifi-cally “its lack of empiricism at the level of music perception,”7 Philip Tagg embarked on a massive empirical exploration of as-sociations elicited by short excerpts from ten movie soundtrack opening themes. For over a fifteen-year period, he asked more than six hundred of his students to write down the first thoughts that came to mind on hearing some or all of the soundtrack examples. The study is carefully documented in the more than eight-hundred-page book entitled Ten Little Tunes,8 and it sheds some light not only on how film music works (and popular music, and music in general) but also on the challenges of multidisciplinary work, a topic to which we return at the end of this article.

Early Studies of Film Music

An early, and possibly the first, psychological study of film music was conducted by Percy H. Tannenbaum in 1956.9 Tannenbaum was one of the three developers of the famous “semantic differential” technique for measur-ing meaning along three affective dimensions, Activity (e.g., fast-slow), Potency (e.g., strong-weak), and Evaluation (e.g., good-bad).10 The technique is still used today, although much research on psychological meaning suggests that two dimensions—a valence measure (good-bad) and an arousal measure (active-calm) are sufficient.11 In this study, Tannen-baum was interested in answering a question that remains current today, whether a musical score significantly influences the judgement of a dramatic piece for the stage, for televi-sion, or as a single camera version of the stage drama. Tannenbaum observed that music increased the judgments on the semantic dif-ferential dimensions of activity and potency but not on the remaining dimension of evalu-ation. The study also showed the possibility of conducting research in such a complex situation as dramatic entertainment. The next pioneering study of the effect of music on the

Figure 2. On the left a drawing from memory of the central symbol after exposure to the con-cept of hour glass, and on the right a drawing from memory of the central symbol after ex-posure to the concept of table. (Based on Car-michael, Hogan, and Walter, “An Experimental Study of the Effect of Language on the Repro-duction of Visually Perceived Form,” Journal of Experimental Psychology 15 [1932]: 80, used with permission of the American Psychological Association.)

MMI 8_2 text.indd 8 5/28/15 10:38 AM


experience of film waited over twenty-five years for psychologists Julian Thayer and Rob-ert Levinson.12 Their purpose was to deter-mine whether different music might increase or decrease the stressfulness of an industrial safety film, often used in psychophysiological studies to induce stress. There were twenty students in each of the three conditions of presentation of a music soundtrack. One con-dition included music that might be found in a documentary. Another condition included music that might accompany a horror film. The final condition had no music. The study revealed evidence of the effect of the docu-mentary and horror music on respectively reducing and increasing stress as measured by skin conductance. The skin conductance measure in the condition without music fell between the other two extremes. At about the same time that this study was published, a student and I tested the hypothesis that music influences the inter-pretation of a simple animation of geometric figures.13 As Thayer and Levenson had used a film that was known to produce stressful reactions,14 the animation film we used was known to give rise to a particular interpreta-tion. Specifically, an aggressive bully (large triangle) was persecuting an innocent couple (small triangle and small circle) as the latter moved in and about a building (a rectangle) that the angry triangle ultimately destroys in a fit of madness. Like Thayer and Levenson’s 1983 experiment, we had three soundtrack conditions, two different music conditions (called Strong and Weak, based on the rating scale that most distinguished them), and a no-music condition. Instead of physiologi-cal measures, we asked participants to rate each of the three film characters and the film overall on twelve semantic differential judge-ment scales, which would ultimately provide meanings of the characters on the three di-mensions of Activity, Evaluation, and Potency (there were four different rating scales for each of the three dimensions). We also had

other groups of participants rate the music on its own, to determine its meaning. What was surprising about the results was that the different music changed the ratings (mean-ings) of the characters in different ways. For example, while under the Strong music condi-tion, as compared to the Weak or no-music conditions, the small triangle became more active while the large triangle and small circle became less active. We had expected instead that responses to all three characters would change in the same direction, if they changed at all. This is the value of empirical studies be-cause the data show what actually is the case. Statistical analysis can immediately determine the extent to which the effects observed could have arisen by chance. If the probability of a chance occurrence is less than 5 percent, by convention, the results are regarded as statisti-cally significant. In other words, the observed effects are reproducible. It is nevertheless excellent scientific practice to repeat the study under the exact same circumstances but with different participants. The results of such a study would add further evidence for the reliability of the previous observation. In our case, the same finding was observed a second time when a key part of the entire study was repeated. The replication gave us confidence in saying that the music appeared to cause one character to be interpreted in one way and another character to be interpreted in another (possibly opposite) way, and more generally, the music did not simply add information to the entire mix of information provided by the film.

Part 2. Congruence Association Model (CAM) and the Working Narrative (WN)

The puzzling finding that the same music had different influences on different char-acters in the film, observed in our original study and its replication,15 led to the pos-tulate that the different visual movement patterns of each character in relation to the

MMI 8_2 text.indd 9 5/28/15 10:38 AM


audio movement (i.e., temporal, rhythmic, accent) pattern of the music created different levels of attentional emphasis. It was hypoth-esized that the film character whose actions were most congruent with the musical pat-tern would be most attended and, conse-quently, the primary recipient of associations of the music. This then became the first ver-sion of the Congruence-Association Model. It is useful at this point to describe some experiments that have separately considered the roles of Congruence and Association.16

1. Congruence

Studies of congruence focus on the structural similarities (and differences) across auditory and visual modalities. This approach recog-nizes the structural or relational elements of every stimulus and the potential structural parallels across different media. As a concrete example, in everyday activity, correlated audiovisual patterning occurs, for example, when in speaking or singing, the shape of the mouth, the slant of the eyebrows, or the vis-ible tension in facial muscles parallels aspects of the produced vocalizations. The vowel “o” will be characterized by a particular shape of the mouth. A vocal expression of shock may be regularly accompanied by arched eye-brows. Examples of research conducted with this focus on music-visual structural paral-lels are reviewed by Kendall and Lipscomb,17 Lipscomb,18 Eitan,19 and Iwamiya.20 Their studies examined the extent to which listener-viewers are sensitive to the degree of cross-modal structural similarity, for example with respect to temporal synchrony. This attribute of film music has sometimes been referred to pejoratively as Mickey-Mousing, after its common use in cartoon animations, but the human response to cross-modal structure is by no means a simple process. Until recently, the focus of perceptual studies in psychology was on a single perceptual modality, in other words primarily on visual perception or audi-tory perception. In fact, the field of perceptual

studies has always been dominated by visual studies. Just several decades ago, the study of perception typically implied the study of visual perception. While auditory perception has come into its own over the last twenty years, including studies of speech and music, the study of multisensory perceptual integra-tion and multisensory perceptual development is much newer.21 Comprehensive volumes that overview the field may fail to even mention the domain of film and media and the chal-lenges of understanding the role that music plays therein.22

2. Association

Studies of association in music and film generally focus on how the music establishes a context for interpretation of the film. One theoretical issue is to determine how the music takes on meaning in the first place. This problem falls within the field dedicated to music psychology or music perception and cognition. Association is one of the key ways, if not the key way, in which music be-comes meaningful. It is obvious that music that is played at weddings brings to mind the concept of weddings when heard in some other context. Music associated with funer-als, birthday celebrations, national events, and religious ceremonies takes on the mean-ing of those events, just as the tones heard by Pavlov’s dogs took on the meaning of dinner time.23 The same account applies to the leitmotiv phenomenon of opera in which a musical theme is paired with an event or character in the drama, and eventually, independently, in the absence of the event or character, this music is believed to bring to mind thoughts of the event or character originally associated with it. The technique borrowed from opera is often implemented in film, the classic example being the theme from “Jaws” depicting the dangerous shark. A nice example of the principle of as-sociation appears in the film Warhorse.24 Early in the story, the young man, Albert,

MMI 8_2 text.indd 10 5/28/15 10:38 AM


teaches Joey, a colt with a mind of his own, to come whenever Albert whistles the “owl call” through his cupped hands (see figure 3). The colt Joey thus has many opportuni-ties to associate the owl call with coming to Albert. Joey and Albert are separated during the war years, each on separate battle fronts. In the final scenes of the film, they both emerge alive, wounded, and in close proxim-ity. Albert, temporarily blind, learns that a remarkable horse has survived against great odds. The horse is about to be shot given its poor condition. Albert, who is in audible range of Joey, performs the “owl call,” which elicits immediate movement of Joey in the direction toward the sound. The conditioned association has been retained. To Joey, the owl call means “run to Albert.” The scene is backed by musical themes heard earlier in the film, adding to the emotional impact for the audience, again due to associations with the film and with other music of this type.25

However, it is one thing to describe a phe-nomenon and provide a theoretical explana-tion for it, and another to support it with evi-dence. Much empirical research in film music has been directed to showing how music can influence the interpretation of film, and to do this, it is usually best practice to determine what the music and the film excerpts in ques-tion mean (what associations do they bring to mind) independently before exploring the effect of the music on the film, or vice versa. I say the influence of one on the other, although usually we think of music in a subordinate or modifying role, serving as an adjective, adverb, or subordinate clause providing ex-planatory context for the main action visually depicted, or depicted with other media such as speech, sound effects, or text on the screen. The early studies previously mentioned of Tannenbaum,26 Thayer and Levenson,27 and Marshall and Cohen28 are studies of the role of music on the meaning of film.

3. Further Studies and Extending the Model

Further studies conducted with my research team led to extensions of the Congruence-Association Model. Again, often an unex-pected finding prompted a revision or exten-sion of the original conceptualization. While a number of studies showed influences of the meaning of music on the interpretation of film, one study of effects of romantic and cri-sis music on two contrasting film clips, each involving two human actors, showed a direct influence of the music on the interpretation of one of the clips but not the other (initially reported by Cohen29). The first account for this was that there was something wrong with the experiment. Why didn’t romantic music reduce the apparent aggressiveness of a fight scene? It then became clear that the meaning of the visual scene was so obvious that the music could play no role in altering the inter-pretation. The music had influence only when the scene was ambiguous. In this case, the

Figure 3. Top panel: Early in the film, warhorse Joey learns to come to Albert (not seen, in the tree) when Albert makes the “owl call.” Bottom panel: Joey looks at Albert in the tree (Both panels from Warhorse, produced by S. Spiel-berg and K. Kennedy, directed by S. Spielberg [Universal City, CA: Touchstone Pictures and Dreamworks Studio, 2011]).

MMI 8_2 text.indd 11 5/28/15 10:38 AM


ambiguous scene could be interpreted as ei-ther a quarrel or amorous play. Music associ-ated with crisis or romance pulled the story in one direction or the other in this ambiguous case. While these studies focused on the influ-ence of music on the interpretation of a film, Boltz, Schulkind, and Kantra compared the effects of background music on memory for filmed events.30 The background music in the study either foreshadowed or accompanied a visual event drawn from television programs such as Alfred Hitchcock Presents. More spe-cifically, the stimulus material consisted of visual and musical material from sixteen 3–4 minute television clips. Each clip established a slightly suspenseful narrative that had a posi-tive or negative outcome revealed at the end of the clip. Prior to the actual study, the mood of the outcome—positive or negative—was verified in a short pilot study with six par-ticipants who rated the visual clips on several dimensions of meaning. Also in pilot work, excerpts of music, 20–30 seconds in duration, taken from the soundtrack of the television clips, were also independently rated for posi-tive and negative mood. Eight visual clips and eight music excerpts reliably reflected a positive mood, and eight reflected a negative mood. In the actual study, the music excerpts were paired with a video presentation in two conditions. Music either accompanied the ending of the clip or it was presented prior to the ending so as to foreshadow the ending, and it either agreed or disagreed in mood with the positive or negative ending. After being presented with the sixteen visual clips that had been combined with music that was either mood consistent or inconsistent, and that either preceded or accompanied the ending, sixty student participants carried out three types of tasks; they were asked to recall as many of the excerpts as they could, they were given a test of recognition of the music that had been presented, and they were tested for their memory of the visual information

presented. Recall of an excerpt was highest when the moods of the music and film were consistent and the music accompanied the end of the clip, or when the moods were in-consistent and music preceded (foreshadowed erroneously) the ending. Thus, accompanying music consistent with the mood of the ending served as a cue for memory of the clip, but for foreshadowing music, inconsistency with the ending, served as a cue for the ending. In other words, the music established expectan-cies or associations that were either confirmed or violated, and depending on the timing, the music became more or less connected with the visual information of the narrative. Boltz,31 continuing to use these same ma-terials in other memory tasks (e.g., recall of the music), further affirmed the formation of connections between music and visual im-ages, particularly when the meanings were consistent, and she proposed that the results both supported and extended the Congru-ence Association Model. To add to the pic-ture of when music could impact meaning of a visual event, Tan, Spackman, and Bezdek showed that music presented after a neutral scene could influence the interpretation of the scene, though not as strongly as when the music preceded the scene.32 Finally, Boltz described research that showed that mean-ingful videos could alter the interpretation of neutral music, indicating that in some cases (for example that of music videos) the visual domain can play a supportive role depicting the meaning of music, although typically the visual dimension dominates.33

These studies remind us that the goal of the audience is to make sense of a narrative presented audiovisually by the director. The CAM Model was further broadened to take this perspective into account.34 The audience member has only two sources of information available—the real-world audiovisual infor-mation provided by the director, and the per-sonal, subjective long term memory of past experience in the world, including the world

MMI 8_2 text.indd 12 5/28/15 10:38 AM


of film. Inspired by the film theorist Christian Metz, who argued for five tracks or chan-nels of film information (one of which was music),35 the later model of CAM represented these five channels of audiovisual informa-tion from music, sound effects, speech, visual scenes, and text. See Level A in figure 4, which shows a current version of the Congruence-Association Model with Working Narrative (CAM-WN). The first stage of each channel is referred to as a surface to reflect the physical energies impinging on the sensory processes of the nervous system at this early stage. More recently, a sixth channel (at A, far right) ac-commodates motor or kinesthetic informa-tion, in recognition of recent discussions of the mirror neuron system and of musical em-bodiment which are purported to engage the same sensory-motor representations in the

brain, regardless of whether an action is either carried out or merely observed.36 As film entails observation of action, the stimulus for activity of the mirror system would constantly prevail. The channel also accommodates real kinesthetic information such as from the the-atre seats (which in the twenty-first century, are sometimes capable of movement related to the film). At B, represented by two triangles that make up the diamond shape, two analyses take place for all six surface types. One is a structural analysis, and the other is a semantic (association) analysis. Thus, information from each surface is decoded into structural and semantic elements, parsimoniously providing only two kinds of information that the brain must deal with. At B, represented by the dashed ascending arrows, a rapid preprocessing of the

Figure 4. The Congruence-Association Model with Working Narrative accounting for the role of music in film interpretation. (From A. J. Cohen, “Film Music and the Unfolding Narrative,” in Lan-guage, Music, and the Brain: A Mysterious Relationship, ed. M. Arbib, [Cambridge, MA: MIT Press, 2013], 19, used with publisher permission.)

MMI 8_2 text.indd 13 5/28/15 10:38 AM


information leaks through to long-term memory (at E) and primes expectations about the narrative based on experience. These ex-pectations are represented by the descending arrows at D. A second slower ascending pro-cess at B provides a more detailed analysis of the sensory information. It is also at B where effects of cross-modal structural congruence take place, as depicted by an arrow moving from music structure to visual structure, al-though many other cross-modal structural congruences are possible. The dark oval within the visual scene channel indicates the visual focus of attention guided by music/visual congruence discussed earlier. Subse-quently the best match between the top-down expectations (at D) and this slower detailed bottom-up analysis of the sensory informa-tion (from B) is the audience member’s dy-namic experience of the film, which is termed the working narrative (at C).

4. A Theoretical Application of the CAM-WN Model

I recently employed the CAM-WN model to account theoretically for the effectiveness of aspects of the music soundtrack in the film Babel.37 Gustavo Santaolala composed the score for the film, often employing a solo oud, a Middle Eastern stringed instrument. My focus was the use of the music in sec-tions of the film that lacked all other sound, that is, when dialogue and sound effects were absent. In reality, dialogue and sound effects rather than music would be present, so the situation of the film was exactly the reverse of reality. An audience member, however, finds nothing unusual about this, and the technique of having music in a sense replac-ing speech and sound effects is used in many films, though typically sparingly. Kulezic-Wilson has previously written in this journal, Music and the Moving Image, about the music of Babel in an article that focused on the strategic use of silence within the music in the film.38 Indeed, for her also,

the film Babel provided an ideal case study, in that director Iñárritu (a composer him-self) manipulates the music soundtrack in unusual ways throughout the film. Particu-larly striking examples arise in a Japanese night club where the audience takes the point of view of Chieko, a deaf teenage fe-male protagonist. At times during this scene the music cuts out entirely and brings home the idea that Chieko cannot hear (although, whether her world is truly one of acoustic silence we do not know, but metaphorically, her hearing loss isolates her from the sound of voices and music). Kulezic-Wilson argues that earlier in the film, silence within the musical context is used to draw attention to the lack of communication between the two main adult protagonists, Susan and Richard, who are on a bus as tourists in Morocco. Kulezic-Wilson notes the silencing of the music track on the bus and claims this is to help depict the failed communication be-tween the husband and wife. She says:

The melancholy theme played on the oud, which starts at the beginning of the scene accompanying images of the desert and occasional passersby, is actually removed at the instant when the wife reaches for her husband’s hand, leaving only a barely audible accompaniment made of electronically sustained sounds. Iñárritu deliberately leaves that moment “dry,” al-lowing the audience to read the emotional weight of silence and the tiniest gestures themselves rather than trying to interpret them through music. On the other hand, the withdrawal of the musical theme at the crucial moment should not be understood only as a simple refusal to comply with the conventions of “emotional” scoring but also as an acknowledgment of the insur-mountable gap between the husband and wife at that point in their relationship.39

One might however interpret the scene differently than did Kulezic-Wilson. For example, Susan, feeling separated from her

MMI 8_2 text.indd 14 5/28/15 10:38 AM


husband, looks out the window of the bus and sees seemingly hapless passersby on foot in the desert and thinks to herself, “It is really not so bad after all to be a western woman on a tourist bus with a currently caring but generally non-communicative husband.” This alternative rather flippant interpretation belies the fact that the couple is getting over the loss of their child; but whatever the interpretation, a consideration of the acoustics of this part of the soundtrack provides fair evidence of sound, and it does so also of the “silence” in the Japanese nightclub. To be fair, Kulezic-Wilson does discuss the difference between the two types of silence: (absolute) silence as something that is out-side human hearing capacity, and cinematic silence, which is perceived as such depending on the context and is not only essentially dif-ferent from absolute silence but is often “as layered and as carefully designed as any film soundscape.”40 That Kulezic-Wilson describes the former section, however, as “barely audi-ble” highlights the value for empirical studies of issues such as these surrounding conscious-ness of the music in film. The discussion of audibility brought to my mind the “unheard melodies” of Claudia Gorbman. This is usually associated with a particular type of aesthetics in which music is not supposed to draw attention to itself. The music in Babel is not the standard Hollywood orchestral sound; however, the issues of conscious registration of the music may be the same. CAM-WN offers an account for the fact that music is often unattended. If one did a brain scan, there would be something registered, but that registration might differ depending on whether the music was re-garded as diegetic or nondiegetic (such brain imaging studies have yet to be conducted). The failure to be cognizant of the acousti-cal aspect of the music is an outcome of the absence of a top-down expectation (at D) to match the bottom-up acoustical information (from B). Only best-matches in the model

lead to conscious attention. There are no cues in the film scene on the desert to sug-gest the performance of music, and thus it is not surprising that the audience fails to hear something that is barely audible, in this case the electronically enhanced oud harmon-ics. Yet, the emotional information from B conveyed by the music would be matched by the expectations at D, and this would be part of CAM’s Working Narrative—the conscious experience of the film at C. Figure 5 shows the sound spectrogram of the scene on the bus. The spectrogram visu-ally depicts the sound frequencies present as a function of time. It is laid out like music notation, with time increasing horizontally along the bottom, and frequency running vertically from low to high pitch. The repre-sentation of the frequencies is rather rough, as the scale is so condensed. The examples show the frequency range from 0 to 15,000 cycles per second or Hz (for Hertz). As a reference, consider that the highest note of the piano is around 4,200 Hz. Young adults can hear sounds to about 22,000 Hz; with age and noise exposure, this range declines. The spectrogram for a pure (single frequency) tone is one thin horizontal line. Most sounds are complex and so are represented by many lines, and in the case of noises, solid bands (clusters of single lines banded together). The music notation of the oud is shown below the spectrogram of the music leading up to the moment of “silence” when Susan moves her hand toward Richard. Each note of the oud contains many overtones (single frequencies at whole integer ratios of the lowest frequency called the fundamental frequency). Because of the physical characteristics of the oud, some of the overtones naturally die quickly while others are sustained. The particular pattern of overtones characterizes the quality of the oud timbre, while the changing fundamental fre-quency produces the melody, which is shown in notation. When the melody of the oud ends on a long sustained note, many overtones are

MMI 8_2 text.indd 15 5/28/15 10:38 AM


sustained as shown by the continuous parallel lines. Some of the sustaining is due to deliber-ate enhancement of these tones electronically by means of music/sound-editing equipment (a decision of the composer, director, and/or sound engineer). Silence would be reflected by an absence of light. Because of the presence of these bands of light, the spectrogram does not reflect physical silence, although to the listener the music may go unnoticed as does much music of film. Figure 6 provides a spectrograph illus-trating, in the upper panel, one minute and fifteen seconds from the film, beginning from the second sequence of four notes from figure 5 and extending to the time of the gunshot. The lower panel, which is on the same time scale, shows a passage from the Japanese nightclub with deliberate periods of intermittent silence, or so it seems. The discontinuity of sound and silence in the Japanese nightclub is clearly shown in the bottom panel. The noise of this poly-phonic loud dance music is reflected by the

dense spectrogram, dense until the cutting out of all higher frequencies. Kulezic-Wilson distinguishes the two kinds of silence;41 yet it is interesting to see the visual depiction of these two situations and to see how both so-called silences entail considerable presence of low frequency in the audible range, in the one case as continuously ringing harmon-ics and in the other as low noise. In no case is the complete absence of sound apparent, although the high frequencies are greatly reduced. Sound spectrographs can be generated from software that is available freely—for example, Praat and Audacity (used for fig-ure 6)42—or commercially, such as Adobe Audition (used for figure 5). Such software provides the researcher of film scores with another means for analyzing and under-standing the score. One might argue that it should be the first tool when beginning any study of film music, although I must admit to having turned to the spectragraph myself only recently, in spite of having worked in

Figure 5. Top and bottom panels represent a continuous film seg-ment from Babel during which, in the top panel, a four-note melody on the oud is repeated twice, and, in the bottom panel, the final tone is sustained over approximately thirty seconds via electronic enhancement of over-tones. This spectrograph reveals the presence of acoustical in-formation throughout this pas-sage. Visual clips represent the simultaneous images. The end of the top panel and beginning of the bottom panel contain iden-tical information momentarily. (Frequency detail is shown in Figure 6.)

MMI 8_2 text.indd 16 5/28/15 10:38 AM


this area for many years. Blumstein, Davi-tian, and Kaye used spectrographs to deter-mine sound pattern similarities within four categories of film (war, horror, action, and drama).43 A host of experiments are waiting to be conducted with the rich clips from this film, Babel, as but one example, in order to address issues of the diegesis, conscious at-tention, and influence of the music on the in-terpretation of the film. So what is stopping us?

Part 3: Toward Collaborative Empirical Research on Film Music

The field of music cognition and, more gen-erally, music psychology has made enormous

strides in the last three and a half decades.44 Studies of the psychology of film music and the role of music in multimedia have been relatively few, in part because of technical challenges associated with controlling com-plex audio and video simultaneously. Tech-nology, however, no longer stands in the way to the same extent as it once did. Those with a laptop, desktop computer, or even a tablet or smart phone have a laboratory at their fingertips. Access to such audiovisual editing tools and storage just a decade ago was an impossibility for most individuals, but ac-cess to storage is no longer the same sort of deterrent for any would-be empiricist of film music. The problem however is that the best

Figure 6. The top panel shows a spectrogram of music of the oud during the fateful bus ride. The horizontal time line extends just over one minute and fifteen seconds. The vertical axis shows fre-quency up to 4000 Hz. Intensity of the sound is represented on a gray scale (lightness/darkness scale, where dark means sound frequency is present), and the continuous shading reveals that sound continues across the entire excerpt. During the first eight seconds, four notes are played. The last is sustained for about thirty seconds with changing emphasis on various harmonics shown as horizon-tal lines below around 500 Hz (also shown in figure 5). The melody continues at forty seconds and is joined by real environmental noise of the bus (the diffuse greyness represents low broadband noise). At fifty-seven seconds, the sound of turning a page of a book is shown, followed by continuing noise of the bus, and, at one minute and twelve seconds, the sound of a rifle shot creates a dark narrow vertical line representing intense broadband noise, all sound frequencies at one brief instant. The bottom panel shows a spectrogram of the sound of the Tokyo nightclub presented at the same time scale as the top panel. The contrast between loud and quiet is evident. While the quiet sections may be perceived as silent, there is clearly sound present in the frequencies below 500 Hz with the grey diffuseness indicative of noise. The sense of silence must arise from the sudden absence of frequen-cies above 500 Hz in both examples. (The specific 500 Hz cutoff is unlikely critical to the effect, i.e., other cutoff frequencies might be equally effective.)

MMI 8_2 text.indd 17 5/28/15 10:38 AM


facilities in the world will be useless without the human resources needed to exploit them. So while the technological barrier no longer impedes progress as it once did, the human and financial unknowns of the empirical re-search equation need a solution. If one does want to begin empirical research and has never done so (or never in this area), what is the starting point? Here are some questions that might be asked.

Preliminaries:What theoretical or pretheoretical question

drives the research?What prediction is to be tested or what kind

of information is sought?What theory or prior evidence grounds the

prediction?What will the expected data look like if the

prediction is supported?Even at this stage one can and should ask:

What kind of statistical analysis will be conducted in order to determine whether the results obtained reflect some “truth” about the world with a high probability? The question needs to be addressed again when experimental design issues are raised in the next stage.

What types of expertise are needed ideally to conduct the study?

Next are the research design issues:What experimental design will best (practi-

cally and theoretically) enable testing the prediction?

What statistical tests will be used (see above section)?

What checks can be put in place so as to avoid missing data, and what will be done if data are still missing (e.g., someone fails to respond to a question, equipment breaks down, etc.)? (Again, ideally this should be addressed before the data collection begins, so that one does not make such decisions after the study is run, where one might be ac-cused of making decisions about miss-ing data on the basis of how the data turned out.)

Will participants be tested only once, or will they be invited to return in order to show the stability of their responses?

What audiovisual materials must be devel-oped?

What permissions are required (if any) to use copyrighted materials?

What kind of responses to them will be ac-quired?

What will be the conditions of testing (such as in a quiet laboratory room, a home theatre setting, a film theatre)?

Who (gender, number, background, educa-tion, experience, age) should be invited to participate in the study?

Where and how will they be invited to par-ticipate?

Where and how will the study be conducted? In a laboratory, theatre, classroom or elsewhere?

What ethical approvals are needed in order to send invitations to participants and to move ahead with the testing?

Who will write the ethics proposal?How long will it take to create the ethics pro-

tocol and have it approved and revised if necessary?

How long will it take to recruit participants?How long will a session of the study take?Who will carry out all of these separate tasks?Who will supervise the assistants (if there are

assistants)?

Next are consideration of the results:Who will enter the collected data for analysis,

and who will check the entry for accu-racy? Similarly, who will conduct pre-liminary analyses (as needed) and what will the check for reliability be?

How will the acquired data be transformed and coded for analysis?

Who will conduct the analyses?Who will draw graphs of the data?How are the data, both raw and coded, to be

stored?Will an attempt be made to share the coded

data and analyses with the research com-munity in a digital data repository?

Who will carry out and/or supervise all of these separate tasks?

MMI 8_2 text.indd 18 5/28/15 10:38 AM


Finally regarding the findings:When the data are analyzed, who will discuss

and interpret them?Who will then be responsible for reporting

the data, such as a working paper, or as a paper to submit to a journal for publica-tion?

What is (are) the target audience(s)?What plan is there for disseminating in ways

besides publication?What questions remain or have arisen out

of the study, and how should the entire cycle be repeated with the new related question?

Mere consideration of this nonexhaus-tive list suggests why choosing to conduct behavioral research on audience response to film music is not to be taken lightly. There are many steps, stages, roles, skills, thought processes, and knowledge bases involved. A practical, potentially enjoyable and reward-ing approach may be to develop collabora-tive interdisciplinary teams, with colleagues either in one’s midst or, anywhere where compatible expertise can be found, relying on the Internet and travel. Bringing people together from different disciplines however poses enormous challenges, the topic of a new scholarly field itself.45 Each discipline has its own methods, language, and priori-ties. Developing a common set of methods, priorities, and meanings takes time, if it can be accomplished at all. The opportunities of-fered by interdisciplinarity are not attractive to all scholars, and for many good reasons. Yet, the enormous expertise required to understand the complexity entailed by just a short film clip can rarely be mastered by a single individual. Researchers with expertise in film, computer science, neuroscience, psy-chology, music, linguistics, acoustics, audio engineering, sound recording, sound design statistics, music composition, musicology, and aesthetics can all provide insight. Teams formed from members of different scholarly disciplines may well be needed in order to

make headway. Philip Tagg raised many of these same issues in his dissertation and book that directed attention to the analysis of just fifty seconds of the Kojak television theme music.46 He noted that the knowledge from many university disciplines could in principle be brought to bear on the issue, but that in practice this was not possible. In contrast, the use of the film music in a film provided an example of successful collabo-ration.47 Tagg reviewed the methodologies from psychological research on the meaning of music and felt that none of these would suit his task. Neuroscience was in its infancy and showed no promise. And so he went his own way, developing a special method of analysis of the music here, and developing his own behavioral research methods later for the Ten Little Tunes study.48 However, time did not stand still, and the domain of perceptual, cognitive, and neuroscientific psychology offers much more than it did then, as does the special area of psycho-musicology. Moreover, the perhaps narrow approach to rigorous statistics of the late twentieth century is expanding to include more qualitative and exploratory analysis. Standards in research are increasing. Today, there is growing demand and capacity for preservation and sharing of data in both sci-ence and humanities, offering opportunities for replication and the application of diverse analyses using techniques from various dis-ciplines. There are reasons to be optimistic about the coming of age for interdisciplinary research in film music. The University of Lund Workshop on Film Music and Experimental Methods of 2013 took an audacious and creative step in embedding an experiment into the two-day program. The aim of the experiment was generally to understand how does film music convey meaning, and specifically to explore the role of film music on eye movement patterns using a state of the art laboratory with approximately twenty eye-movement

MMI 8_2 text.indd 19 5/28/15 10:38 AM


recording workstations (see Wallengren & Strukelj, in this special issue). Organizers of the workshop and others at Lund University had set up the study and served as the exper-imenters, while participants in the workshop were invited to serve as participants. After the experiment, as part of the workshop program, there was an open critical discus-sion about the experience (Cabak Rédei, this special issue). Persons from backgrounds in film, music, psychology, computer science, semiotics, and philosophy were in dialogue, a dialogue that is continuing and expanding through this special issue of Music and the Moving Image—a further critical positive outcome of the collaboration. The various viewpoints of authors from several disci-plines are now available in text, providing the contributors an opportunity to build on the foundation laid at the workshop, to see and resolve contradictions and integrate related ideas. As well, the special issue opens the dialogue to readers around the world. As a pilot study, the Lund Experiment has been successful on two levels, advancing knowledge about investigating the role of film music by employing a new group eye-movement recording methodology, and fur-thering understanding about the challenges of interdisciplinary research in film music.49

AcknowledgementsThe input from two anonymous reviewers and journal editors Ron Sadoff and Gillian Anderson is greatly appreciated particularly with respect to the discussion of the work of Danijela Kulezic-Wilson and Philip Tagg, and issues surrounding interdisciplinary work, and inattention to film music. Appreciation is expressed to Anne-Kristen Wallenden and the organizing committee for the invitation to participate in the Lund University Workshop on Film Music in 2013. The Social Sciences and Humanities Research Council of Canada, and the Canada Foundation for Innova-tion are acknowledged for support of the author’s research on the psychology of film music. Robert Drew, laboratory assistant for the Department of

Psychology, University of Prince Edward Island, kindly assisted with the spectrograms, music tran-scriptions, and graphic compilations.

Notes 1. One of two forms of associate learning, the other being operant conditioning, for which B. F. Skinner was renowned some years later. 2. J. Watson, “Psychology as the Behaviorist Views It,” Psychological Review 20 (1913): 158–77. 3. J. L. Bermúdez, Cognitive Science: An Intro-duction to the Science of the Mind (Cambridge: Cambridge University Press, 2010). 4. A. J. Cohen, “Film Music from the Perspec-tive of Cognitive Science,” in Oxford Handbook of Film Music Studies, ed. D. Neumeyer (New York: Oxford University Press, 2014), 96–130. 5. A. J. Cohen, “Film Music and the Unfolding Narrative,” in Language, Music, and the Brain: A Mysterious Relationship, ed. M. Arbib (Cambridge, MA: MIT Press, 2013), 173–201. 6. L. Carmichael, H. P. Hogan, and A. A. Walter, “An Experimental Study of the Effect of Language on the Reproduction of Visually Perceived Form,” Journal of Experimental Psychology 15 (1932): 73–86. 7. P. Tagg, and B. Clarida, Ten Little Title Tunes: Towards a Musicology of the Mass Media (New York: Mass Media Music Scholars’ Press, 2003), 105. 8. Ibid. 9. P. H. Tannenbaum, “Music Background in the Judgment of Stage and Television Drama,” Audiovisual Communication Review (1956): 92–101 10. C. E. Osgood, G. J. Suci, and P. H. Tannen-baum, The Measurement of Meaning (Urbana: University of Illinois Press, 1957). 11. L. Quinto and W. F. Thompson, “Composers and Performers Have Different Capacities to Ma-nipulate Arousal and Valence,” Psychomusicology: Music, Mind, & Brain 23 (2013): 137–50; and J. A. Russell, “A Circumplex Model of Affect,” Journal of Personality and Social Psychology 39 (1980): 1161–78. 12. J. F. Thayer and R. W. Levenson, “Effects of Music on Psychophysiological Responses to a Stressful Film,” Psychomusicology: Music, Mind & Brain 3, no. 1 (1983): 44–52. 13. S. K. Marshall and A. J. Cohen, “Effects of Musical Soundtracks on Attitudes to Geometric Figures, Music Perception 6 (1988): 95–12.

MMI 8_2 text.indd 20 5/28/15 10:38 AM


14. Thayer and Levenson, “Effects of Music,” 44–52. 15. Marshall and Cohen, “Effects of Musical Soundtracks,” 44–52. 16. I would like to acknowledge the work of Katherine Spring that has related the concepts described below of congruence, association, and working narrative to her treatment of audiovis-ual synchronization in three films directed by Tom Tykwer. Spring, “Chance Encounters of the Musical Kind: Electronica and Audiovisual Syn-chronization in Three Films Directed by Tom Tyk-wer,” Music and the Moving Image 3 (2010): 1–14. 17. R. A. Kendall and S. D. Lipscomb, “Experi-mental Semiotics Applied to Visual, Sound, and Musical Structures,” in The Psychology of Musical Multimedia, ed. S. -L. Tan , A. J. Cohen , S. D. Lipscomb, and R. A. Kendall (Oxford: Oxford University Press, 2013), 48–65. 18. S. D. Lipscomb, “Cross-modal Alignment of Accent Structures in Multimedia,” in Tan et al., Psychology of Musical Multimedia, 193–213. 19. Z. Eitan, “How Pitch and Loudness Shape Musical Space and Motion,” in Tan et al., Psychol-ogy of Musical Multimedia, 165–91. 20. S. Iwamiya, “Perceived Congruence between Auditory and Visual Elements in Multimedia,” in Tan et al., Psychology of Musical Multimedia, 141–64. 21. See, e.g., G. A. Calvert, C. Spence, and B. E. Stein, eds., The Handbook of Multisensory Processes (Cambridge, MA: MIT Press, 2004); M. Kubovy and M. Yu, “Multistability, Cross-modal Binding and the Additivity of Conjoined Group-ing Principles,” Philosophical Transactions of the Royal Society, B 367 (2102): 954–64. 22. See, e.g., A. J. Bremner, D. J. Lewkowicz, and C. Spence, Multisensory Development (Oxford: Oxford University Press, 2013); and Calvert, et al., Handbook. 23. See also, Cohen, “Associationism and Musical Soundtrack Phenomena,” Contemporary Music Review 9 (1993): 166–67; and Donnelly, The Spectre of Sound: Music in Film and Television (London: British Film Institute, 2005), 6. 24. Warhorse, produced by S. Spielberg and K. Kennedy, directed by S. Spielberg (Universal City, CA: Touchstone Pictures and Dreamworks Studio, 2011). 25. see: https://www.youtube.com/watch?v =sF7o4Ckjhl4

26. P. H. Tannenbaum, “Music Background in the Judgment of Stage and Television Drama,” Audiovisual Communication Review (1956): 92–101. 27. Thayer and Levenson, “Effects of Music,” 44–52. 28. Marshall and Cohen, “Effects of Musical Soundtracks,” 44–52. 29. Cohen, Associationism, 166–67. 30. M. Boltz, M. Schulkind, and S. Kantra, “Ef-fects of Background Music on Remembering of Filmed Events,” Memory and Cognition 19 (1991): 595–606. 31. M. G. Boltz, “The Cognitive Processing of Film and Musical Soundtracks,” Memory & Cogni-tion 32 (2004): 1194–205. 32. S.-L.Tan, M. P. Spackman, and M. A. Bez-dek, “Viewers’ Interpretation of Film Characters’ Emotions: Effects of Presenting Film Music Before or After a Character is Shown,” Music Perception 25 (2007): 135–52. 33. M. G. Boltz, “Music Videos and Visual In-fluences on Music Perception and Appreciation: Should You Want Your MTV?,” in Tan et al., Psychology of Musical Multimedia, 213–30. 34. Cf., A. J. Cohen, “Music as a Source of Emotion in Film,” in Oxford Handbook of Music and Emotion, ed. P. Juslin and J. Sloboda (Oxford: Oxford University Press, 2010), 879–908; and Cohen, “Film Music and the Unfolding Narrative,” in Language, music, and the brain: A mysterious relationship, ed. M. Arbib (Cambridge, MA: MIT Press, 2013), 173–201. 35. R. Stam, R. Burgoyne, and S. Fitter-man-Lewis, New Vocabularies in Film Semiotics (London: Routledge, 1992), 37. 36. K. Overy and I. Molnar-Szakacs, “Being Together in Time: Musical Experience and the Mirror-Neuron System,” Music Perception 26 (2009): 489–504. 37. Babel, produced by S. Golin, J. Kilik, and A. G. Iñárritu, directed by A. G. Iñárritu (Paris: Cen-tral Films Studio; and Universal City, CA: Media Rights Capital Studio, 2006); and A. J. Cohen, “Resolving the Paradox of Film Music,” in The Social Science of Cinema, ed. J. C. Kaufman and D. K. Simonton (New York: Oxford University Press, 2014), 57–83. 38. D. Kulezic-Wilson, “The Music of Film Si-lence,” Music and the Moving Image 2 (2009): 1–10. 39. Ibid, 6.

MMI 8_2 text.indd 21 5/28/15 10:38 AM


40. Ibid, 2. 41. Ibid. 42. P. Boersma and D. Weenink, Praat: Doing Phonetics by Computer, version 5.3.51 [Computer program] (2013), retrieved June 2, 2013, http://www.praat.org/. 43. D. T. Blumstein, R. Davitian, and P. D. Kaye, “Do Film Soundtracks Contain Nonlinear Ana-logues to Influence Emotion?,” Biology Letters 6 (2010): 751–54. 44. A. J. Cohen and A. B. Graziano, eds., “A History of Music Psychology in Autobiography,” special issue, Psychomusicology 20 (2009): 1–162; D. Deustch, ed., The Psychology of Music: Third Edition (New York: Elsevier, 2013). 45. See, e.g., S. J. Derry, C. D. Schunn, and M. A. Gernsbacher, eds., Toward a Cognitive Science of Interdisciplinary Collaboration (Mahweh, NJ: Erlbaum, 2005); and M. O. O’Rourke, S. Crowley, S. D. Eigenbrode, and J. D. Wulfhorst, eds., Enhan-cing Communication and Collaboration in Inter-disciplinary Research (Los Angeles: Sage, 2013). 46. P. Tagg, Kojak—50 Seconds of Television Music (Towards the Analysis of Affect in Popular Music), 2nd ed. (1st published as PhD dissertation, Göteborg, Sweden: University of Göteborg, 1979; New York: Mass Media Music Scholars’ Press, 2000). 47. Ibid, 101–2. 48. While Tagg and Clarida’s 2003 empirical work (Ten Little Title Tunes) was in sympathy with scientific psychology, their methodology and analysis procedures were unconventional. For example, information about gender and age of the participants had not been recorded. The care in analysis, the effort to obtain reliability in scoring, the provision of an enormous data corpus and the rationale for the methodological decisions, and the insightful discussions are impressive. More-over, the results of the research (e.g., the finding of the gendering of the music examples) seem not to have been communicated through the channels that music psychologists normally use, and cur-rent music-psychological literature is not cited. I can imagine that the “translation” of the work of Philip Tagg into the language of contemporary psychomusicology could be the focus of one or more interdisciplinary graduate theses bridging cognitive psychology, music psychology, musicol-ogy, and semiotics.

49. Yet another recent example of cross-disci-plinary collaboration is the doctoral dissertation by Carlos Silveira at Université de Lyon, cosuper-vised by faculty members of the departments of film studies and neurosciences (Martin Barnier, Professeur des Universités en études cinémato-graphiques, Université Lyon; Barbara Tillman, head of Cognition Auditive et Psychoacoustique Centre de Recherche en Neurosciences de Lyon). Silveira, “La musique de film: Introduction à l’étude des attentes musico-filmiques du specta-teur” [Film Music: Introduction to the Study of the Viewers’ Film-Music Expectations] (doctoral thesis in film studies, Université Lumière Lyon 2, 2014).

ReferencesAudacity 2.0.4 [Computer program] (2013). Auda-

city.sourceforge.net.Babel. Produced by S. Golin, J. Kilik, and A. G.

Iñárritu. Directed by A. G. Iñárritu. Universal City, CA: Summit Entertainment; Paris: Central Films Studio; and Beverly Hills, CA: Media Rights Capital Studio, 2006.

Bermúdez, J. L. Cognitive Science: An Introduction to the Science of the Mind. Cambridge: Cam-bridge University Press. 2010.

Boersma, P., and D. Weenink. Praat: Doing phon-etics by computer, version 5.3.51 [Computer program] (2013). Retrieved June 2, 2013, http://www.praat.org/.

Boltz, M. G. “The Cognitive Processing of Film and Musical Soundtracks.” Memory & Cognition 32 (2004): 1194–205.

Boltz , M. G. “Music Videos and Visual Influences on Music Perception and Appreciation: Should You Want Your MTV?” In The Psychology of Musical Multimedia, edited by S.-L. Tan, A. J. Cohen, S. D. Lipscomb, and R. A. Kendall, 213–30. Oxford: Oxford University Press, 2013.

Boltz, M. G., B. Ebendorf, and B. Field. “Audiovis-ual Interactions: The Impact of Visual Informa-tion on Music Perception and Memory.” Music Perception 27 (2009): 43–59.

Boltz, M., M. Schulkind, and S. Kantra. “Effects of Background Music on Remembering of Filmed Events.” Memory and Cognition 19 (1991): 595–606.

Blumstein, D. T., R. Davitian, and P. D. Kaye. “Do Film Soundtracks Contain Nonlinear Ana-

MMI 8_2 text.indd 22 5/28/15 10:38 AM


logues to Influence Emotion?” Biology Letters 6 (2010): 751–54.

Bremner, A. J., D. J. Lewkowicz, and C. Spence. Multisensory Development. Oxford: Oxford University Press, 2013.

Cabak Rédei, A. “Film, Music and Experimental Psychology: Reflections and Further Direc-tions.” Music and the Moving Image 8, no. 2 (2015): this issue.

Calvert, G. A., C. Spence, and B. E. Stein, eds. The Handbook of Multisensory Processes. Cam-bridge, MA: MIT Press, 2004.

Carmichael, L., H. P. Hogan, and A. A. Walter. “An Experimental Study of the Effect of Language on the Reproduction of Visually Perceived Form.” Journal of Experimental Psychology 15 (1932): 73–86.

Cohen, A. J. “Associationism and Musical Sound-track Phenomena.” Contemporary Music Review 9 (1993): 163–78.

Cohen, A. J. “Autobiography and Psychomusic-ology: Introduction to the Special Volume—‘A History of Music Psychology in Autobiog-raphy.’” Psychomusicology: Music, Mind, & Brain 20 (2011): 10–19.

Cohen, A. J. “Film Music and the Unfolding Narrative.” In Language, Music, and the Brain: A Mysterious Relationship, edited by M. Arbib, 173–201. Cambridge, MA: MIT Press, 2013.

Cohen, A. J. “Film Music from the Perspective of Cognitive Science.” In Oxford Handbook of Film Music Studies, edited by D. Neumeyer, 96–130. New York: Oxford Univ. Press, 2014.

Cohen, A. J. “Music as a Source of Emotion in Film.” In Oxford Handbook of Music and Emo-tion, edited by P. Juslin and J. Sloboda, 879–908. Oxford: Oxford University Press, 2010.

Cohen, A. J. “Resolving the Paradox of Film Music.” In The Social Science of Cinema, edited by J. C. Kaufman & D. K. Simonton, 57–83. New York: Oxford University Press, 2014.

Derry, S. J., C. D. Schunn, and M. A. Gernsbacher, eds. Interdisciplinary Collaboration. Mahwah, N. J.: Erbaum, 2005.

Deutsch, D. Preface to The Psychology of Music: Third Edition, edited by D. Deutsch, xiii–xviii. New York: Elsevier, 2013.

Donnelly, K. The Spectre of Sound: Music in Film and Television. London: British Film Institute, 2005.

Gorbman, C. Unheard Melodies: Narrative Film Music. Bloomington, IN: Indiana University Press, 1987.

Köhler, W. Gestalt Psychology: An Introduction to New Concepts in Modern Psychology, rev. ed. New York: Liveright, 1947.

Kubovy, M., and M. Yu. “Multistability, Cross-modal Binding and the Additivity of Conjoined Grouping Principles.” Philosophical Transactions of the Royal Society, B 367 (2012): 954–64.

Kulezic-Wilson, D. “The Music of Film Silence.” Music and the Moving Image 2 (2009): 1–10. Marshall, S. K., and A. J. Cohen. “Effects of Musical Soundtracks on Attitudes to Geometric Figures.” Music Perception 6 (1988): 95–112.

O’Rourke, M. O., S. Crowley, S. D. Eigenbrode, and J. D. Wulfhorst, eds. Enhancing Communi-cation and Collaboration in Interdisciplinary Research. Los Angeles, CA: Sage, 2013.

Osgood, C. E., G. J. Suci, and P. H. Tannenbaum. The Measurement of Meaning. Urbana: Univer-sity of Illinois Press, 1957.

Quinto, L., and W. F. Thompson. “Composers and Performers Have Different Capacities to Ma-nipulate Arousal and Valence.” Psychomusicol-ogy: Music, Mind, & Brain 23 (2013): 137–50.

Russell, J. A. “A Circumplex Model of Affect.” Journal of Personality and Social Psychology 39 (1980): 1161–78.

Silveira, C. H. G. “La musique de film: Introduc-tion à l’étude des attentes musico-filmiques du spectateur” [Film Music: Introduction to the Study of the Viewers’ Film-Music Expecta-tions]. Doctoral thesis in film studies, Univer-sité Lumière Lyon 2, 2014.

Spring, K. “Chance Encounters of the Musical Kind: Electronica and Audiovisual Synchroniz-ation in Three Films Directed by Tom Tykwer.” Music and the Moving Image 3 (2010): 1–14.

Stam, R., R. Burgoyne, and S. Fitterman-Lewis. New Vocabularies in Film Semiotics. London: Routledge. 1992.

Tagg, P. Kojak—50 Seconds of Television Music (To-wards the Analysis of Affect in Popular Music), 2nd ed. First published as PhD diss., Göteborg, Sweden: University of Göteborg, 1979; New York: Mass Media Music Scholars’ Press, 2000.

Tagg, P., and B. Clarida. Ten Little Title Tunes: Towards a Musicology of the Mass Media. New York: Mass Media Music Scholars’ Press, 2003.

MMI 8_2 text.indd 23 5/28/15 10:38 AM


Tan, S.-L., M. P. Spackman, and M. A. Bezdek. “Viewers’ Interpretation of Film Characters’ Emotions: Effects of Presenting Film Music Before or After a Character is Shown.” Music Perception 25 (2007): 135–52.

Tannenbaum, P. H. “Music Background in the Judgment of Stage and Television Drama.” Audiovisual Communication Review (1956); 92–101.

Thayer, J. F., and R. Levenson. “Effects of Music on Psychophysiological Responses to a Stressful Film.” Psychomusicology: Music, Mind & Brain 3, no. 1 (1983): 44–52.

Wallengren, A.-K., and A. Strukelj. “Film Music and Visual Attention: An Eye-Tracking Pilot Experiment.” Music and the Moving Image 8, no. 2 (2015): this issue.

Warhorse. Produced by S. Spielberg and K. Ken-nedy. Directed by S. Spielberg. Universal City, CA: Touchstone Pictures and Dreamworks Studio, 2011.

Watson, J. “Psychology as the Behaviorist Views It.” Psychological Review 20 (1913): 158–77.

MMI 8_2 text.indd 24 5/28/15 10:38 AM

congruence-association model and experiments in film music - toward interdisciplinary collaboration...

Documents