multi mod al discourse

7/28/2019 Multi Mod Al Discourse

1/5

Book review

New Directions in the Analysis of Multimodal Discourse

Terry D. Royce and Wendy L. Bowcher (Eds.), Mahwah, NJ: Lawrence Erlbaum, 2007. ISBN 0-8058-5106-2 (cloth).

403 pp.

A rapidly waxing flow of multimodality floods publishers catalogues, journals, and conferences. This is

understandable, since discourses that draw on more than just the good old language modality present themselves in

ever more emphatic ways, due to developments such as the growing multifunctionality of cell phones, the rise of

computer games, and the huge popularity of Youtube and Internet-surfing in general. In this monomodal review I

discuss a book that promises innovative perspectives on multimodality, Royce and Bowchers edited volume New

Directions in the Analysis of Multimodal Discourse.The coherence between the twelve chapters in the book resides mainly in the adherence to the Systemic

Functional Linguistics (SFL) approach rooted in Hallidayan grammar, while Kress and Van Leeuwens (1996/2006)

pioneering application of this approach to the visual realm is cited in every chapter. I should say straightaway that I

have no first-hand knowledge of Hallidays work. This has the drawback that I cannot verify how true to the letter or

spirit the authors are to their primary sources of inspiration, and the advantage that I can assess with relative

freshness what promising tools SFL has to offer for analyzing multimodality. Let me further add that my own

primary affiliation is to Cognitive Linguistics and Relevance Theory models, as developed by such scholars

as Lakoff, Johnson, Gibbs, Kovecses, Turner, Fauconnier, and Sperber and Wilson. However, I consider myself

a critical and eclectic follower of these paradigms. The tree is known by its fruits, and I will happily make use

of any theoretical insight that can help understand multimodality, irrespective of its source. I will first briefly, and

with no claim to exhaustiveness, discuss each chapter separately, and end by charting and evaluating sharedtendencies.

The 60 page-long first chapter by Matthiessen is devoted to The multimodal page: A Systemic Functional

exploration. After some preliminary remarks, the author proposes first to examine multimodality within language

(p. 11), because allegedly the ideational resources of language provide us with a range of strategies for modeling [. . .]

all modes and media of expression (p. 21). Then I more or less lose track. I cannot help feeling that the many

technical terms Matthiessen introduces come as a substitute for explanation, rather than as its prerequisite. The

complex panels and figures (figure 1.14 has more than 80 boxes!) are not particularly encouraging either. And while

the author makes some interesting observations on articulation and prosody as distinguishable modalities in

spoken language, and on the rhetorical patterns in verbal texts, he has very little to say about the visual realm, let alone

about interrelations between word and image.

Royce usefully begins his chapter Intersemiotic complementarity: A framework for multimodal discourse

analysis by briefly outlining the three metafunctions Halliday distinguishes in language: the ideational (pertaining

to meaning as it arises out of the relation between text and world), the interpersonal (pertaining to the relation

between communicator and addressee), and the textual (pertaining to the internal coherence of a discourse and the

way it links up to extratextual context). Royce argues that these metafunctions are applicable to non-verbal

communication as well and uses an Economistarticle, accompanied by a cartoon and a diagram, to demonstrate that

repetition, antonymy, hyponymy, meronymy, and collocation are concepts that define relations both

within and straddling the verbal and visual modalities on the ideational level. He comes up with similar modality-

independent concepts for the interpersonal and textual levels. Moreover, he introduces the notion of visual message

elements (VMEs), visual features which carry semantic properties [. . .] potentially realized by a variety of visual

www.elsevier.com/locate/pragma

Available online at www.sciencedirect.com

Journal of Pragmatics 41 (2009) 14591463

0378-2166/$ see front matter # 2009 Elsevier B.V. All rights reserved.

doi:10.1016/j.pragma.2009.01.007
http://dx.doi.org/10.1016/j.pragma.2009.01.007http://dx.doi.org/10.1016/j.pragma.2009.01.007


2/5

techniques at the disposal of the visual designers (p. 70). The in-depth analysis of a single case-study in the light

of clearly outlined theoretical concepts provides a useful starting point for applications by other scholars to other

(types of) texts.

Thibaults chapter focuses on the visual dimensions of language. He proposes a typology of graphic dimensions of

verbal elements (e.g., a grapheme is the visual expression of the verbal letter, and logogram corresponds to

word). This is insightful, but it is a pity that the author tends to be rather verbose, and to use an overdose of jargon to

explain sensible but uncomplicated ideas. He goes on to illustrate how the levels of content and graphic expressioncombined create meaning in a Duracell ad, drawing on Gestalt theory concepts (figure-ground, spatial orientation,

distance, similarity, continuity of direction, directionality or orientation, closure). Thibault ends his chapter by

showing how visually deviant language (as in comics and cartoons) can resonate with visuals in a way that enhances a

sense of rhythm that is difficult to capture in words alone.

In a fine chapter, Bateman, Delin, and Henschel convincingly expound how the impact of content shifts when it is

presented in different media. They present lists of verbal, visual, and production/consumption-related dimensions that

need minimally to be taken into account for an adequate description of how multimodal documents communicate, and

that allow for comparison between documents belonging to the same genre. What I particularly like in this research,

which is part of a larger Genre and multimodality project, is that it combines commonsense theoretical concepts

with honest, systematic data analysis, the central idea being that one can only discover interesting patterns after

identifying and quantifying salient variables. The case study, which compares traditional and electronic versions ofnewspapers, elegantly underscores this point.

Baldry reports about the benefits and prospects of the Multimodal Corpus Authoring (MCA) system he

helped develop. He, too, emphasizes the importance of corpus research, and advances proposals for the labelling and

description of pertinent variables in multimodal discourse indispensable for enabling generalizations and

formulating testable hypotheses. Among other things, he discusses the issue of segmenting a text, taking into account

linguistic, prosodic, and visual features. A case study exemplifies the approach, and further cooperation is invited via

the MCA, accessible online.

Lim takes his cue from fellow contributor Martins earlier work to develop visual counterparts of verbal

organization mechanisms, specifically visual taxonomy, visual configuration, visual reference, and visual taxis, and

applies them to comics. He relies on McClouds (1993: 7072) six types of frame-transition, even though these types

are sometimes difficult to distinguish. Among the concepts he develops and discusses are the Associating Element(e.g., a cornflakes package connotes breakfast and a tree connotes outdoor scene), and the visual linking

device (VLD), the latter enabling a viewer to understand how the action in one panel relates to that in a previous

panel, e.g., by the repetition of (part of) a character or scene.

Martin and Stenglin describe how spatial organization and layout in a post-colonial museum in New

Zealand, bridging indigenous and white perspectives on history and culture, is made communicatively

significant. Among the tools they deploy in their analyses are the given-new dichotomy, bound versus

unbound spaces, and orbital versus serial navigation paths. The authors focus on a single modality only,

but persuasively argue that this modality space needs to be taken into account in all communication that

involves physical motion.

Bowcher draws on her co-editors model for A multimodal analysis of good guys and bad guys in Rugby League

Week. Her analysis, of a newspaper article, is dauntingly detailed. Although mainly focusing on verbal aspects, the

author also discusses photographs and headings, and their location on the page. Importantly, she tries to show howinformation in the verbal and visual modalities are connected. Her tentative claim that the direction in which the rugby

players in the photographs look correlates with where in the verbal text goodness and badness are discussed

is suggestive. But even if correct many more case-studies will have to be conducted to assess whether such

gaze-direction mechanisms are in any sense generalizable.

Mohan et al., claiming that the mere fact that similar graphics occur in textbooks [i.e., scientific textbooks in

different languages, ChF] does not mean that readers of those textbooks interpret them in the same way (p. 277),

asked Chinese, English, and Japanese readers to respond in their own language to a diagram representing the water

cycle. A sample of the texts was subsequently analyzed in more detail, revealing that some participants produced

texts which had a sequential line of meaning, while others produced texts that explicitly reflected causal relationships

(p. 278). The authors then hypothesize that causal line texts exemplify the features of scientific discourse more than

do time line texts (ibid.). The hypothesis is confirmed but what has been proven? Surely there is something circular

Book review1460


3/5

going on here: the only conclusion we can draw is that subjects who produced causal texts (Y happens because of X)

understood the water cycle better than those who produced timeline texts (Y happensafterX), and that language and

cultural background had no effect on this.Ergo, some English, Japanese, and Chinese subjects, but not others, would pass

their Physics test . . .

Japanese semiotic vernaculars in ESP multiliteracies projects is Ferreiras report of a classroom task performed

by Japanese students in two consecutive years to produce a multimodal tourist brochure aimed at an English-speaking

audience. He notes, among other things, that Japanese right-left writing and the possibilities for vertical layout of textcan have (sometimes: unintended) effects on the visuals in the brochure.

Unsworth investigates how stories are affected by their translation into a different medium. Here, this classic

type of remediation research assumes some urgency because the target audience consists of children. Unsworth

emphatically presents the comparative multimodal analyses of Stellaluna, Shrek, and The Little Prince and their

CD-ROM/filmic counterparts in terms of fostering a critical pedagogic practice (p. 333). He unveils a number of

salient differences, both in the language and visuals used, which particularly in Shrekreveal that much of the prickly

unconventionality of the original has been smoothed out in the big-screen version.

Royces programmatic Multimodal communicative competence in second language contexts would have made a

good opening chapter. He takes his inspiration from the New London Group to promote the teaching of multimodal

competences by extending Dell Hymes four criteria for effective communication (grammatical/formal correctness,

ease of production and processing, appropriateness to context, and effects in the real world). For the analysis of graphicelements he draws on a list by Donis Dondis which consists of dot, line, shape, direction, tone, color, texture, scale/

proportion, and dimension/motion. He ends the chapter by presenting a number of classroom tasks that can aid

multimodal communicative competence.

Let me signal and comment on some themes and patterns in the book. First, the positive and promising

developments I discern:

1. Most chapters insist that teaching multimodal communicative competence is a necessary component in (secondary)

education. Having to teach multimodality forces researchers and educators to develop clear-cut tools and models,

whose usefulness is demonstrated by sample analyses but which are flexible enough to be applicable to new texts.

Focusing on multimodal discourses also sharpens pupils awareness of the relative strengths and weaknesses of

different modalities. At the very least, this work outlines an important roadmap for the humanities curriculum ofthe near future.

2. The laudable use of corpora, though sometimes small ones, surfaces in several chapters. By definition, detecting

patterns requires comparing things. Even comparing two things using the same measuring tool is already infinitely

more useful for theorizing than an application to a single case-study. Comparing more than two items (dozens,

hundreds) is better, of course, but given that qualitative research of multimodal texts can be very time-consuming,

this is not always feasible. A bonus of a corpus approach is that, given a clear theoretical framework, comparative

work can be divided over a group of students who are to earn credits for course work.

3. In the visual modality, spatial dimensions are crucial. Moreover, given that space involves movement it is

not surprising that several chapters mention embodiment a central concept in the Cognitive Linguistics

approach. Specifically the source-path-goal schema (discussed by Mohan et al.) has overlap with the importance

attributed to vectors in the SFL framework, and provides fruitful angles for analyzing audiovisual modalities

(see Johnson, 1987).4. An often-quoted term in the volume is Gibsons (1979: chapter 8) notion of affordance (roughly: that what you

can do with a thing or concept). It makes sense to add its opposite, constraints (that what you can not do with a

thing or concept). My view is that this pair is very helpful in the assessment of the opportunities and impossibilities

pertaining to media, modalities, and genres (see Forceville, 2006a).

I also see serious problems:

1. Many chapters subscribe to the idea that the visual modality constitutes a system comparable to that exemplified

in the grammar and vocabulary of the verbal modality. This confidence, typical of old-style semiotics, is not

warranted, at least not to the degree voiced in this book. The notion of a visual grammar (Kress and Van

Leeuwen, 1996/2006) needs to be used with extreme caution. The term betrays the extent to which work on

Book review 1461


4/5

multimodality is undertaken by scholars trained as linguists, and entails the risk that claims about the need to

acquire visual competence are either exaggerated or misdirected the former because understanding pictures

comes to humans a lot more naturally than understanding language; the latter because the language model may

blind researchers to affordances characteristic of other modalities and media that, by definition, are absent or

downplayed in verbal communication. Moreover, while many of the concepts proposed byKress and Van Leeuwen

(1996/2006), such as given-new, centre-margin, and vectors, are exciting and potentially fruitful, I am worried that

they are simply considered as facts that can be unproblematically used to prove things, whereas in reality theseconcepts are at present no more than hypotheses requiring further critical evaluation as well as empirical testing

(see Forceville, 1999).

2. Apart from the fact that modality itself is not adequately defined, this book focuses predominantly on (the

interrelations between) two modes only: the verbal and the visual. Martin and Stenglin are absolutely right to point

out that the two-dimensional page affords representing language and (static) pictures at the expense of, for

instance, space, sound, music, moving images, and gestures. We thus run the risk of being blind to the specificities

of these other modalities, and of dangerously narrowing the concept of multimodality.

3. Even the visual modality gets little attention, compared to the verbal modality. As to the verbal modality:

analysis in this vein continues (uncited) work in the venerable tradition of stylistics, pioneered by Leech and

Short (Leech, 1966, 1969; Leech and Short, 1981), and associated with the journal Language and Literature. As

to the short shrift paid to the visual modality: to some extent, this is inevitable, since there are few tried-and-tested tools for the analysis of contemporary visual culture (film theory is an exception; see Bordwell and

Thompson (2008), while McClouds work (1993, 2006) holds promise for comics). But in order to discuss how

different modalities interact in multimodal discourse, basic knowledge of how they function on their own is

indispensable. Hard work is needed to develop this non-linguistic knowledge; otherwise what is bound to

happen a problem surfacing in many of the chapters of the current book is that the visual modality is seen as

always merely supportive of the verbal.

For my own recent attempts to practise what I preach, see Forceville (2005a,b, 2006b), Forceville and Jeulink

(2007), Eggertsson and Forceville (in press), and Forceville et al. (forthcoming).

References

Bordwell, David, Thompson, Kristin, 2008. Film Art: An Introduction, 8th ed. McGraw-Hill, Boston.

Eggertsson, Gunnar Theodor, Forceville, Charles, in press. The HUMAN VICTIM IS ANIMAL metaphor in horror films. In: Forceville, C., Urios-Aparisi, E.

(Eds.), Multimodal Metaphor, Mouton de Gruyter, Berlin, pp. 423444.

Forceville, Charles, 1999. Educating the eye? Kress & Van Leeuwens Reading Images: the grammar of visual design. Language and Literature 8,

163178.

Forceville, Charles, 2005a. Visual representations of the Idealized Cognitive Model of anger in the Asterix album La Zizanie. Journal of Pragmatics

37, 6988.

Forceville, Charles, 2005b. Addressing an audience: time, place, and genre in Peter Van Straatens calendar cartoons. Humor 18, 247278.

Forceville, Charles, 2006a. Non-verbal and multimodal metaphor in a cognitivist framework: agendas for research. In: Kristiansen, G., Achard,

M., Dirven, R., Ruiz de Mendoza Ibanez, F. (Eds.), Cognitive Linguistics: Current Applications and Future Perspectives. Mouton de

Gruyter, Berlin, pp. 379402.

Forceville, Charles, 2006b. The source-path-goal schema in the autobiographical journey documentary: McElwee, Van der Keuken, Cole. The New

Review of Film and Television Studies 4, 241261.Forceville, Charles, Jeulink, Marloes, 2007. The source-path-goal schema in animation film. Paper Presented at the International Cognitive

Linguistics Conference, Jagiellonian University Krakow, Poland, 1520 July.

Forceville, Charles, Veale, Tony, Feyaerts, Kurt, forthcoming. Balloonics: the visuals of balloons in comics. In: Goggins, J., Hassler-Forest, D.

(Eds.), Out of the Gutter, McFarland, Jefferson NC.

Gibson, James J., 1979. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston.

Johnson, Mark, 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. University of Chicago Press, Chicago.

Kress, Gunther, Van Leeuwen, Theo, 1996/2006. Reading Images: The Grammar of Visual Design. Routledge, London.

Leech, Geoffrey N., 1966. English in Advertising: A Linguistic Study of Advertising in Great Britain. Longman, London.

Leech, Geoffrey N., 1969. A Linguistic Guide to English Poetry. Longman, London.

Leech, Geoffrey N., Short, Michael H., 1981. Style in Fiction: A Linguistic Introduction to English Fictional Prose. Longman, London.

McCloud, Scott, 1993. Understanding Comics. Paradox, New York.

McCloud, Scott, 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. HarperCollins, New York.

Book review1462


5/5

Charles Forceville studied English language and literature at the Vrije Universiteit, Amsterdam, where he subsequently taught in the Departments

of English, comparative literature, and Word & Image. Since 1999 he has been employed in the Department of Media Studies of the Universiteit van

Amsterdam, where he is now associate professor and directs the Departments Research Master programme. In 1996, he published Pictorial

Metaphor in Advertising (Routledge); the volume Multimodal Metaphor (Mouton de Gruyter), co-edited with Eduardo Urios-Aparisi, is

forthcoming. His teaching and research interests focus on the structure and rhetoric of multimodal discourse in various media and genres

(documentary, animation, comics, cartoons). Between 1987 and 2007 Forceville wrote some 200 literary reviews for the Dutch newspaper Trouw.

Ch.J. Forceville*Department of Media Studies, University of Amsterdam Turfdraagsterpad 9,

1012 XT Amsterdam, The Netherlands

*Tel.: +31 20 5254596; fax: +31 20 5254599.

E-mail address: [email protected].

URL: http://home.medewerker.uva.nl/c.j.forceville/

Book review 1463
mailto:[email protected]://home.medewerker.uva.nl/c.j.forceville/http://home.medewerker.uva.nl/c.j.forceville/mailto:[email protected]

multi mod al discourse

Documents