Quantitative Analysis of Non-Obvious PerformerGestures

Marcelo M. Wanderley

Ircam - Centre Pompidou1, Place Igor Stravinsky75004 - Paris - Francemwanderley@acm.org

Abstract. This article presents quantitative results from movement analysis ofseveral clarinet performers with respect to non-obvious or ancillary gestures pro-duced while playing a piece. The comparison of various performances of a pieceby the same clarinetist shows a striking consistency of movement patterns, aswell as performances of the same piece with different expressive characteristics.Different clarinetists show different overall patterns, although clear similaritiesmay be found, suggesting the existence of various levels of information in theresulting movement. The relationship of these non-obvious movements to mate-rial/physiological, structural and interpretative parameters will be highlighted.

1 Introduction

Musical performance is an interesting field for the analysis of human-human commu-nication due to its expressive content. Not only traditional musical parameters are con-veyed – melody, rhythm, articulation – but also information on different levels, such asa high content of emotion [4], contribute to the overall listening experience.

The analysis of musical performance is therefore a broad research subject basedon knowledge and methods from several domains, as shown in large reviews of thedifferent studies in this area [3] [6].

Among the various works on musical performance, few studies have focused onthe gestural behavior of instrumentalists. These have shown that musicians not onlyperform skilled movements directly related to sound production, but also movementsthat seem not to have a straight link to the generation of sound. These works havemostly focused on piano playing [2], piano and violin [1], and more recently on theclarinet [7].

Delalande [2] studied the gestural behavior of late pianist Glenn Gould from videorecordings of his performances. He was able to show a strong correlation between ex-pressive or, as he termed, accompanist movements, to the score being played. Davidson[1] presented a study on the perception of visual cues related to expressive movementsof four violin players as well as of a pianist. She showed that the visual informationavailable from the different performances (standard, exaggerated and deadpan) couldeven surpass the auditory information when related to the perception of expressive mu-sical features.

In [7], it has been shown that, unlike the case of piano playing, non-obvious, an-cillary or expressive movements performed by wind instrumentalists do influence theresulting sound. This influence is the result of the displacement of the sound sources(the open holes and bell of a clarinet) with respect to a fixed close microphone in a re-verberant environment [8]. Therefore, when using a computer simulation of an acousticwind instrument, the modeling of these effects will contribute to a naturalness found incertain real situations, expressed by the temporal fluctuations of sound partials’ ampli-tudes 1.

In this work, three main points will be addressed based on the quantitative analysisof clarinetists’ movements:

– The production of ancillary gestures, i.e., whether it is common to most playersto move the instrument while playing, what is the magnitude order of these move-ments, whether one can identify basic movement patterns, and what is the influenceof external factors on movement production.

– The repeatability of ancillary gestures, i.e., whether one clarinetist repeats the samemovements while playing one piece different times, and what can be the correlationbetween movements during different performances of the same piece.

– The possible production of similar movement patterns by different performers, i.e.,whether there exist any movement patterns common to different performers andwhether there exists a direct relation of movements of different performers of thesame score.

1.1 Quantitative Data Analysis - Methods an Materials

Quantitative movement data acquisition was performed in collaboration with the Fac-ulty of Movement Studies at the Free University of Amsterdam and the Music, Mindand Machine Group at the NICI, Nijmegen, the Netherlands.

An Optotrak system was used to measure players’ movements during performance.It consists of a three-dimensional highly accurate acquisition system based on infrared(IR) markers tracked by three IR cameras. In this study, between eight and ten markerswere placed on the musicians, who were standing at a distance of around 3 to 4 me-ters away from the camera. The sampling rate used for each marker was 100 Hz, eachmarker giving a three dimensional coordinate at each sample time (horizontal, verticaland sagittal coordinates).

Four clarinet players have been recorded – French clarinetist Pierre Dutrieu andDutch clarinetists Oskar Ramspek, Lars Wouters and Hans Mossel. All of them areprofessional musicians with many years experience in both solo, chamber and/or or-chestral performances.

Figure 1 shows Pierre Dutrieu and the placement of the markers during the recordingsession at the Free University Amsterdam.

1 May also be true for the case of the violin and other instruments where performer movementscause displacements of the sound sources.

Fig. 1. Pierre Dutrieu at the movement data acquisition session at the Free UniversityAmsterdam. The coordinate system and the positions of each marker are displayed inthe right picture.

The pieces selected consisted of basic standard clarinet repertoire, both solo piecesand sonatas for clarinet and piano. The list of recorded pieces is the following:

– Domaines, by Pierre Boulez. Original. Cahiers A to E and F (incomplete).– First Clarinet Sonata, by Johannes Brahms. First movement (incomplete).– Clarinet Sonata, by Francis Poulenc. First and second movements (both incom-

plete).– Three Clarinet Pieces, by Igor Stravinsky. First and Second Pieces.

In this paper two of the recorded pieces are analyzed in detail: Domaines, by PierreBoulez, Cahier A, and an excerpt of the first movement of the first clarinet sonata, byJ.Brahms2.

The players were asked to play these pieces three times: a standard performance,an expressive performance, and a performance trying not to move the instrument atall3. Sometimes these different performances were repeated in order to get supplemen-tary data. Finally, the players also performed all pieces seated, in order to verify theirgestural behavior in different contexts.

2 Production of Ancillary Gestures

Figure 2 shows the standard performance of Domaines, cahier A original. The curvedisplays the vertical movement of the marker (number 5) placed on the clarinet bell.From top to bottom: the standard, expressive and immobilized performances. The ver-tical axis shows the vertical movement of the instrument’s bell in millimeters, and thehorizontal axis shows time in cents of seconds.

2 No piano was used in the recording of the pieces3 This performance style is different from Davidson’s deadpan style, where deadpan meant

inexpressive. I am here interested in knowing whether it is possible or not to a performer toplay a piece without producing expressive gestures.

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500-600

-400

-200

0

posi

tion

[mm

]

CahierA - Normal Playing - Bell (5) - Vertical (Z)

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000-600

-400

-200

0

posi

tion

[mm

]CahierA - Exagg. Playing - Bell (5) - Vertical (Z)

0 1000 2000 3000 4000 5000 6000-600

-400

-200

0

time [s/100]

posi

tion

[mm

]

CahierA - Immob. Playing - Bell (5) - Vertical (Z)

Fig. 2.Three performances of Domaines, cahier A Original.

One can see from the three plots of figure 2 that there exists a strong correlation be-tween vertical movements of the bell during the standard and expressive performances.Also, it is apparent that Dutrieu again can play without significant movements, as shownin the third plot.

Figure 3 shows again the three plots, this time the third one being zoomed in.

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500-600

-400

-200

0

posi

tion

[mm

]

CahierA - Normal Playing - Bell (5) - Vertical (Z)

2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000-600

-400

-200

0

posi

tion

[mm

]

CahierA - Exagg. Playing - Bell (5) - Vertical (Z)

0 1000 2000 3000 4000 5000 6000

-350

-300

-250

time [s/100]

posi

tion

[mm

] **

CahierA -Immob. Playing Zoomed- Bell (5) - Vertical (Z)

Fig. 3. Three performances of Domaines, cahier A Original. Third performance withzoomed scale.

Analyzing figure 3, one can see that even for a very reduced movement amplitude(maximum vertical scale of 6 cm), certain movement patterns tend to be reproduced.For instance, notice the downward movement around 38 s that corresponds to the be-ginning of the fourth section of the piece. As can be seen from figure 3, this representsa movement pattern consistent with the beginning of this section in all performances.

Finally, figure 4 shows vertical, horizontal and sagittal coordinates of the instru-ment’s bell for the three performances. Ancillary gestures were produced in all threedirections, presenting fairly large amplitudes, around several dozen centimeters.

1500 2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000-600

-400

-200

0

ampl

itude

[mm

]

CahierA - Normal Playing - Bell (5) - vertical (Z)

1500 2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000

-400

-200

0

ampl

itude

[mm

]

CahierA - Normal Playing - Bell (5) - horizontal (Y)

1500 2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000200

400

600

800

ampl

itude

[mm

]

CahierA - Normal Playing - Bell (5) - depth (X)

time [s/100]

Fig. 4.Domaines, cahier A, Pierre Dutrieu. Standard performance, vertical, horizontal and sagittalmovements of the instrument’s bell.

2.1 Repetition of independent Movement Patterns

One point to be studied refers to the possible existence of individual movement pat-terns and whether these could be produced by different players. In [7] three movementpatterns were proposed:

– Changes in posture (usually at the beginning of phrases).– Slow continuous gestures.– Fast sweeping movements of the bell.

Postural Adjustments Figure 5 shows clarinetist Lars Wouters performing the firstmovement of Brahms’ clarinet sonata, expressive performance. Extreme postural ad-justments are part of the performance, dependent on each player style.

Fig. 5.Three shots showing different postures during the (expressive) performance.

Continuous Circular Gestures The production of slow gestures, mostly circular pat-terns, can be also seen plotting the horizontal and sagittal coordinates. This is done inthe next figure were two instrumentalists perform part of the Brahms’ clarinet sonata.One can notice the circular patterns throughout the performance.

-500 -450 -400 -350 -300 -250 -200 -150 -100 -50 0250

300

350

400

450

500

550

600

650

700

750Pierre Dutrieu - Brahms Sonata1 (excerpt) - Normal Playing - Horizontal Plane

sagi

ttal p

ositi

on [m

m]

horizontal position [mm]-500 -450 -400 -350 -300 -250 -200 -150

-250

-200

-150

-100

-50

0

50

100Lars Wouters - Brahms Sonata1 (excerpt) - Normal Playing - Horizontal Plane

sagi

ttal p

ositi

on [m

m]

horizontal position [mm]

Fig. 6. Circular movements patterns (horizontal plane) during the first 2 seconds ofBrahms clarinet sonata. Left: Pierre Dutrieu, Right: Lars Wouters.

Fast Upwards MovementsThe same can be said about fast sweeping gestures. As anexample, the last upwards movement of Cahier A corresponds to an F played mf to ff,where Dutrieu has the tendency of “throwing the note away”. This can be seen in figure2 around 62 seconds in the first and second performances.

3 Repetition of Equivalent Movements

This section presents detailed analysis of data from multiple performances of the samepiece. For instance, figure 7, shows the superposed vertical movement of the marker

placed on the clarinet bell for two performances of Brahms’s clarinet sonata by HansMossel. Note the constancy of timing during more than 30 seconds of playing (with-out the piano accompaniment and without the use of a metronome) and the strikingsimilarity of the movements throughout the piece.

0 500 1000 1500 2000 2500 3000 3500450

500

550

600

650

ampl

itude

[mm

]

Hans MOSSEL (2,3) - Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time [s]

Fig. 7.Two standard performances of the first movement of Brahms’ clarinet sonata byHans Mossel.

0 500 1000 1500 2000 2500 3000 3500 4000

360

380

400

420

440

460

480

500

520

540

560

ampl

itude

Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time

Fig. 8.Two standard performances of the first movement of Brahms’ clarinet sonata byOskar Ramspek.

The same can be seen for the other three players. For instance, figure 8 shows twostandard performances of the clarinet sonata by Oskar Ramspek. Generally, one againmay note the temporal and movement consistencies between the two performances byRamspek. Only twice there have been differences in the movement pattern: around20 seconds and around 33 seconds. These moments seem to correspond to changesin movement strategy during the performance, since its temporal evolution was keptunchanged.

3.1 Correlation Analysis -Domaines - Cahier A

In the following figures, the third and fourth sections of Domaines, cahier A, are shownseparately for the three coordinates of the marker placed on the clarinet bell, from topto bottom, vertical, horizontal and sagittal.

0 100 200 300 400 500 600 700 800-500

-450

-400

-350

-300

-250

-200

-150

time [s/100]

posi

tion

[mm

]

0 100 200 300 400 500 600 700 800 900-600

-550

-500

-450

-400

-350

-300

-250

-200

-150

time [s/100]

posi

tion

[mm

]

0 100 200 300 400 500 600 700 800-400

-350

-300

-250

-200

-150

-100

-50

0

time [s/100]

posi

tion

[mm

]

0 100 200 300 400 500 600 700 800 900-400

-350

-300

-250

-200

-150

-100

-50

0

50

100

time [s/100]

posi

tion

[mm

]

0 100 200 300 400 500 600 700 800400

450

500

550

600

650

time [s/100]

posi

tion

[mm

]

0 100 200 300 400 500 600 700 800 900250

300

350

400

450

500

550

600

650

700

750

time [s/100]

posi

tion

[mm

]

Fig. 9. Standard (–) and Expressive (- -) Performances. From top to bottom: vertical,horizontal and sagittal movements of the bell. Left: third section; right: fourth section.Player: Pierre Dutrieu.

Correlation Coefficients Correlation coefficient between each pair of curves (standardand expressive performances) have been calculated and are shown 4 in the table below.

Correlation coef. 3rd section 4th sectionVertical movement 0.7917 0.9125

Horizontal movement 0.8022 0.8945sagittal movement —– 0.7229

4 Coefficients higher than 0.5

From the results above, one can see that the vertical and horizontal directions pre-sented high correlation coefficients. The sagittal direction presented a high correlationin the fourth section. Considering the number of points compared at each time (morethan eighty for each section), the results show a striking consistency of movement pat-tern, independently of the expressive nuance for the case of this player.

4 Repetition of Movements by Different Players

Finally, let us analyze the third question raised in the introduction of this chapter, i.e.,whether there are equivalent movements across multiple performers.

In figure 10, standard performances of Brahms’ clarinet sonata by all four musiciansare shown side-by-side.

0 500 1000 1500 2000 2500 3000 3500 4000-550

-500

-450

-400

-350

-300

-250

-200

-150

-100

-50

ampl

itude

[mm

]

Pierre DUTRIEU - Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time [s]0 500 1000 1500 2000 2500 3000 3500

300

350

400

450

500

550

600

650

ampl

itude

Lars WOUTERS - Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time

0 500 1000 1500 2000 2500 3000 3500400

450

500

550

600

650

700

ampl

itude

[mm

]

Hans MOSSEL (2) - Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time [s]0 500 1000 1500 2000 2500 3000 3500 4000

360

380

400

420

440

460

480

500

520

540

560

ampl

itude

Oskar RAMSPEK - Brahms Sonata1 - Normal Playing - Bell (5) - vertical (Z)

time

Fig. 10.Four performances of the clarinet sonata. From top to down, left to right: PierreDutrieu, Lars Wouters, Hans Mossel and Oskar Ramspek.

Figure 10 shows that there is no one-to-one similarity between the movements ofthe four players, as could be expected. Important differences concerning the amplitudeof movements as well as the temporal characteristics of each performance can be seen.

Performances of some players can be closer than from others, concerning the tem-poral evolution of the movement, for instance. Last two performances of figure 10 showsimilar number of dips, while the first performance presents half of that number, i.e.,the first player swings the instrument’s bell at roughly half the frequency of the latertwo.

This point is interesting as it indicates a rhythmic component of the total movement,related to the performance of the piece. In fact, this is a common feature of all perfor-mances of the Brahms’ sonata, but absent from the performance of Domaines since thetempo in contemporary music is not as well defined as in classical pieces.

5 Movement Parameters

It has already been shown that ancillary gestures performed by wind instrumentalistsare related to musical features in different levels [2]. In fact, from the different analysispresented in this paper, one can suggest different levels influencing clarinetists expres-sive movements:

– Material/Physiological, e.g. the influence of respiration, fingering, ergonomics ofthe instrument, etc. It can be seen, for instance, that at each time performers breathethere is a tendency to bring the instrument down to a vertical position and soonafterwards start an upwards movement again.

– Structural, i.e., dependent on the characteristics of the piece being performed. Wehave seen that all performers presented a rhythmic pattern associated to the piece.Although there are various differences in these movements, there exist similaritiesthat cannot be explained only by randomness alone.

– Interpretative, i.e., related to the mental model of the piece developed by the per-former and/or to specific interpretation conditions. For instance, in figure 8 the per-former changes the movement pattern twice, at around 18 s and at 32 s, althoughthe rhythmic component remains the same in both performances. The interpretativelevel will likely be different for different performers.

As an example of the second level, figure 11 shows three standard performancesof the clarinet sonata by Hans Mossel. The files have been time-warped to a referencescore in order to reduce small time variations across performances and allow an exactcomparison of each moment in the performances.

In this figure, solid vertical lines correspond to the note at the start of each bar,while dashed vertical lines correspond to the other notes in the score placed exactly atthe moment they occur for the case of a mechanical performance, i.e., with no deviationfrom absolute tempo.

One can see that at the beginning of each bar, this performer is at the same pointin an upwards part of the movement. This is an interesting finding as it suggests a kindof beat-keeping function of the gesture, although further research is needed in order toverify whether this fact can be extrapolated to different performers.

6 Conclusions

From what has been presented, general conclusions regarding clarinetists ancillary ges-tures can be drawn.

– Clarinet players are able to play a piece almost without any perceptible movement,although it has been stressed that this causes difficulties in respiration5. Therefore

5 Dutrieu considered that breathing was unnatural since it had to be done with the upper part ofthe chest.

0 5 10 15 20 25 30 35 40 45-150

-100

-50

0

50

100

150

Fig. 11.Three dynamically time-warped standard performances of the first movement of Brahms’clarinet sonata by Hans Mossel.

it is possible to disconsider physiological influences or aspects of the playing tech-nique as the main reasons for the production of ancillary gestures.

– The influence of environmental factors is not essential for the production of an-cillary gestures, since they were present in different circumstances, such as warmups, rehearsals and data collection sections. On the other hand, it is reasonable toconsider that in the presence of an audience, the emotion related to the context mayinfluence the performance and consequently the production of ancillary gestures.But these gestures will not completely cease to be produced because of environ-mental factors, at least for the case of professional performers.

– For the case of the same expert clarinet player performing one piece different times,one can reasonably consider that there will likely be a strong correlation betweenthe player’s movements at the same points in the score in the different perfor-mances. This fact suggests a strong relation between what is being played and howit is played. Therefore it is evident that ancillary gestures by clarinet players arenot randomly produced or just produced as a visual effect. On the contrary, thesegestures seem to be produced as a supplementary means of information communi-cation.

– Quantitative data from performances of different players shows that most of ancil-lary gestures present a high proportion of idiosyncratic movements. A three leveltypology of ancillary gestures (cf. section 5) according to their possible origin have

been proposed in order to take into account similarities and differences among thedifferent gestural patterns.

Finally, another interest of this research is to extrapolate findings presented in themusical context to general human-human communication. Understanding the behaviorof people manipulating instruments in expressive communicational situations such asmusic can lead to better designs of computer interfaces, not only for musical use, butalso in general. This may eventually help reducing the lack of expressive or affectivecontent [5] in current human-computer interaction interfaces.

7 Acknowledgements

Many researchers and performers contributed with comments and suggestions, contributions thatI would like to acknowledge here.

First of all, many thanks to all performers for their effort, patience during the experimentsand various comments.

Thanks to Peter Desain from the Music, Mind and Machine Group et NICI, Nijmegen andPeter Beek form the Faculty of Movement Studies, Free University of Amsterdam, for their col-laboration in this project making possible the use of the Optotrak system. I am also indebtedto Erwin Schoonderwaldt who performed all Optotrak measurements and also was instrumentalarranging the recording sessions at both NICI and the Free University.

Xavier Rodet and the Analysis-Synthesis Team at Ircam provided the necessary means toaccomplish this research.

This paper also greatly benefited from several comments and suggestions by Philippe De-palle, Ross Kirk, Stephan Tassart, Daniel Matzkin and David Ralley. Nicola Orio provided thedynamically time warped files from figure 11 and Thomas Helie provided the Matlab code (figure11) and several comments. To all of them, my warmest thanks.

Part of this research was funded by a doctoral scholarship from CNPq, Brazil.

References

1. J.-W. Davidson. Visual Perception of Performance Manner in the Movements of Solo Musi-cians. In Psychology of Music, volume 21, pages 103–113, 1993.

2. F. Delalande. La gestique de Gould. In Glenn Gould Pluriel, pages 85–111. Louise Courteau,editrice, inc., 1988.

3. A. Gabrielsson. Music Performance. In The Psychology of Music, pages 501–602, 1999.Diana Deutsch, editor. 2nd edition.

4. S. Hashimoto. Kansei as the Third Target of Information Processing and Related Topics inJapan. In Proc. KANSEI - The Technology of Emotion Workshop, pages 101–104, 1997.

5. R. Picard. Affective Computing. MIT Press, 1997.6. J. Sloboda. Music Performance. In The Psychology of Music. Academic Press Inc., 1982.

Diana Deutsch, editor.7. M. M. Wanderley. Non-Obvious Performer Gestures in Instrumental Music. In Gesture Based

Communication in Human-Computer Interaction, pages 37–48. Springer-Verlag, 1999. A.Braffort et al. editors.

8. M. M. Wanderley, P. Depalle, and O. Warusfel. Improving Instrumental Sound Synthesis byModeling the Effects of Performer Gesture. In Proc. of the 1999 International ComputerMusic Conference. San Francisco, Calif.: International Computer Music Association, pages418–421, 1999.