Rouages: Revealing the Mechanisms of Digital MusicalInstruments to the Audience

Florent BerthautPotioc, LaBRI

University of Bordeauxflorent.berthaut@labri.fr

Mark T. MarshallDepartment of Computer

ScienceUniversity of Bristol, UK

mark.t.marshall@bristol.ac.uk

Sriram SubramanianDepartment of Computer

ScienceUniversity of Bristol, UK

sriram.subramanian@bristol.ac.uk

Martin HachetPotioc, LaBRI

INRIA Bordeaux Sud-Ouestmartin.hachet@inria.fr

ABSTRACTDigital musical instruments bring new possibilities for mu-sical performance. They are also more complex for the au-dience to understand, due to the diversity of their com-ponents and the magical aspect of the musicians’ actionswhen compared to acoustic instruments. This complexityresults in a loss of liveness and possibly a poor experiencefor the audience. Our approach, called Rouages, is based ona mixed-reality display system and a 3D visualization appli-cation. Rouages reveals the mechanisms of digital musicalinstruments by amplifying musicians’ gestures with virtualextensions of the sensors, by representing the sound compo-nents with 3D shapes and specific behaviors and by showingthe impact of musicians’ gestures on these components. Inits current implementation, it focuses on MIDI controllers asinput devices. We believe that Rouages opens up new per-spectives for instrument makers and musicians to improveaudience experience with their digital musical instruments.

Keywordsrouages; digital musical instruments; mappings; 3D inter-face; mixed-reality;

1. INTRODUCTIONThe variety of digital musical instruments (DMI) is contin-uously increasing, with a plethora of commercial softwareand hardware production (e.g. the Reactable, the Monomeor the Novation Launchpad) and with the development ofprototyping platforms and languages. Electronic musiciansare now able to create their own instruments easily, eachwith very specific sets of sensors, sound processes/modulesand mappings [7] between the gestures/sensors and soundparameters, all of which can be different from other musi-cians. While this diversity allows the electronic musiciansto creatively express themselves it can often be confusing tothe audience. Research done on recreating the link betweengestures and digital sound for musicians should now alsoconsider this link from the audience point of view.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME’13, May 27 – 30, 2013, KAIST, Daejeon, Korea.Copyright remains with the author(s).

Figure 1: A digital musical instrument composed of16 pads, 8 knobs, 4 faders, two audio loops and5 synthesizers, without (top) and with (bottom)Rouages

1.1 Audience experience in electronic musicperformances

Members of the audience of electronic music performanceshave little or no information about the DMIs being playedand in contrast to acoustic instruments, they cannot deducethe behavior of DMIs from familiar physical properties suchas material, shape, and size. To add to this confusion, themost common configuration in these performances is a setof sensors, or sometimes simply a laptop, laid on a tableon the stage. The sensors themselves are then difficult forthe audience to see. The same is true for the small ges-tures that musicians make when performing. According to[14], performances with DMIs would be classified as magical,i.e. manipulations are hidden while their effects are known.Recent work by Marshall et al. [13] indicates that the per-

�1�6�4

ceived liveness of DMIs, which involves the identificationof relationships between gestures and resulting sound, im-pacts on the emotional response of members of the audience.When the perceived liveness is low, as in the case of laptopperformances, members of the audience might even thinkthat musicians only click on a play button and then spendthe entire performance reading their emails while shakingtheir head [5]. In short, a key problem in enhancing theaudience experience with DMIs is to make the performanceless magical and more expressive, so that members of theaudience perceive the involvement of musicians.

1.2 Musicians strategiesRecognizing this need to enhance audience experience, somemusicians adopt different strategies to overcome this lack ofvisibility of their actions. The controller is sometimes tiltedas done by Moldover 1 or filmed and reprojected as in Hi-fana’s performances 2 or in Bjork’s performances with theReactable 3. However, the impact of musicians’ gestures onthe sound is still hard to understand because one sensor maybe mapped to several sound parameters and because DMIsare often multiprocess instruments, i.e. instruments com-posed of several sound processes that may produce soundindependently from musicians’ actions. In addition, when are-projection is used the spatial consistency, i.e. collocationof the musician and the instrument, is broken. Accordingto Armstrong [1], ”the perceptual localization of the originof the sound is an important indicator of the instrument’sphenomenal presence, both for the performer, fellow per-formers, and the audience”. This also applies to the visualfeedback that combines the musician and instrument andthe virtual representation. In addition, when members ofthe audience watch a close-up of the sensors they miss theadditional accompanying gestures that also contribute tothe liveness of the performance [15]. More recently, someelectronic musicians such as Danger 4 have used a videoprojection of the graphical user interface of their DMI. Thisprovides some feedback on the modules of the instrument,but once again information such as accompanying gestures,the link between gestures and sound, and the precise be-havior of the DMI are hidden.

One may argue that graphical interfaces used in digitalmusical instruments such as the Reactable [9], FMOL [8],Different strokes [16] or Sound from shapes [11], while pro-viding valuable feedback and control possibilities for mu-sicians as pointed out by Jorda [8], also give the audienceenough information to understand the instrument. Howeverthese instruments may also confuse members of the audi-ence because of occlusions due to musicians’ gestures, thesmall size of the interface and non-collocated re-projectionshiding non-musical gestures. Finally the high level of detailneeded by musicians to properly control the instrument, e.g.the complete graph in the Reactable or additional graphicalelements which do not directly affect the sound, may pre-vent members of the audience from properly understandingboth the impact of musicians on the sound and the struc-ture of the instrument. Instead of relying on musicians’interfaces that are specific to each instrument, we believethat dedicated graphical interfaces should be developed forthe audience, which are collocated with the musicians andwhich include simplified representations of DMIs compo-nents, with generic behaviors that members of the audiencecan become accustomed to.

1http://www.youtube.com/watch?v=bOKIRNg rKI2http://www.youtube.com/watch?v=HFi-pKap3Vc3http://www.reactable.com/videos/bjork paris.mp44http://vimeo.com/23213096

In this paper, we present Rouages, a mixed-reality 3D vi-sualization system that is designed to improve the audienceexperience by revealing the mechanisms of digital musicalinstruments. Rouages consists of a hardware part that in itscurrent form handles MIDI controllers and other tabletopinput devices of DMIs and of a software application thatsimulates the inner mechanisms of the instrument througha 3D rendering. The main contributions of this paper arethreefold: 1) a mixed-reality system that extends DMIs sen-sors with virtual counterparts to amplify musicians’ ges-tures, 2) 3D representations of the dynamically analyzedsound processes of the instruments to reveal how the soundis produced, 3) dynamic links between virtual sensors andvisualized sound processes to show how musicians impactthe sound.

2. ROUAGES2.1 General ApproachA typical DMI, as depicted in Figure 2, is broadly made ofthree layers: the input device with sensors, a mapping layerto process/combine data from the sensors, and a sound layercomposed of sound processes and modules with controllableparameters. For example, in the case of a DJ, one of thesensors would be a linear potentiometer whose value wouldbe processed by a mapping layer to control simultaneouslythe volumes of two sound processes, increasing one volumewhile decreasing the other. In order to give the audience abetter perception of how a DMI produces sound and of howmusicians control it, Rouages uses three distinct approachesthat are different from previous attempts.

Audience

Figure 3: Left: Side view of the physical Rouagessystem: tiltable controller, room for the laptop,screen facing the audience. Right: Side view ofthe mixed-reality box as perceived by the audience(with one pad pressed).

- Virtually extending sensors: We amplify the musicians’gestures by virtually extending the sensors so that the au-dience can see small/hidden gestures.- 3D representation of the sound modules: We visualizethe sound production with 3D representations of the soundmodules so that members of the audience can understandthe sonic structure and behavior of the instrument.- Link virtual sensors with sound modules: We reveal theimpact of gestures by linking the virtual sensors and thesound modules, so that the audience can understand howthe musicians are actually modifying the sound.These three steps are defined in the next sections.

Rouages is composed of a mixed reality hardware setup asdepicted in Figure 3 and a software application that gathersdata from the sensors and the sound modules.The currenthardware part is oriented towards MIDI controllers as inputdevices as these remain the most common choice for elec-tronic music performances. However, the same approachcan be adapted with gestural controllers or multitouch sur-faces, provided that the manipulated sensors or widgetscan be tracked. Common sensors on MIDI controllers in-clude linear potentiometers (faders), angular potentiome-ters (knobs), and pressure / contact sensors such as but-tons and drum pads. These controllers are usually placedon a table and often tilted either towards the musicians or

�1�6�5

Musiciangestures

Digital Musical Instrument

Hardware Software

Input Device

sensor

sensor

sensor

Amplifying gestures

Mappings

mapping

mapping

Revealing impact

Sound Processes

module module

module module

Visualizing sound production

Figure 2: Structure of a Digital Musical Instrument and steps of the Rouages approach

towards the audience, therefore changing the sensors visi-bility. They are mostly used in combination with a laptopcomputer which might hide the controller. The hardwarepart of our prototype is a box made so that musicians cantilt their controller and see their laptop as they do withtheir current setup. As shown on Figure 3, the controller isplaced above the box and the laptop inside it. A screen isplaced on the front side of the box, which has an opening.This screen is used to display a virtual scene inside of thebox, containing 3D virtual parts attached to the real sensorsand 3D representations of the sound modules of the DMI.

2.2 Amplifying gestures by virtually extend-ing sensors

Figure 4: Two different controllers (seen fromabove) and the associated sets of virtual sensors ex-tensions (seen from below)

Knob

rotation

rotation + scale

Fader

translation

translation+scale

Pad - Button

translation/scale

scale

PHYSICAL

VIRTUAL

Figure 5: Physical sensors transformations andtheir amplifications by virtual sensors extensions

In the context of public performances with MIDI con-trollers, used sensors are small, distant, and often grouped,therefore hiding each other. Moreover, musicians’ gesturesare often unnoticeable. The first step of Rouages consistsin extending the physical sensors with virtual counterpartsin order to graphically amplify sensor movements and theirassociated gestures. Each sensor has a specific virtual exten-sion attached to it through the top of the box, as depictedin Figure 5. In order to be effectively associated with thesensors, these virtual extensions must be spatially mergedusing a 3D visualization matching the point of view of theaudience. To that extent, positions and orientations of the

controller, as well as positions of the screen and of the au-dience, are tracked. This approach is similar to the ideaof Collapsible Tools developed in [10]. However, in Rouagesthe virtual augmentation is used for visualization, not inter-action. We define three strategies for amplifying visibilityof sensors, especially within groups. The first is to add avirtual transformation to the physical one, as depicted inFigure 5. The second is to differentiate sensors that aregrouped and potentially occluding each other by modulat-ing their heights and colors. The third is to take advantageof screens larger than controllers to increase the size of /extend the virtual counterparts. For instance, mechanicalarms can be used to amplify the movements of virtual sen-sors. These last two strategies are demonstrated on a groupof pads on Figure 6.

Figure 6: Grid of 16 pads being played. First col-umn with undifferentiated gray pads. Second andthird columns with different colors for each pad.Fourth column with colored extended virtual pads.

2.3 Representing the audio modules/processesThe second step of our approach consists in revealing howsound is generated in DMIs. To that extent, modules ofsound processes are represented by 3D objects placed in-side the virtual box. The activity of each module can bevisualized by changing graphical parameters of the associ-ated 3D object based on audio analysis done on its soundoutput and on control data coming from the sensors.

Three issues arise with this step of our approach. Thefirst relates to the level of detail of the representation of thesound processes. While it is possible to visualize all modulesof all sound processes of the instrument, this may lead tovisual clutter, which may affect the audience understandingof the instrument. The visualization of several modules cantherefore be combined on a single 3D object. The represen-tation of their impact may then be shown by using differentgraphical parameters of the same object, as done in [2] andformalized in [6]. For example, a pitch shifter applied toan audio loop can be visualized by changing the color ofthe loop visualization when the pitch is modified. Some

�1�6�6

Figure 7: Left: A playlist module displaying the au-dio spectrum on a sequence of 3D elements. Cen-ter: Three synthesizer modules with shapes display-ing the spectrum of the produced sounds. Right:A generic audio loop module, with rotation speedmapped to the loudness of the loop.

sound modules can also be hidden because their impact onthe resulting sound can not be discriminated, for examplewhen it is too subtle or when it does not vary during theperformance.

Second, when representing whole sound processes as sin-gle 3D objects, rather than completely specifying their rep-resentation, we define behaviors for the three following cat-egories of processes: playlists, loops and synthesizers. Thesebehaviors illustrate the associated level of interruption tol-erance that these categories represent as explained in [12],from completely written and autonomous to completely pas-sive without musician actions. They also relate to the free-dom of the musicians as described in [8].

• Playlists/scores have a high level of tolerance to inter-ruption. They consist in predefined / written musicalevents that provide interesting musical results withoutany actions of the musician, and that will be manip-ulated using effects. Since they do not involve repeti-tion, but rather a complete predefined sequence, thebehavior of the associated objects is a linear non-cyclicmotion / variation. They may also originate from out-side the virtual box, to illustrate that the musicianshave limited control over them.

• Loops may also be predefined but they need more ma-nipulations in order to provide an interesting musicalresult. The behavior of the associated objects shouldbe a periodic motion / variation, in order to show thatthey are partly written. They should be completelycontained in the virtual box in contrast to Playlists.

• Synthesizers need to be activated in order to produceany sound. The musician has a complete control overthem. The behavior of the associated objects is there-fore an absence of motion, which reveals their passiv-ity.

Each of these behaviors can obviously be represented inmany different ways. The last issue for this step of ourapproach is therefore the specialization of module represen-tations. Indeed, representations should be chosen so thatthey give enough information on what the sound modulesdo, in particular to ensure that members of the audiencecan deduce what is generated by each represented module,without overloading the instrument visualization. The in-formation may be extracted only from the modules audiooutput. It may also be combined with sensors values, espe-cially for sound parameters that are harder to analyze.

Three examples of representations are depicted on Fig-ure 7. In the case of synthesizers, the passive behavior iscombined with a fast loudness analysis and a slower Barkbands spectrum analysis in order to give a different shapeand activity to each representation. Bark bands amplitudescontrol the diameters of piled cylinders from the lowest fre-quency for the bottom one to the highest frequency for thetop one. The signal loudness gives the overall scale of theshape. Therefore, when synthesizers are not played, theirshapes indicate which sounds they represent, e.g. bass orcymbals, allowing member of the audience to identify eachof them. When played, variations of their scales followingthe loudness provide additional information.

Loops in our example have a less detailed representation.Their cyclic behavior and their activity are displayed at thesame time by mapping the loudness of the sound to thespeed of the rotation. Therefore, when the loop is eitherstopped or silent, the rotation stops. With more advancedaudio analysis, such as repetition detection, the shape couldalso be used to display the entire waveform of the loop asits contours, and the speed could be mapped to the actualdetected speed of the loop.

Finally, for the playlist behavior we use the metaphorof the barrel organ. A chain of 3D elements comes out ofa pipe at a speed that is mapped to the sound loudness.The playlist therefore originates from outside the box andis revealed as it plays. The sound output is analyzed and thespectrum and loudness are used to define the shape of theupcoming element, therefore mimicking physical punchedcards.

2.4 Revealing the impact of gestures on thesound

Figure 8: Top: Merging the links from two padsto a synthesizer. Bottom: Splitting the link from afader to five synthesizers.

The purpose of the third step is to help members of theaudience understand how exactly musicians’ gestures im-

�1�6�7

pact the resulting sound, i.e. which modules they manipu-late with each sensor. In other words, this step reveals themappings used in the DMI. To that extent, virtual sensorextensions must be connected to the sound processing mod-ules, to visualize the path from gestures to sound and theflows of data. In order to achieve this, 3D links are drawnbetween the virtual sensors and the 3D representations ofthe modules each time the physical sensors are manipulatedto control parameters of the modules, i.e. each time themusician influences the instrument. They are not displayedpermanently so that the virtual box is not visually over-loaded. As depicted on Figure 8, links can be merged orsplit to simplify the visualization of one-to-many mappings[7], e.g. with one sensor controlling several sound modules.Finally, the links slowly fade away once a manipulation isover so that simultaneous gestures can be comprehended.

3. USING ROUAGESWhen setting up Rouages for their own digital musical in-strument, musicians need to follow three main steps.

3.1 Setting up the 3D renderingRouages allows musicians to precisely define several param-eters that will be used to correctly render the virtual spaceso that it matches the physical space. The 3D renderingdone using the Irrlicht5 library is then adjusted by modify-ing the frustum of the virtual camera. First of all, screen di-mensions and box dimensions, position and orientation needto be defined, together with the user head position that willdefine the sweet spot for a group of spectator. A spectatorplaced on that sweet spot will perceive a perfect alignmentbetween the virtual and physical spaces, and so will spec-tators behind the spot within an angle, as described foranother mixed-reality performance system in [17]. Displaydevices allowing for a other viewing angle are also beinginvestigated as explained in section 5. In addition, stere-oscopy can be enabled in order to improve the perceptionof the depth of 3D components. Furthermore, head positionand box orientation can be acquired by connecting to a Vir-tual Reality Peripheral Network (VRPN) 6 server that maysend tracking data from a large variety of 3 and 6 degrees offreedom tracking systems. The 3D view is then be modifiedaccordingly dynamically.

3.2 Defining sensors extensions, sound mod-ules and links

The second step consists in defining the modules that willrepresent the components of the DMI and connecting themto their associated visual components. Virtual sensor ex-tensions can be created and placed individually but groupsof sensors are also predefined for various commercial MIDIcontrollers. For example, Figure 4 shows sensors groupsfor the M-Audio Trigger Finger with 16 pads, 8 knobs and4 faders, and for the Berhinger BCF2000 with 8 faders, 8knobs and 16 buttons. Sensors can then be connected tosoftware or hardware MIDI ports using the JACK AudioConnection Kit (JACK) multi-platform audio server 7, andassociated with one or several MIDI messages. 3D represen-tations of sound modules of the selected types (e.g. loop,synthesizer, playlist) may then be added inside the virtualbox and connected to the modules. Once again, this isdone with the JACK server, by creating audio input portsfor each 3D object and connecting these ports to the cor-responding audio output ports of the sound modules, and

5http://irrlicht.sourceforge.net/6http://www.cs.unc.edu/Research/vrpn/7http://jackaudio.org

MIDI ports of the sensors. Finally, links between virtualsensor extensions and sound modules representations aredefined to match the actual mappings of the DMI.

3.3 Choosing a skinIn addition to revealing musicians’ impact and sometimesvirtuosity, Rouages may also be used to illustrate the moodof an entire performance or of a particular track. Severalskins can therefore be defined for each instrument, chang-ing the appearance and animation of sensors extensions,representations of sound modules, links and box. Skinsmay follow our recommended behaviors for specific typesof modules but they may also freely modify any parameteraccording to MIDI and audio input. For example, modifica-tions of parameters such as color, translation, rotation, orany combinations of these parameters may be mapped toloudness variations. In addition, 3D animated models maybe loaded. The animation is then mapped to value of thecorresponding MIDI input or the loudness of the associatedaudio input, allowing for advanced 3D shape transforma-tions. The third possibility is to use 2D animations in formof a set of PNG files, which are sequentially read as textureson 3D planes. Skins may be set for each 3D representationindependently or for all representations of a type, e.g. allknobs, all loops. Two example skins for the same instru-ment are depicted on Figure 9.

Figure 9: Skin examples. Top: 3D objects withshape/orientation animations, Bottom: 2D Anima-tions

4. FEEDBACKInformal feedback on our approach was obtained by sendinga link to a video8 of a performance with a DMI revealed byRouages on private mailing-lists of the University of Bor-deaux. The specific DMI is depicted on Figure 1. It is com-posed of three sound processes, two with one sound module(audio loop) and one with 5 modules (granular synthesiz-ers). Additional audio effects modules such as compressors

8https://vimeo.com/63147015

�1�6�8

are not displayed in order to simplify the representation, astheir effect on the overall sound is constant. Each of thetwo loops can be started and stopped with two pads andthe triggering offset can be controlled with a knob. The 5synthesizer modules are played using various groups of padsand their parameters are all controlled with the same fadersand knobs. We did not use a questionnaire but rather gath-ered the spontaneous comments of 10 people about the per-formance. Responses to the video ranged from positive tovery enthusiastic, and they brought interesting comments.For example, one spectator (non-musician) said that:”It is very meaningful. Drawing the links between the but-tons and 3D objects is a big improvement for understanding.I guess that gears turn to show that a piece of music is be-ing played, and that the pits allow for the activation of aparticular sound”.This example illustrates that the spectator could under-stand both the structure of the instrument and the impactof musicians’ gestures, thanks to the visualization providedby Rouages. A first suggested improvement was to providea more detailed representation of loops, by displaying theirwaveforms or the played notes.. This comment relates tothe different visualization possibilities that we describe insection 2.3, and will require more advanced audio analysissuch as repetitions detection. Another recurrent suggestionis to expand the 3D visualization outside the box for a morespectacular effect. Although this first feedback is positive, aproper evaluation with both subjective and objective mea-surements needs to be conducted as explained in section5.

5. CONCLUSION AND PERSPECTIVESRouages aims at improving the audience experience dur-ing electronic music performances by revealing the innermechanisms of digital musical instruments. This is done byfirst amplifying musicians’ gestures using virtual extensionsof sensors, then describing how the sound is produced us-ing 3D representations of the sound processes and finallyshowing the impact of musicians’ gestures on these pro-cesses with dynamic graphical connections. Our approachis implemented for DMIs based on MIDI controllers with amixed-reality display system and a software application.

Four main perspectives arise from our approach. Thefirst pertains to the hardware side of our system. In orderto allow for several correct viewing angles and to extend ourcurrent system to gestural controllers, technologies such aslarge multi-angle semi-transparent screens, or other multi-ple views display technologies will be evaluated. The secondperspective relates to the classification and representationof sound processes and modules. Contrary to other clas-sification approaches that define general properties [3] ormathematical representations [4] of DMIs, we need to de-fine a taxonomy of commonly used sound modules. Thistaxonomy will range from generic modules to specializedones, i.e. types of audio effects or synthesis techniques, inorder to extend the defined behaviors for 3D representa-tions and define adapted sound analysis techniques. Thethird perspective is the evaluation of the impact of the dif-ferent parameters of our approach on the understandingof DMIs and more generally on the experience of the au-dience. These parameters are the representation level-of-detail, the physical-virtual collocation and the visualizationof gestures-sound links. The final perspective is the exten-sion to laptop orchestras. In this case, Rouages may en-hance the understanding of the activity of each musician inthe orchestra.Rouages opens up new perspectives for improving audi-

ence experience with digital musical instruments. We hopethat instrument makers and musicians will lean on this ap-proach in their work. We also believe that Rouages couldeasily be transposed to other contexts such as artistic in-stallations that involve digital interfaces.

6. REFERENCES[1] N. Armstrong. An Enactive Approach to Digital Musical

Instrument Design. PhD thesis, Princeton University, 2006.

[2] F. Berthaut, M. Desainte-Catherine, and M. Hachet.Combining audiovisual mappings for 3d musicalinteraction. In Proceedings of the International ComputerMusic Conference (ICMC10), pages 357–364, New York,USA, 2010.

[3] D. Birnbaum, R. Fiebrink, J. Malloch, and M. M.Wanderley. Towards a dimension space for musical devices.In NIME ’05: Proceedings of the 2005 conference on Newinterfaces for musical expression, pages 192–195,Singapore, 2005. National University of Singapore.

[4] A. Bonardi and J. Barthelemy. The preservation,emulation, migration, and virtualization of live electronicsfor performing arts: An overview of musical and technicalissues. J. Comput. Cult. Herit., 1(1):6:1–6:16, June 2008.

[5] K. Cascone. Laptop music - counterfeiting aura in the ageof infinite reproduction. Parachute, issue 107, 2002.

[6] C. G. Healey. Building a perceptual visualisationarchitecture. Behaviour & Information Technology,19:349–366, 2000.

[7] A. Hunt and R. Kirk. Mapping strategies for musicalperformance. Trends in Gestural Control of Music, pages231–258, 2000.

[8] S. Jorda. Interactive music systems for everyone: exploringvisual feedback as a way for creating more intuitive,efficient and learnable instruments. In Proceedings of theStockholm Music Acoustics Conference (SMAC03),Stockholm, Sweden, 2003.

[9] S. Jorda, M. Kaltenbrunner, G. Geiger, and R. Bencina.The reactable*. In Proceedings of the InternationalComputer Music Conference, 2005.

[10] J. Lee and H. Ishii. Beyond: collapsible tools and gesturesfor computational design. In Proceedings of the 28th of theinternational conference extended abstracts on Humanfactors in computing systems, CHI EA ’10, pages3931–3936, New York, NY, USA, 2010. ACM.

[11] G. Levin and Z. Lieberman. Sounds from shapes:audiovisual performance with hand silhouette contours inthe manual input sessions. In Proceedings of the 2005conference on New interfaces for musical expression,NIME ’05, pages 115–120, Singapore, Singapore, 2005.National University of Singapore.

[12] J. Malloch, D. Birnbaum, E. Sinyor, and M. M. Wanderley.Towards a new conceptual framework for digital musicalinstruments. In Proceedings of the 9th internationalconference on digital audio effects (DAFX-06), pages49–52, 2006.

[13] M. Marshall, P. Bennett, M. Fraser, and S. Subramanian.Emotional response as a measure of liveness in new musicalinstrument performance. In CHI 2012 Workshop onExploring HCI Relationship with Liveness, May 2012.

[14] S. Reeves, S. Benford, C. O’Malley, and M. Fraser.Designing the spectator experience. In Proceedings of theSIGCHI conference on Human factors in computingsystems, CHI ’05, pages 741–750, New York, NY, USA,2005. ACM.

[15] M. M. Wanderley and B. Vines. Music and Gesture,chapter Origins and functions of clarinettist’s AncillaryGestures, page p.167. Ashgate, 2006.

[16] M. Zadel and G. Scavone. Different strokes: a prototypesoftware system for laptop performance and improvisation.In Proceedings of the 2006 conference on New interfacesfor musical expression, NIME ’06, pages 168–171, Paris,France, France, 2006.

[17] V. Zappi, D. Mazzanti, A. Brogni, and D. Caldwell. Designand evaluation of a hybrid reality performance. In NIME’11: Proceedings of the 2011 conference on New interfacesfor musical expression, pages 355–360, Oslo, Norway, 2011.

�1�6�9