Digital Digital LutherieLutherie
Crafting musical computers Crafting musical computers for new for new musicsmusics’’ performance performance
and improvisationand improvisation
Sergi Jordà
Music Technology GroupInstitut Universitari de l’AudiovisualUniversitat Pompeu Fabra, Barcelona
Barcelona, June 2005
11/04/2007 Digital Lutherie - Sergi Jordà 2
When asked what musical instrument they play, few computer musicians respond spontaneously with ’’I play the computer.’’Why not? (Wessel & Wright 2002).
11/04/2007 Digital Lutherie - Sergi Jordà 3
As a performer and composer, my goal when constructing the instruments I will play is clear. I need instruments which will allow me to enjoy playing, particularly with other musicians, which will hopefully allow these other musicians to enjoy playing with me, and which will allow me to create music that will surprise me as much as possible, that will keep showing me little hidden secrets at every new listening, that I will therefore enjoy listening back.
Music not necessarily better, not worst either, than the one I could compose on my studio; but music, in essence, that could not have been created in any other possible way.
11/04/2007 Digital Lutherie - Sergi Jordà 4
Musical interface construction proceeds as more art than science, and possibly this is the only way it can be done (Cook 2001).
11/04/2007 Digital Lutherie - Sergi Jordà 5
INTRODUCTIONINTRODUCTION
11/04/2007 Digital Lutherie - Sergi Jordà 6
New digital instrumentsNew digital instruments’’ designdesign……… is a broad field, which
encompasses highly technological areas (e.g. electronics and sensor technology, sound synthesis and processing techniques, software engineering, etc.)
11/04/2007 Digital Lutherie - Sergi Jordà 7
New digital instrumentsNew digital instruments’’ designdesign……… is a broad field, which
encompasses highly technological areas (e.g. electronics and sensor technology, sound synthesis and processing techniques, software engineering, etc.)
and disciplines related to the study of human behavior (e.g. psychology, physiology, ergonomics and human-computer interaction components, etc.)
11/04/2007 Digital Lutherie - Sergi Jordà 8
New digital instrumentsNew digital instruments’’ designdesign……… is a broad field, which
encompasses highly technological areas (e.g. electronics and sensor technology, sound synthesis and processing techniques, software engineering, etc.)
and disciplines related to the study of human behavior (e.g. psychology, physiology, ergonomics and human-computer interaction components, etc.)
Trying to solve separate parts of the problem is essentialessential…
11/04/2007 Digital Lutherie - Sergi Jordà 9
New digital instrumentsNew digital instruments’’ designdesign……… is a broad field, which
encompasses highly technological areas (e.g. electronics and sensor technology, sound synthesis and processing techniques, software engineering, etc.)
and disciplines related to the study of human behavior (e.g. psychology, physiology, ergonomics and human-computer interaction components, etc.)
Trying to solve separate parts of the problem is essential……but also clearly insufficient.
11/04/2007 Digital Lutherie - Sergi Jordà 10
Two axioms constitute the starting point of this thesis:
New instruments for new New instruments for new musicsmusics
11/04/2007 Digital Lutherie - Sergi Jordà 11
Two axioms constitute the starting point of this thesis:
• New musics tend to be the result of new techniques, which can be both compositional or instrumental.
• New instruments will be able to survive and succeed in the measure they are really innovative; i.e. they have something new to bring to music, and not merely because they are based on novel, previously unavailable technologies.
New instruments for new New instruments for new musicsmusics
11/04/2007 Digital Lutherie - Sergi Jordà 12
Premises & MethodologiesPremises & MethodologiesNew musical instruments based on computers can only be
improved by exploring three parallel paths
11/04/2007 Digital Lutherie - Sergi Jordà 13
Premises & MethodologiesPremises & MethodologiesNew musical instruments based on computers can only be
improved by exploring three parallel paths
1) identifying the quintessence of new digital instruments; what they can bring of really original to the act of music performance; how can they redefine it;
11/04/2007 Digital Lutherie - Sergi Jordà 14
Premises & MethodologiesPremises & MethodologiesNew musical instruments based on computers can only be
improved by exploring three parallel paths
1) identifying the quintessence of new digital instruments; what they can bring of really original to the act of music performance; how can they redefine it;
2) identifying the drawbacks or obsolescences of traditional instruments; what limitations or problems could be eliminated, improved or solved;
11/04/2007 Digital Lutherie - Sergi Jordà 15
Premises & MethodologiesPremises & MethodologiesNew musical instruments based on computers can only be
improved by exploring three parallel paths
1) identifying the quintessence of new digital instruments; what they can bring of really original to the act of music performance; how can they redefine it;
2) identifying the drawbacks or obsolescences of traditional instruments; what limitations or problems could be eliminated, improved or solved;
3) without forgetting the essential generic assets of traditional instruments; those qualities that should never be forgotten nor discarded.
11/04/2007 Digital Lutherie - Sergi Jordà 16
Dissertation OutlineDissertation Outline
• Introduction [chap. 1]• Part I: Theory [chaps. 2-7]• Part II: Praxis [chaps. 8-10]• Part III: Teachings & Conclusions [chaps. 11-12]
• plus …§ bibliography§ discography§ appendixes (list of publications, etc.)
11/04/2007 Digital Lutherie - Sergi Jordà 17
PART IPART I• The purpose of this part is to progressively enlighten the three
fundamental aforementioned questions: a) quintessence of new digital instrumentsb) drawbacks or obsolescences of traditional instrumentsc) qualities of trad. instruments that should not be forgotten)
• Starts providing a thorough overview of the current know-hows and conceptual frameworks in which new instruments designers and researchers are currently working [chaps. 2-6]
• Ends presenting a theoretical framework for evaluating the potential, the diversity of new digital music instruments, as well as the possibilities and expressive freedom of human performers…[chap. 7]
11/04/2007 Digital Lutherie - Sergi Jordà 18
PartPart I : I : CriteriaCriteria & & PurposesPurposes
• Overview of contemporary thinking, bodies of knowledge and concepts, more than of implementations
• Some real examples are however also presented and discussed§ chosen either for their historical relevance of for their
paradigmatical nature (when possible, for both)• Overview is neither neutral or aseptic, but filled with personal
ideas and criteria (critical analysis & synthesis)• Serves a dual purpose:§ useful per se (comprehensive coverage of the topic)§ introduces problems, limitations of current approaches…
11/04/2007 Digital Lutherie - Sergi Jordà 19
PartPart I : I : OrganisationOrganisation• Reflects the interdisciplinarity of the field (technology
engineering, ergonomy, cognitive science, philosophy, musicology, history...)
11/04/2007 Digital Lutherie - Sergi Jordà 20
acoustic instruments’ prelude
inputs & gestural controllers
interactive music systems (1)
interactive instruments: old & new paradigms
mapping & research in digital instruments design
towards a conceptual framework for new instruments design
11/04/2007 Digital Lutherie - Sergi Jordà 21
Introduces possibilities & limitations of acoustic instruments concerning both mechanical inputs & control, and sonic output, and how both aspects are hardwired by physical constraints
Acoustic instruments tend to impose by construction their own playability rules, which allow listeners to infer the type and form of the gesture from the generated sound
acoustic instruments’ prelude
11/04/2007 Digital Lutherie - Sergi Jordà 22
NON-HUMAN EXCITATION ENERGY
AUTOMATED CONTROL PROCESSES
Hammond+Leslie
Analog modular synthesizer
Human vs. non human (a) excitation energy and (b) process automationin acoustic and electric instruments
11/04/2007 Digital Lutherie - Sergi Jordà 23
If in acoustic instruments the separation between the control interface and the sound-generating subsystems is often fuzzy, in digital instruments, any input can be linked to any output and both components can offer a virtually infinite number of possibilities
• Sound synthesizers are not instruments!• Gestural controllers are not instruments!
acoustic instruments’ prelude
11/04/2007 Digital Lutherie - Sergi Jordà 24
inputs & gestural controllers
• Introduces controllers, responsible for extracting control information coming from the performers. These input devices constitute the first component of the digital instrument chain.
• Several types and taxonomies are presented (e.g. instrument-like controllers, extended controllers and hyperinstruments, non-imitative controllers, batons, bodysuits and exoskeletons, use of biosignals, cheap and borrowed controllers …)
• Several basic concepts such as dimensionality, resolution, sampling rate, dimension coupling, bandwidth... are also introduced.
11/04/2007 Digital Lutherie - Sergi Jordà 25
interactive music systems (1)
interactive instruments: old & new paradigms
11/04/2007 Digital Lutherie - Sergi Jordà 26
L’instrument philosophe est sensible; il est en même temps le musicien et l’instrument. Comme sensible, il a la conscience momentanée du son qu’il rend; comme l’animal il en a la mémoire. Cette faculté organique, en liant les sons en lui-même, y produit et conserve la mélodie. Supposez au clavecin de la sensibilité et de la mémoire, et dites-moi s’il ne se répétera pas de lui-même les airs que vous aurez exécutés sur ces touches. Nous sommes des instruments doués de sensibilité et de mémoire (Diderot à D’Alembert ~1770).
11/04/2007 Digital Lutherie - Sergi Jordà 27
From From ‘‘scoredscored’’ instruments instruments to to ‘‘intelligentintelligent’’ instrumentsinstruments
It …seemed to me that there were a number of aspects of music where the performer didn’t have any freedom of choice. The sequence of notes to be played, for example, was an area in which the computer could legitimately help a performer in making it easier to do things correctly (Roads 1989, interview with Max Mathews).
Intelligent Musical Instruments are instruments that first sensed its performer’s intent by the type and context of the performer’s actions and then extended the performer’s controls to execute the music automatically. Intelligent instruments let people play the music on a compositional level (Gagne 1993, interview with Laurie Spiegel).
11/04/2007 Digital Lutherie - Sergi Jordà 28
J.Cage “cartridgemusic” (1961)K.Stockhaussen(1964)
Live electronicsMEV, D.Tudor, G.Mumma, D.Buchla …
First hybrid systemsMathew’s GROOVE, J.Chadabe, S.Martirano
MicrocomputersThe League, G.Lewis, M.Waisvisz…
Early MIDIK.C.S., Music Mouse, M & Jam Factory, HMSL…
1960 19901970 1980
11/04/2007 Digital Lutherie - Sergi Jordà 29
Interactive music/instruments Interactive music/instruments definitions, metaphors, taxonomiesdefinitions, metaphors, taxonomies
• John Bischoff et al. (1978)• Joel Chadabe (1984)• George Lewis (1985)• Michel Waisvisz (1985)• Jeff Pressing (1990)• David Wessel (1991)• Miller Puckette (1991)• Laurie Spiegel (1992)• Robert Rowe (1993)• Todd Winkler (1998)• ….Concepts (not implementations) are discussed here…leading sometimes to contradictory statements…
11/04/2007 Digital Lutherie - Sergi Jordà 30
Some musical interaction Some musical interaction questions, problems or paradoxesquestions, problems or paradoxes
• Interaction (‘surprise’) vs. instrument control• Interactive or merely reactive? (e.g. score following)• Interaction and no improvisation? Is that possible?• Improvising with computers does necessarily mean
improvising with computers that try to behave like humans? § Intelligent instruments: knowledge?
Knowing/modeling what?§ Machine listening… the “only way”?
11/04/2007 Digital Lutherie - Sergi Jordà 31
Interactive or merely reactive?
11/04/2007 Digital Lutherie - Sergi Jordà 32
Some musical interaction Some musical interaction questions, problems or paradoxesquestions, problems or paradoxes
• Interaction (‘surprise’) vs. instrument control• Interactive or merely reactive? (e.g. score following)• Interaction and no improvisation? Is that possible?• Improvising with computers does necessarily mean
improvising with computers that try to behave like humans? § Intelligent instruments: knowledge?
Knowing/modeling what?§ Machine listening… the “only way”?
11/04/2007 Digital Lutherie - Sergi Jordà 33
Some musical interaction Some musical interaction questions, problems or paradoxesquestions, problems or paradoxes
• Interaction (‘surprise’) vs. instrument control• Interactive or merely reactive? (e.g. score following)• Interaction and no improvisation? Is that possible?• Improvising with computers does necessarily mean
improvising with computers that try to behave like humans? § Intelligent instruments: knowledge?
Knowing/modeling what?§ Machine listening… the “only way”?
11/04/2007 Digital Lutherie - Sergi Jordà 34
Multithreaded instruments Multithreaded instruments & shared control& shared control
• In traditional instruments the performer is responsible for controlling every smallest detail, leaving nothing to the instrument responsibility
• In new interactive music instruments, the instrument’s ‘intelligence’ may be partially responsible for one or more musical processes,the control of which may be entirely left to the instrument’s responsibility or may be shared in different ways with the performer
• When performers delegate some control to the instrument, their role is also approaching that of composers, who do have to delegate control of their work to instrumentalists
11/04/2007 Digital Lutherie - Sergi Jordà 35
11/04/2007 Digital Lutherie - Sergi Jordà 36
NON-HUMAN EXCITATION ENERGY
AUTOMATED CONTROL PROCESSES
Hammond+Leslie
Analog modular synthesizer
11/04/2007 Digital Lutherie - Sergi Jordà 37
Some multithread & Some multithread & shared control corollariesshared control corollaries
These systems (instruments) can • surpass the one gesture to one acoustic event’ paradigm• go beyond the sound and note control level• current systems (i.e. after MIDI) allow in fact to bridge a
continuum between micro (i.e. timbre) & macrostructure, micro & macrocontrol (e.g. soundprocessingà machine listening continuum)
• run multiple and parallel musical processes in a shared control between the instrument and the performer è the possibility to have multiple performers seems as a logical and promising extension
11/04/2007 Digital Lutherie - Sergi Jordà 38
11/04/2007 Digital Lutherie - Sergi Jordà 39
mapping & research in digital instruments design
11/04/2007 Digital Lutherie - Sergi Jordà 40
• The physical and logical separation of the input device from the sound production, brings the necessity to process and map the information coming from the input device in a variety of ways
• Mapping, studies the connection between gesturalparameters (or any other kind on input) and sound and music control parameters (or any other output results). It is an essential element in new instruments design
• This field is probably the closest that digital lutherieever gets to science; one of the few areas in which knowledge seems to behave in an incremental, accumulative way
MappingMapping
11/04/2007 Digital Lutherie - Sergi Jordà 41
On ergonomics, motor & cognitive loads• Fitts (1954), Miller (1956)• Pressing (1988)• Rubine & McAvinney (1990)• Vertegaal & Eaglestone (1996) …On Mapping• Pressing (1990)• Lee & Wessel (1992)• Wanderley, Schnell, Rovan, Depalle, Hunt, Kirk…(1996-200x)• Arfib (2002)• Goudeseune (2002) …On haptic feedback• Cadoz et al. (1984, 2003)…• Bongers (1994)• Gillespie (1999) …
MainMain theorethicaltheorethical contentscontents
11/04/2007 Digital Lutherie - Sergi Jordà 42
The unfulfilled expectations of The unfulfilled expectations of mappingmapping
• Current mapping tests rely on simple systems: simple inputs devices (or imitative controllers) and simple output control, (usually pitch, amplitude and one timbre parameter)
• No attempts to affront mapping in a context different than a sound synthesizer
• Few real musical implementations as a result of studies
• Hard to evaluate the music performance of new unknown musics (Music is not ‘simple’ HCI - what’s the goal? what are the tasks? )
11/04/2007 Digital Lutherie - Sergi Jordà 43
Problems? Solutions?Problems? Solutions?• Low-level and focused research that tries to solve
independent parts of the problem is clearly essential for any real progression in this field, but it is also clearly insufficient..
• Crafting new instruments has to be more than designing hardware controllers, studying mappings and gluing components
• Integrated designs are needed• Holistic approaches are needed (…psychology,
philosophy… & MUSIC!!)• How do we study, compare or evaluate infinitely diverse
instruments that make musics that do still not exist?
11/04/2007 Digital Lutherie - Sergi Jordà 44
REM: Premises & MethodologiesREM: Premises & MethodologiesNew musical instruments based on computers can only be
improved by exploring three parallel paths
1) identifying the quintessence of new digital instruments; what they can bring of really original to the act of music performance; how can they redefine it;
2) identifying the drawbacks or obsolescences of traditional instruments; what limitations or problems could be eliminated, improved or solved;
3) without forgetting the essential generic assets of traditional instruments; those qualities that should never be forgotten nor discarded.
11/04/2007 Digital Lutherie - Sergi Jordà 45
The reality: The reality: new instruments possibilities are endless new instruments possibilities are endless
…… anything can be done, but anything can be done, but ……
• New virtuosi and/or professional musicians who use them as their main instrument is surprisingly small
• If live electronics and laptop music is so widespread why so many performers prefer to rely on the mouse or on generic and dull midi fader boxes ?
• Commercially available new instruments are scarce and hardly imaginative and ground-breaking (e.g KAOSS)
• A new standard electronic instrument is yet to arrive (?)• Technology evolves too fast for it ?
11/04/2007 Digital Lutherie - Sergi Jordà 46
in shortin short……• We need useful, playable, thought-provoking enjoyable
instruments, capable of interesting, surprising, enjoyable music
• What is a good music instrument, anyway?• Myriad of criteria, subjective, changeable over history…
• Before we find the hidden, implicit properties that make traditional instruments good enough (!), we will try to find in what they could get better …
• … and we will try to do it, completely independently of the types of music they can play …
11/04/2007 Digital Lutherie - Sergi Jordà 47
Most traditional instruments take a long time to master
We should be able to design well balanced instruments that can appeal to both professionals and dilettanti; instruments that like the piano, can offer a low entry fee with no ceiling on virtuosity (Wessel 2001)
How do we create controls and interactions that feel inevitable to expert and amateur users? (Machover, 2002)
11/04/2007 Digital Lutherie - Sergi Jordà 48
Some inspiration sourcesSome inspiration sources……
the writersthe writers……
•• Pressing (1990)Pressing (1990)•• RubineRubine & & McAvinneyMcAvinney
(1990)(1990)•• Ryan (1991)Ryan (1991)•• ChadabeChadabe (1997)(1997)•• WaisviszWaisvisz (2000)(2000)•• Cook (2001)Cook (2001)•• WesselWessel & Wright (2001)& Wright (2001)•• Robson (2001)Robson (2001)•• SettelSettel and and LippeLippe (2003)(2003)•• Blaine and Blaine and FelsFels (2003)(2003)• ….
the improvisers the improvisers ……
•• Nicolas CollinsNicolas Collins•• Michael WaisviszMichael Waisvisz•• Tim PerkisTim Perkis
……..
and above all,and above all,all the all the unpluggedunplugged and/or and/or
analogue ones !!!analogue ones !!!
11/04/2007 Digital Lutherie - Sergi Jordà 49
• Jordà, S. (2002). Improvising with Computers: A personal Survey (1989-2001). Journal of New Music Research, 31(1), 1-10.
• Jordà, S. (2004). Digital Instruments and Players: Part I – Efficiency and Apprenticeship. Proceedings of the 2004 Conference on New Interfaces for Musical Expression (NIME-04), 70-76. Hamamatsu.
• Jordà, S. (2004). Digital Instruments and Players: Part II–Diversity, Freedom and Control. Proceedings of the 2004 International Computer Music Conference. San Francisco, CA: International Computer Music Association.
• Jordà, S. (2005). Instruments and Players: Some thoughts on digital lutherie. Journal of New Music Research, 33(3),1-21.
11/04/2007 Digital Lutherie - Sergi Jordà 50
Covered topicsCovered topics
• Learning curve [of a music instrument]• Music efficiency …• Music diversity …• Diversity, improvisation, configurability & identity• Virtuosity, non-linearity vs. control, predictability,
confidence, explorability…• Time considerations• Expressiveness
11/04/2007 Digital Lutherie - Sergi Jordà 51
Musical efficiency & Musical complexitiesMusical efficiency & Musical complexities
Output complexity should include• the micro-sonic richness of the violin• the mid-sonic richness of the piano• the potential macro-structural complexities that could be managed
by any kind of “intelligent” digital instrumentInput complexity will depend on• degrees of freedom of control• correlation of controls• range and resolution• mappings• ergonomics…
InputOutput
Efficiency = EffortessEffectiven
iencyHumanEffic =
tyutComplexiControlInpityputComplexMusicalOut
iencyumentEfficMusicInstr =
11/04/2007 Digital Lutherie - Sergi Jordà 52
Out
put C
ompl
exity
Input Complexity
tyutComplexiMusicalInpityputComplexMusicalOut
Efficiency =
?
11/04/2007 Digital Lutherie - Sergi Jordà 53
Out
put C
ompl
exity
Input Complexity
tyutComplexiMusicalInpityputComplexMusicalOut
Efficiency =
?
!!!
11/04/2007 Digital Lutherie - Sergi Jordà 54
Playing [with] music ?Playing [with] music ?
In ancient Greece, the kithara was usually played by In ancient Greece, the kithara was usually played by professional musicians as it required some kind of professional musicians as it required some kind of exhibitionism and virtuosity that could be considered vulgar exhibitionism and virtuosity that could be considered vulgar and not adequate for freeand not adequate for free--born men. The born men. The lyralyra instead, was instead, was commonly played by amateurs, and was, according to Plato, a commonly played by amateurs, and was, according to Plato, a well suited instrument for free citizen. well suited instrument for free citizen.
Before the advent of the radio and recorded music, families Before the advent of the radio and recorded music, families who could afford it, tended to have an instrument at home who could afford it, tended to have an instrument at home (often a piano) and some of the family members were enough (often a piano) and some of the family members were enough talented to become talented to become ‘‘animated interactive CD playersanimated interactive CD players’’ every every evening.evening.
11/04/2007 Digital Lutherie - Sergi Jordà 55
The art of making music made easy (?)
It's true, making music is an Art, traditionally demanding years of study and practice to be successful. -- until now! Enter the new Suzuki Omnicord. If you can read this text, you can play the Omnicord, and play it well. There's no need for lessons or years of study to play and sing your favorite songs right now! Just press a chord button and strum the SonicStrings. It's that easy.
11/04/2007 Digital Lutherie - Sergi Jordà 56
good instruments make bad music!• A good instrument should not impose its music to its
player.
• A good instrument should not be able to produce only good music! (What is good music anyway?)
• A good instrument should also be able to produce “terribly bad” music, either at the player’s will or at the player’s misuse.
11/04/2007 Digital Lutherie - Sergi Jordà 57
Musical diversity & performer freedom:Musical diversity & performer freedom:a departure from the a departure from the efficiencyefficiency engineering engineering
viewpointviewpoint
The more ‘intelligent’ an instrument becomes, when complex processes are not anymore directly controlled by the user but possibly by other processes, the less meaningful a strict structural or informational dimensionality turns out to be, and only a perceptual musical complexity acquires sense
tyutComplexiControlInpersityMusicalDivytComplexitMusicOutpu
EfficMusicInstr×
=
11/04/2007 Digital Lutherie - Sergi Jordà 58
On diversityOn diversity……
• Vertegaal et al. (1996)• Blaine & Fels (2003)• Nishimoto et al. (2003)• Settel & Lippe (2003)• Arfib et al. (2005) …
discuss in different ways “instrument versatility or flexibility”
11/04/2007 Digital Lutherie - Sergi Jordà 59
Our diversityOur diversity
•• microdiversitymicrodiversity•• mididiversitymididiversity•• macrodiversitymacrodiversity
Some considerations …• They do not represent different time scales• Mididiversity nothing to do with MIDI!• Related to “music”, not to sound è not to
psychoacoustical perception• A continuum could be more realistic, but also less
clarifying
11/04/2007 Digital Lutherie - Sergi Jordà 60
MicrodiversityMicrodiversity ((MicDMicD) : Performance nuances) : Performance nuances• Subtle nuances give a measure of how much a performer
can turn a piece her ownèMicD measures how much two performances of the
same piece can differ• All professional traditional instruments offer many
possible nuances and subtleties• For acoustic instruments MicD is clearly linked to the
richness and the fine control of sonic details --psychoacoustics and just noticeable differences (Rubine& McAvinney 1990)
• In digital instruments, also related to any kind of fine structural variation (time, tempo, number of voices, density of the voices and relations between them) that could occur within a given piece, while keeping it recognizable
11/04/2007 Digital Lutherie - Sergi Jordà 61
The PhonographThe Phonographa a MicDMicD case studycase study
• Cage’s Imaginary Landscape No.1 (1939) includesvariable-speed phonograph turntables, playing test-tones
With a … phonograph it is now possible to control …any of these sounds and give to it rhythms within or beyond the reach of imagination. Given … four phonographs we can compose and perform a quartet for explosive motor, wind, heartbeat and landslide. (Cage 1937)
• Kool Herc, DJ Grand Wizard Theodore… late 1970s
11/04/2007 Digital Lutherie - Sergi Jordà 62
MididiversityMididiversity ((MidDMidD) : Performance contrasts) : Performance contrasts
• How distinct two different pieces or two performances played with the same instrument can be?
• very low MidD instruments seem to be “always playing the same piece”
• New instruments for novices & interactive sound installations tend to have very low MidD
• Open or interactive compositionsA highly sophisticated ‘instrument’ with a low MidDmay be a very good interactive composition, but should not be considered as an instrument, even if it comes bundled with a hardware controller
11/04/2007 Digital Lutherie - Sergi Jordà 63
MacrodiversityMacrodiversity ((MacDMacD) : Stylistic flexibility) : Stylistic flexibility
• The ability to be played in different contexts or music styles, or assuming varied musical roles
• The capabilities of integration with different instrumental partners
• Instruments with a high MacD could be considered all-purpose, while a low MacD denotes a more specialized and less versatile instrument
• In a traditional music context, MacD does not seem to be necessarily a quality. Generic instruments that do fairly well in completely different musical contexts will probably hardly excel in any of them
• In our context, new instruments with higher MacD may be more sexy & affordable è easier “proselytism”
11/04/2007 Digital Lutherie - Sergi Jordà 64
•• MicroMicro--diversitydiversity (performance nuances)(performance nuances)èè turning musicians into virtuositurning musicians into virtuosi
•• MidMid--diversitydiversity (performance contrasts) (performance contrasts) èè turning gamers into musiciansturning gamers into musicians
•• MacroMacro--diversitydiversity (instrument flexibility, stylistic (instrument flexibility, stylistic diversity) diversity) èè catching candidatescatching candidates
Diversities summaryDiversities summary
11/04/2007 Digital Lutherie - Sergi Jordà 65
Low MacD instruments have more chances to give birth to new music
stylesHigh MacD instruments have more chances for
starting being used
Elastic and translation properties of the instrument
hyperspace when submitted to external influences (other
instruments, other music styles, etc.)
‘Specialized’instruments (e.g.
double bass, piccolo)Roland TB303 bassline
‘All-purpose’instruments (e.g.
guitar)Laptop+mouse+
Max/Msp
Stilistic flexibility (how the instrument adapts to different
contexts)
Macrodiversity
Can turn music gamers into music performers
Shape and measurements of the instrument hyperspace
Jaws harpPercussion and
instruments with fixed pitch
Interactive sound installationsInteractive
compositions
Most traditional instruments.
Performances’contrasts (how
distinct two pieces played with the
same instrument can be)
Mid-diversity
Can turn music perfomers into virtuosi
Hypervolume around one hyperpoint (representing the
piece)
Player piano“John Cage turntables”
Most traditional instruments
“HipHopturntables”
Performance nuances (how two
performances of the same piece can
differ)
Microdiversity
Influence on performer
Spatial metaphorLowHighDescription
Diversities summaryDiversities summary
11/04/2007 Digital Lutherie - Sergi Jordà 66
Time, musical sharpness & inertiaTime, musical sharpness & inertia
• a perceptual shorter level ranging somewhere around 2-10 msec (relates to technical constraints latency, resolution…)
tMicD∆
∆t
MidD∆
∆t
MacD∆
∆
100 msec (Askenfeld & Jansson 1990)delay on piano (piano)
30 msec (Askenfeld & Jansson 1990delay on piano (staccato-forte)
30-50 msec (Rasch 1979)playing “together”
20-30 (Posse Lago & Kon 2004)tolerable instrumental latency
20 msec (Hirsch 1959)separate 2 musical tones
10-20 msec (Wessel & Wright 2002)tolerable control granularity
10 msec (Gurevich et al. 2004)preferred delay between musicians
4 msec (Vorberg & Hamburg 1978)∆ in keeping steady pulse
2 mseccontrol granularity (100 samples window)
2 msec (Clynes 1984; Stewart 1987)distinguish variation in duration
2 msec (Licklinder 1951; Woodrow 1951)
separate 2 clicks
20 µsec1 sample (at 50 KHz)
20 µsec (Pierce 1999)binaural azimuth
11/04/2007 Digital Lutherie - Sergi Jordà 67
Time, musical sharpness & inertiaTime, musical sharpness & inertia
• Reacting to music (50 msec-500 msec)Sternberg et al. 1978; Lennie 1981; Pressing 1988…
• The more “intelligent”, the more inertial music systems tend to be, but maximum contrasts should be in the 250-1000 msec range
• Musical sharpness (e.g. Carl Stalling or John Zorn vs. laptop music)
tMicD∆
∆t
MidD∆
∆t
MacD∆
∆
11/04/2007 Digital Lutherie - Sergi Jordà 68
Multithreaded instruments & diversityMultithreaded instruments & diversity• Direct access to all final parameters will favor MicD• But accuracy should not happen at the expenses of leaving
unattended the macrocontrol processes
• MidD & MacD in digital instruments are often related to the concept of presets
11/04/2007 Digital Lutherie - Sergi Jordà 69
DiversityDiversity, , improvisationimprovisation, , virtuosityvirtuosity ……
• Configurations & presets vs. free improvisation and continuum performance space (any instrument must permit improvisation!)
• MacD vs. instrument idiomatism• Non-linearity, control, reproducibility and virtuosity (the
goldsmith vs. the torero)• Non-linearity should not interfere exploration,
learnability (confidence is essential for starting exploration)
• Technical motor virtuosity vs. Music sensibility virtuosity (e.g. Monk, Coleman, Ayler)
• Lack of repertoire, non-linearity, exploration in new digital instruments favor 2nd approach
11/04/2007 Digital Lutherie - Sergi Jordà 70
ExpressivenessExpressiveness• Expressive Computer Music [rendering] ?§ Expressiveness as (imitation of ) deviation (e.g.
Sundberg)
We regard expression as a rhetorically shaped transfer of structural score contents by means of the ‘‘deformation’’ mapping of symbolic data into a physical parameter space (Müller & Mazzola 2003)
• Expressive instruments? è fine, grained, accurate control
• Eurocentric view è expressiveness ~ MicD
11/04/2007 Digital Lutherie - Sergi Jordà 71
ExpressivenessExpressiveness• Expressive Computer Music [rendering] ?§ Expressiveness as (imitation of ) deviation (e.g.
Sundberg)
We regard expression as a rhetorically shaped transfer of structural score contents by means of the ‘‘deformation’’ mapping of symbolic data into a physical parameter space (Müller & Mazzola 2003)
• Expressiveness as effort (Ryan 1991)• Expressiveness as a shared idiom (Fels et. al 2002)• Expressiveness as flexibility (Arfib 2005) (~ MacD)
11/04/2007 Digital Lutherie - Sergi Jordà 72
Instruments, as machines, can be more or less efficient. They cannot be expressive though, machines do not have anything to express. But they can be used to transmit human expressiveness. Our instruments will achieve this for better or worse, in the measure they permit the performer to transform a mental musical representation into musical gesture and sound. Efficiency and expressiveness do not always go hand by hand. Take the pencil, the pen and the word processor. The latter is surely more efficient, but I personally find the fountain pen’s smooth and continuous ink flow over the paper more inspiring and better suited for many expressive purposes.Inspiration. Music instruments are not only in charge of transmitting human expressiveness like passive channels. They are, with theirfeedback, responsible for provoking and instigating on the performer new ideas or feelings to express.
11/04/2007 Digital Lutherie - Sergi Jordà 73
First conclusionsFirst conclusions
11/04/2007 Digital Lutherie - Sergi Jordà 74
1. New musics tend to be the result of new techniques, which can be both compositional or instrumental.
2. New instruments will be able to survive and succeed in the measure they are truly innovative; i.e. they have something new to bring to music, and not merely because they are based on novel, previously unavailable technologies.
3. It is therefore crucial to identify the quintessence of new digital instruments; what they can bring that is essentially original to the act of music performance; how can they redefine it.
4. Moreover, new instruments will manage to be better if luthiers do not over look the history of traditional instruments.
5. Learning from the past does not merely mean emulating it. It also means to try avoiding or improving upon passed errors and limitations (the past was not perfect).
11/04/2007 Digital Lutherie - Sergi Jordà 75
6. Efficiency can be clearly improved is that of. Traditional instruments take years to master, whereas, given the speed at which technology and fashion shift, new instruments have to hook from the first minute.
7. New instruments whose output is based on traditional paradigms, such as one-to-one control of pitch+timbre+amplitude, have a hard time competing with existing traditional instruments (can only contend on the grounds of efficiency).
8. These new ‘traditionally-modeled’ instruments with increased efficiency may appeal to non-musicians and to dilettante, but less to advanced musicians as well, who will look for new-fangled possibilities.
9. New instruments have to profit from computers’ intelligence, with their ability to remember, to make decisions on the fly, to run multiple, concurrent processes in parallel.
10. In such cases, the performer will need to share and delegate part of the control of the music being created to the instrument; such as the traditional composer delegates on the performers.
11/04/2007 Digital Lutherie - Sergi Jordà 76
11. By running processes at different temporal and formal scales and different levels of complexity, new instruments naturally surpass the one-action to one-event model inescapable in all traditional instruments. New instruments are not only sound producers, they become music producers too.
12. Performing with such powerful instruments should not mean leaving all the musical decisions to the instrument. Playing music is a ‘serious’ activity, different from ‘playing with music’ (also very important but not studied here).
13. An instrument of these characteristics will probably involve many internal dimensions or control parameters. Otherwise it will always seem to play ‘the same composition’.
14. The feeling of power and macrocontrol these instruments can bring to their performers, should not let us forget about the expressive nuances and the time precision mechanisms, which traditional instruments have always incorporated.
15. The performer must thus be able to affect all these dimensions: both the metacontrol of ongoing processes and the microcontrol of final parameters should be permitted. All of them with the maximum simplicity, flexibility and speed.
11/04/2007 Digital Lutherie - Sergi Jordà 77
PartPart II : II : CriteriaCriteria & & PurposesPurposes
• Describes (in 3 chaps.) 5 implementations I’ve developed between 1989-2001
• They illustrate different paradigms exposed in [the first chapters of] Part I
• They are the origin of the ideas exposed in [in the last chapter of] Part I
• If digital lutherie is not a science, experience & maturity are important for validating empirical & intuitive mechanism è my “credentials”
11/04/2007 Digital Lutherie - Sergi Jordà 78
PITEL machine listening & improvisation (chap 8)
The QWERTYCaster, JoAn, Epizoo, Afasiascore-based systems & multimodality (chap 9)
FMOL (chap 10)
11/04/2007 Digital Lutherie - Sergi Jordà 79
PITEL machine listening & improvisation (1989-91)
Jordà, S. (1991). A Real-Time MIDI Composer and Interactive Improviser by Means of Feedback Systems. Proceedings of the 1991 International Computer Music Conference, 463–466.
11/04/2007 Digital Lutherie - Sergi Jordà 80
Based on two-term non-linear feedback relations:
xa,i = f (xa,i-N, xb,i-k)
11/04/2007 Digital Lutherie - Sergi Jordà 81
PitelPitel’’ss drawbacksdrawbacks
• Lack of serious interface design• Human performers didn’t share the machine’s logic
• Only macro/meta control / not direct access• Diversity: performances of the “same” piece as diff. as
diff. pieces! (MicD & MidD)• All non-linear processes at symbolic (not sonic) level
è big inertia è more “conductor” than “player”paradigm è driving a truck
11/04/2007 Digital Lutherie - Sergi Jordà 82
Other topics covered in this chapterOther topics covered in this chapter……
• Machine listening• Musical knowledge• Markov chains• 1/f noise, non-linearity & feedback• Generative vs. subtractive (corrective) algorithmic music
11/04/2007 Digital Lutherie - Sergi Jordà 83
The The QWERTYCasterQWERTYCaster, , JoAnJoAn, , EpizooEpizoo, , AfasiaAfasia
• QWERTYCaster Simplicity (the “moped”)• JoAn Expresiveness• Epizoo Interaction illusion• Afasia Score-based systems
11/04/2007 Digital Lutherie - Sergi Jordà 84
11/04/2007 Digital Lutherie - Sergi Jordà 85
11/04/2007 Digital Lutherie - Sergi Jordà 86
JoAnJoAn, , ll’’homehome de de carncarnJordà & Antúnez 1992
Mimicking human expressivity has nothing to do with being
expressive !
11/04/2007 Digital Lutherie - Sergi Jordà 87
EpizooEpizooAntúnez. Jordà & Olbeter 1994
The fallacies of interaction:
• Interaction does not make you more creative or more free!
• The illusion of control
11/04/2007 Digital Lutherie - Sergi Jordà 88
• One-man interactive multimedia (& wordless) performance based on Homer’s Odyssey
• 3rd collaboration between M.Antúnez (visual artist & performer), Roland Olbeter (mechanical sculptor) and S.Jordà (music, interactivity and software design)
• Premiered in Barcelona 1998, presented in many European and American countries, and several international awards
• A robot quartet, an exoskeleton fitted with sensors, interactive visuals, films & music all controlled by the performer
Jordà, S. (2002). Afasia: the Ultimate Homeric One-man-multimedia-band. Proceedings of the 2002 Conference on New Interfaces for Musical Expression (NIME-02), 132-137. Dublin.
AfasiaAfasia ((AntAntúúneznez, , JordJordàà, , OlbeterOlbeter 1998)1998)
11/04/2007 Digital Lutherie - Sergi Jordà 89
Computer Computer
Effects racks
MIDI controlled
Audio Mixer
MIDI
MIDI
OUT
Internal PC
CD Audio
Internal PC
Sampler AudioTo parallel D/A cards
Rack mounted
D/A parallel
cards and relays
Parallel
communication
Switches to
electro-valves and
step motors
Serial
RS-232
selects
projector
input
Screen fills the whole backstage
Air Compressor
Compressed air to
electro-valves
Violin Guitar Bagpipes
S-Video
from
internal
DVD
SVGA with
2D
animations
486 with
custom card
for decoding
wireless
transmission
Pentium II
with CD
Audio,
internal
sampler,
DVD and
MIDI OUT,
running
Afasia
Software
Projector
Drum
Performer with
exoskeleton
sends wireless
data to 486
4 Turn
potenciometers
continous
controllers
4 Mercury
tilt sensors
“Keys”
Video
Input
switcher
PedalsAux
11/04/2007 Digital Lutherie - Sergi Jordà 90
11/04/2007 Digital Lutherie - Sergi Jordà 91
11/04/2007 Digital Lutherie - Sergi Jordà 92
AfasiaAfasia Musical interactivityMusical interactivity
• Limited semantics of sensors employed è score-driven interactive model, which can in a continuum go from 100% deterministic sequences to 100% generative
• Each of Homer’s islands is a diff. state with its own file, reflecting particular behavior and mapping
• Format 1 Standard MIDI Files expanded with 40 text meta-events: tell the custom sequencer how to process the data according to the performers inputs
e.g. TRANSPOSE=4,3,12$
11/04/2007 Digital Lutherie - Sergi Jordà 93
Some possibilitiesSome possibilities• Switch between blocks of tracks• Mute/unmute tracks• Transpose tracks• Loop sequences - modify their
length• Modify current play position• Modify any MIDI control (by
value, by increment, by random range…)
• Quantize/delay tracks• Change tempo• Define chords or scales (for
correcting generated notes)• Gestures and solo modes …
• Open form compositions (J. Cage, Earle Brown, Boulez)
• Expands the possibilities of some early 80s MIDI sequencers (e.g. KCS, MusicX)
11/04/2007 Digital Lutherie - Sergi Jordà 94
11/04/2007 Digital Lutherie - Sergi Jordà 95
Drawbacks Drawbacks –– Multimodality: too much is not enoughMultimodality: too much is not enough
• The temptation given by digital technologies to control ‘everything’, including lightning, image, machinery, etc. cohabits with the danger of ending up with a sort of theme park ‘one-man-band’ (Puckette1993; Tanaka 2000…)
• Within all that digital power in his hands and below his knees, Antúnez can do little to convey his expression. He may feel like Homer, Ulysses and Zeus all-in-one; he surely can express his feelings and ideas as a director, but he is too busy as a performer, he has too many things to think about.
• Afasia is not indeed a music instrument.Besides, it cannot be played without compositions (albeit open ones)
11/04/2007 Digital Lutherie - Sergi Jordà 96
FMOL (FMOL (F@ustF@ust Music On Line) 1997Music On Line) 1997--20032003• Jordà, S. & Aguilar, T. (1998). A graphical and net oriented approach to
interactive sonic composition and real-time synthesis for low cost computer systems.” Proceedings of COST G6 Conference on Digital Audio Effects, 207-210, Institut Universitari de l’Audiovisual, Barcelona.
• Jordà, S. (1999). Faust Music On Line (FMOL): An approach to Real-time Collective Composition on the Internet. Leonardo Music Journal 9.
• Jordà, S. (2002). FMOL: Toward User-Friendly, Sophisticated New Musical Instruments. Computer Music Journal, 26(3), 23-39.
• Jordà, S. (2003). Sonigraphical Instruments: From FMOL to the reacTable*. Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03), 70-76. Montreal.
• Jordà, S. (2003). Interactive Music Systems For Everyone: Exploring Visual Feedback as a Way for Creating More Intuitive, Efficient and Learnable Instruments. Proceedings of Stockholm Music Acoustics Conference (SMAC03) 681-684. Stockholm.
11/04/2007 Digital Lutherie - Sergi Jordà 97
FMOL Decalogue (1997)FMOL Decalogue (1997)Project for la Fura dels Baus, F@ust 3.0 show (based on Goethe’s)
“Given the important role symbolized by the Internet in our play , we would like part of its music to be composed by cybercomposersaround the world”
• Collaboration Collective composition on the Net• Aesthetics “Sound over notes” priority (NO General MIDI,
no controllable pitches, no obvious rhythms…)• Proselytism Introduce newcomers into “experimental” electronic
music• Scalability Attractive to both trained and non-trained electronic
musicians, intuitive & complex, learnable• Affordability Cheap (free) and available (no special hardware)
èMouse driven client software for RT synthesis and RT composition with a peculiar “visual feedback” interface
11/04/2007 Digital Lutherie - Sergi Jordà 98
FMOLFMOL’’ss Interface main characteristicsInterface main characteristics
• The interface works at a very low level• Developed in parallel to the sound engine• All functionalities are visible (D.Norman’s affordance?)• Audiovisual feedback, visuals work both as input & as
output (abacus)• No indirect mappings between what you see & what you
hear• As music needs time, visuals need animations
11/04/2007 Digital Lutherie - Sergi Jordà 99Bamboo screenshot
11/04/2007 Digital Lutherie - Sergi Jordà 100
• 6 audio channels • For each channel: 1 generator & 3
processors (selectable from more than 100 algorithms)
• For each generator-processor: 4 LFOs
• For each LFO, dynamic control on frequency, amplitude & shape (sin, square, saw, triangle, random)
• LFOs are fundamental for all time evolution (no use of prerecorded sequences)
• Strings are plucked and fretted• They draw the sound they generate• Horizontal lines oscillate according
to their LFOs
11/04/2007 Digital Lutherie - Sergi Jordà 101
FMOL OutputFMOL Output
• More than 1,100 Internet pieces by more than 100 authors (Jan-April, 1998), 50 used in the play F@ust 3.0 by La Fura dels Baus
• Collective CD published in 1998• More than 600 pieces (September 2000), used in the opera DQ by
la Fura dels Baus• Several workshops for musicians & non-musicians (visual artists,
kids, teenagers, “pro” improvisers… in Lisbon, Berlin, Dresden, Barcelona, Galicia…)
• FMOL Trio (1999-2003): The only instrument I designed for “everybody” is now the only one I personally use live -Improvised electronic music + visuals (3 live CDs released, 1 long-feature film soundtrack…)
• Continues to be my main music instrument
11/04/2007 Digital Lutherie - Sergi Jordà 102
11/04/2007 Digital Lutherie - Sergi Jordà 103
Reasons for FMOL Reasons for FMOL ““successsuccess”” ??Is rich & “interesting”• Non-linearity, feedback and ‘complexity’ (at the audio level…)• Allows to play sound & form simultaneously• Allows development of both mental & physical virtuosity• ‘Sharpness’• Passes the 3 Diversity tests
Is efficient• Visual feedback• No symbolic interface indirection• “N-dimensional” interface (graphic & mental model reflects the
engine structure) è not thin & not merely wide (hold, skip, through, random access…) (150 parameters at a time)
11/04/2007 Digital Lutherie - Sergi Jordà 104
RecouplingRecoupling & Visual feedback& Visual feedback• By this seamless integration of output and input, of
representation and control, already present in the abacus, FMOL manages to recouple the typically decoupled new digital instruments’ two main components: the controller and the generator.
• The ‘audio’ feedback presented to the performer in an interactive visual form, intuitively helps the understanding and the mastery of the interface, enabling the simultaneous control of a high number of parameters that could not be possible without this visual feedback.
• In multithreaded instruments, the performer is permanently delegating and shifting control to the instrument, and thus needs all the affordable ways for monitoring the instrument’s processes activities.
11/04/2007 Digital Lutherie - Sergi Jordà 105
PART III PART III CONCLUSIONS &CONCLUSIONS &FUTURE WORKFUTURE WORK
11/04/2007 Digital Lutherie - Sergi Jordà 106
15. The performer must thus be able to affect all these dimensions: both the metacontrol of ongoing processes and the microcontrol of final parameters should be permitted. All of them with the maximum simplicity, flexibility and speed.
16. The many internal dimensions of the instrument, should not be controlled by four or five input dimensions, leaving all the dirty work to some clever mappings.
17. To allow this type of control, new instruments have to be ‘wider’ rather than ‘deeper’. ‘Wide’ instruments permit a better direct access to all of their complexity.
18. Since not all the parameters can be controlled simultaneously, these instruments need to integrate intuitive and efficient multiple sample/hold/return/bypass options.
11/04/2007 Digital Lutherie - Sergi Jordà 107
19. These mechanisms favor a type of selective or asynchronous playing (not all the parameters are permanently modified; not all the processes are equally active).
20. In a ‘wide and concurrent instrument’, multi-user possibilities, though not a must, can represent an appealing added value. A multi-user approach can enrich the instrument’s bandwidth without provoking a single-user overload.
21. Wide instruments demand wide controllers or interfaces.
22. Moreover, sophisticated and complex concurrent instruments will benefit from interfaces that reflect and help to understand the instrument’s ongoing internal mechanisms.
11/04/2007 Digital Lutherie - Sergi Jordà 108
23. The potential of computer graphics for representing and monitoring complex processes, is not easily surmounted. This is probably the reason why many of the more interesting recent instruments, are screen based. While, the mouse is a very limited controller that we should definitely try to avoid.
24. Screen based visual feedback can still be used in combination with physical controllers, but in these cases an undesirable indirection component is added (the performer does not play IN the screen).
25. For including realtime interactive visualizations and, at the same time, overcoming mouse limitations without adding indirections, interfaces should be able to reflect their own states and behaviors. They should integrate, like the abacus, both representation and control.
11/04/2007 Digital Lutherie - Sergi Jordà 109
Being developed by the Interactive Sonic Systems Team at MTG (UPF)
• Sergi Jordà (conception & direction)• Günter Geiger (sound synthesis & control)• Martin Kaltenbrunner (computer vision & net logic)• Marcos Alonso (visual synthesis)• Hugo Solis (computer vision & hardware)
With the collaboration of • Ross Bencina (computer vision)• Ignasi Casasnovas (visual synthesis)
Postlude: The Postlude: The reacTablereacTable**
11/04/2007 Digital Lutherie - Sergi Jordà 110
11/04/2007 Digital Lutherie - Sergi Jordà 111
Thank you !Thank you !
Sergi Jordà
Music Technology GroupInstitut Universitari de l’AudiovisualUniversitat Pompeu Fabra, Barcelona
[email protected]://www.iua.upf.es/~sergi