transdisciplinary music research at the cross-roads

1

InstItuto de estudos AvAnçAdos trAnsdIscIplInAres - IeAtunIversIdAde FederAl de MInAs GerAIs - uFMG

TRANSDISCIPLINARY MUSIC RESEARCH AT THE CROSS-ROADS: TENDENCIES, PERSPECTIVES AND OPPORTUNITIES1

Introduction

A transdisciplinary approach to music suggests that music cannot be fully studied and understood if it is approached by only one single discipline, or perhaps by even by different disciplines which are just put next to each other without much interaction. Indeed, transdisciplinarity suggests that music can or should be approached by transcending scientific disciplines. This implies a dynamics that goes beyond the boundaries of what separate disciplines can offer to music.

The idea of transdisciplinarity is not entirely new. Since the late 19th Century, systematic musicology has been promoting an integrated multidisciplinary approach to music research (see e.g. Adler, 1885; Elschek, 1992). Based on a tight collaboration between scientific disciplines, systematic musicology offers a way to understand how people engage with music, and how music functions in perception, performance, and as an aesthetic and social phenomenon. In its scientific approach, systematic musicology was first influenced by the Gestalt theory and, later, by information psychology and cybernetics. In the 1970ies, with the advent of computers, systematic musicology culminated in the so-called cognitive musicology approach which, up until today, still offers a main scientific research paradigm to systematic musicology (Leman and Schneider, 1997). A key aspect of this methodology is that it relies on scientific measurement for gathering empirical data, and on data-analysis and computer modelling for hypothesis testing. Since music is both a subjective experience (related to thinking, emotions, feelings) and an objective matter (related to the physical world), a combined naturalistic and culturalistic approach is needed. This double root is at the core of the whole discussion of musical transdisciplinarity. It fully subscribes to the idea that to understand the true nature of music and what it does to people, it is necessary to combine methods from human sciences and from natural sciences. Transdisciplinarity, from the very beginning of systematic musicology, was not a idle concept, nor a luxury, but a core necessity.

The reason why a single discipline, be it music theory, psychology, sociology, acoustics, computer science, or even brain science is too narrow a basis to grasp the different ways in which people deal with music is that music is a highly multimodal phenomenon involving all human faculties and very different social and cultural contexts. Single

1. Texto apresentado como resultado da participação de Marc Leman, da Universidade de Ghent, junto ao Programa Cátedras FUNDEP/IEAT, no período de 30 de julho de 2007 a 17 de agosto de 2007

MArc leMAn, unIversIty oF Ghent

trAnsdIscIplInAry MusIc reseArch At the cross-roAds: tendencIes, perspectIves And opportunItIes

2

MArc leMAn - unIversIty oF Gent


disciplines often focus on particular aspects and fail to address aspects that go beyond the confines of the discipline. Indeed, for most subjects, a steal drum is a loud instrument, and a guitar is a soft instrument, even if the number of decibels used for the propagation of a steal drum through head phones is less than the number of decibels of the guitar sound. The reason is that perceived sounds are linked with actions, contexts, and previous experiences. Loudness is therefore not only a matter of perceived intensity (dB, dB(A), or sone, in the psychoacoustic definition), but also of the context in which musical sounds are normally perceived. A too narrow focus may fail to understand why the subjective concept of loudness could differ from the psychoacoustic definition and why users would tend to use it in a different way in a semantic music information retrieval context (e.g. a music google where they search for steel drum music, using the adjective “loud” for characterizing the instrument). Indeed, music perception has a strong action-related component on which users rely when they address the semantics of music. A transdisciplinary approach could address the subjective and context-dependent way in which humans deal with music, without neglecting the physical environment in which music is perceived either. Indeed, focussing solely on subjective matters only is equally problematic.

It should be added that the recent interest in transdisciplinarity is not just a postmodern trend based the use of some new words. To the contrary, the use of this term in musicology is well-conceived and believed to be a core aspect of the actual music research methodology. Indeed, since a few years, the terms “transdisciplinarity” and “multidisciplinary”2 have been used in European initiatives that aimed at identifying the role of music research in relation to the upcoming creative and cultural industries. It is commonly believed that the transdisciplinary nature of music research may be a strong asset to the development of a new type of creative music industry.

In what follows, I intend to clarify the transdisciplinary nature of music research in more detail, and I will do this in view of the new upcoming creative and cultural industries. I will try to show that music research is not only about understanding music, but that this research can also contribute to societal goals, like cultural participation and social interaction.

In the first part, I will argue that a motivation for developing transdisciplinarity in music research may be grounded in ethical principles that foster the progress and well-being of society. In the second part, I will argue that music research in Europe forms part of a research space which imposes particular constraints on how transdisciplinarity can be worked out. In the third part, I will go deeper into the problem of the semantic gap, which music research has to solve and I propose a particular approach, called

2. In particular, the UK-roadmap for music research uses the term “transdisciplinary” extensively, whereas the (Continental) S2S2-roadmap uses the term “multidisciplinary”. The difference between multidisciplinary and transdiscplinary is subtle and I would propose to use the terms here as synonym. The rationale behind this is that true multidisciplinary work is also based on work that goes beyond the boundaries of the disciplines involved, and from that moment on, this work can be said to transcend the disciplines on which it is based.

3


embodied music cognition, as a way to work towards a solution of this problem. All of this is followed by a discussion of a roadmap for future music research which was recently presented at the head quarters of the European Research Council in Brussels. The roadmap identifies transdisciplinarity (or true interdisciplinarity) as a core feature of its methodology. Overall, I show that “transdisciplinarity” can be considered from different perspectives, namely, (i) research space, (ii) research topic, (iii) research strategy.

A motivation for developing transdisciplinarity in music research

Motivations relate to personal drives and in my case, the motivation for developing transdisciplinary in music research relates to my former role as editor-in-chief for Journal of New Music Research (from 1987 till 2004), and my engagement in a number of National and European research projects on music3 . These activities provided a fruitful background for the development of a generalized opinion of what music research could be about, both at the institutional, national and international levels. However, as opinions are personal and perhaps mainly biased by the particular research context, they typically need a broader justification and grounding, which I believe can be found in philosophy, more particularly in the work of the founding fathers of the modern philosophy such as Descartes, Spinoza, Kant, as well as in the more recent approaches to the philosophy of mind and the philosophy of science. Issues in epistemology (what is sure, or reliable, knowledge) and methodology (how to acquire this knowledge) are crucial to music research, as are the ideas on topics such as the mind-body problem, the status of perception, cognition, embodiment and social engagement, social theory and ethics.

A broad cultural and societal context for music research

If my students come up with a summary of the notes that they counted in a score (yes, this is done in musicology!), then I tend to say that it is useful if a proper justification for doing it can be given. Saying that by counting notes, we know the distribution of

3. Since 2000, the major National Projects at IPEM, Dept. of Musicology, at Ghent University, Belgium have been: • MAMI (2001-2005, HOBU-project IWT) about audio/data-mining for music information retrieval• DEKKMMA (2003-2007, Federal) about digitalization of the etnomusicological sound archive of the KMMA • MELO (2005) about digital learning objects for systematic musicology• DEMCO-FWO (2003-2007), about the exploration of kinematics for music analysisWe were also involved in the following international projects:• MEGA (2001-2004, IST-EU) about a platform for interactive music applications• S2S2 (EU, coordinated action) design of a roadmap for future music research• ConGAS (COST 287) about kinematic control of audio systems• SOCRATES/ERASMUS (IP) Organisation of the ISSSM (UGent, 2006 en 2007).• POFADEAM (EU-culture2000) (2006) explorative project for quality control of audio digitalisation for etnographic collections

4



the notes and can infer from it the tonality of the musical piece, is not sufficient. Why should we know the tonality of a piece? What could be the use of that knowledge? To what extent does it progresses society?

In Europe, the enlightened philosophers of the 17th and 18th centuries have led the foundation for these top goals and values, and after a number of political developments (sometimes very dramatic ones), we have finally managed to put these values as top goals of our European society. The top goals are those of freedom, equality and brotherhood (solidarity) for all people. In general, I believe that it is a good attitude of transdisciplinary research to justify the work that is done within a broader cultural and societal context, and to confront the work with the top societal goals and the major values of our society.

Obviously, these values form a broad framework and they need to be scaled down to more practical goals and guidelines for music research. In fact, that is what a roadmap for music research should be doing, namely, to identify the main societal goals, and to scale them down all the way to goals that music research can achieve, thus providing a rationale and justification for the research at short, middle and long term. For example, freedom, equality and solidarity imply respect for the diversity of the social and cultural identity of people, the care for cultural heritage, the openness to cultural change and new forms of cultural expression, the democratic access to knowledge in promoting a culture of participation and participation in culture. These values can be addressed in more concrete ways through research that aims at giving public audiences access to music libraries. In order to be able to do this, it is necessary to develop computational tools related to the development of music ontologies, music information retrieval tools, annotation and music feature extraction tools, contextual descriptions of music cultures and societal function of music and so on. In that perspective, counting notes in a score, in order to infer the tonality of a piece, can be of interest, as it can be used by other people to retrieve that piece or to find similar pieces with similar tonal structures.

Music research without a proper justification of what kind of societal goal it aims at is questionable, and I believe that a transdisciplinary approach is needed to allow particular musicological preoccupations to be connected with broader contexts, such as creative and cultural industries, in which this knowledge may be used.

The need for a unified framework for music research

Apart from the link with the broad cultural and societal context, I believe that transdisciplinary is also grounded in the need for a unified framework for music research. The rationale for a unified framework for music research came to me after several visits to institutions in Europe, among which a visit the postgraduate school of the Music Technology Group of Pompeu Fabra University in Barcelona (June 2003), and later on that same year my guest-professorship at the Kulturwissenschaftliches Forschungskolleg Media und Kulturelle Kommunikation (December 2003) of the University of Cologne, and then a visit to Max Planck Institute for Cognitive Neuroscience

5


in Leipzig (May 2005). These activities introduced me to a new phenomenon in which research groups from disciplines that different from musicology, specialize in music and cover particular problems which, before, were mainly addressed by musicologists. Through specialisation in the methodologies of their own discipline, such as engineering in Barcelona, cultural studies in Cologne, and neuroscience in Leipzig, these groups can advance the field a lot. However, their diversity in approaching the matter is quite striking and largely determined by the specific research discipline in which they are involved. Discussions with people from different disciplines convinced me that there was a great need for an approach that would unify these different approaches to music research, thereby focusing on what music is about and what it does to people. I thought that such a unifying approach, a true multidisciplinary or transdisciplinary approach, could be of value to a broad range of scholars and students with backgrounds in musicology, philosophy, engineering, physics, psychology, and neuroscience.

My main concern was that such a unifying approach could no longer be offered by the mainstream cognitive musicology approach because it did not fully take into account the role of the human body in musical activities. In fact, the focus on music cognition research was much a disembodied approach as it focused attention mainly on acoustics, perception and cognitive structure. What was needed was an approach that would more fully take into account the role of action in all music activities, thus providing a unified framework for music psychology, music technology or engineering and music neuroscience of the brain science of music.

During my guest-professorship in Cologne, in 2003, I decided to write a book that would address this issue. Meanwhile, this resulted in “Embodied music cognition and mediation technologies”, which MIT-Press will release this September (Leman, 2007). Although the term “transdisciplinary” is not mentioned at all in this book, it is fully in the spirit of true multidisciplinary work. The human body is thereby considered to be central component in all music related activities, including perception and meaning formation, and its dynamics is much contributing to mental processes. Rather than looking for ways in which environmental information emerges in mental structures, the focus is now more on the active role of the human body in perception disambiguation and ways in which emergent mental processes draw upon human embodiment in their relationship with the environment. I believe this approach holds the germ for a unified framework for music research, as I tried to show in two fields of application, namely, interactive music systems, and music information retrieval.

Most of the doctoral work at IPEM is now focusing on embodiment and music. For example, the doctoral thesis of the Brazilian student Luiz Naveda at my institute focuses on the hypothesis that in Samba music, body movement (likely also socially mediated) is turning an ambiguous musical stimulus into a meaningful activity. We develop a scientific methodology to show that this is indeed the case. By doing this, we can use the same measurement technology to extend the body and develop new mediation

6



technologies (e.g. the HOP-sensors)4 that are relevant for the so-called creative and cultural industry.

The participation in an European project called S2S2 (sound to sense, sense to sound) offered a further opportunity to develop this idea into a roadmap that can have some impact on the strategic initiatives which the European Union is taking with respect to music research (further details are given below).

In short, a motivation for trandisciplinarity in music research is rooted in the general ethical principles of Enlightenment, and in an epistemology and methodology for attaining reliable knowledge, as well as a drive to unify ongoing different directions in music research (related to brain science, technology and cultural studies).

The Sound and Music Research Space

In order to work out the notion of transdisciplinarity in music research, one needs a research space for music research. Due to my ignorance of the Brazilian situation, I will confine myself here to a discussion of the European research space. As known, the music research space forms a very small part of a more general European research space that is characterized by a number of large research institutes and a large number of universities and smaller research institutes. All these institutes are spread over different countries and languages, and they have different structures, traditions, and missions. Often the large institutes are geographically spread too and they are integrated within universities and smaller research institutions5.

Clearly, the geographical spread and the cultural tradition of distributed research centres imposes particular constraints on research, related to issues of language, lack of resources, specialization and the need for collaboration, the mastering of different languages and so on. In order to be able to play a role in the world, small centres have to specialize and rely on intensive collaboration in a networked environment. Over the last two decennia, the European Commission has much contributed to the creation of a research space where competitive centres form changing alliances that closely collaborate on a project basis. This research space proved to be beneficial for innovation, especially in domains where long term goals are ill-defined (Nowotny et al., 2001).

This general context should be kept in mind when speaking about transdisciplinarity in music research. Indeed, the music research space in Europe is much a reflection of

4. The HOP-sensors are developed in collaboration with IMEC/UGent (Prof. Jan Vanfleteren). The aim is to develop wireless, non-obstrusive and cost-effective sensors that capture movement in view of social embodied music cognition applications. The HOP-sensors will be demonstrated to the public for the first time in September 2007 at the ACCENTA exhibition.5. In fact, it is only recently that the European Research Council (ERC) decided to create a concentrated large research institute at a scale that would be able to compete with MIT or Stanford in the US, but it remains to be seen how this will work out in the European context.

7


this general research space. As known, there is only one big music research institute, namely IRCAM (Paris, France), whereas the rest are smaller institutes. However, it is through specialization and networking that an attractive, lively, and strong research space for music research has been able to emerge throughout Europe. However, at the same time, music research is vulnerable (mainly in terms of research funding) and perhaps not fully adapted to the upcoming creative and cultural industry (mainly in terms of the global attitude and ignorance with respect to these industries).

The European music research space has recently been studied in the context of the S2S2-project, where this space is called “sound and music computing” (SMC). The S2S2-roadmap defines SMC research as a multidisciplinary (also: transdisciplinary) approach to sound and music communication. By combining scientific, technological and artistic methodologies, SMC aims at understanding, modelling and generating sound and music through computational approaches. Music is thereby understood as the intended organisation of sounds for particular uses in social and cultural contexts. What SMC research aims at is the study of all aspects of the relationship between sonic energy and meaningful information, both from sound to sense (as in musical content extraction or perception), and from sense to sound (as in musical composition or sound synthesis). The transdisciplinary viewpoint also includes research goals that deal with cross-modality, such as the relationship between perception and action, and the integration of different senses involved in human-machine-human interaction. This approach is empirically-based and modelling-based and draws upon advanced tools for measuring and processing information. Sound and music computing also refers to artistic methodologies that explore human experience and expression. Part of the mission of SMC is indeed related to the social and cultural context in which music is functioning (S2S2 , 2007).

The above music research space offers a proper context to discuss and critical evaluate the current challenges and opportunities of music research. These challenges are not only related to research challenges, but also to strategic challenges (such as research infrastructures and stable funding). Often the one cannot be considered without the other. Indeed, the research space in which music research is currently situated, is largely influenced by economical, cultural, scientific and educational trends, and even ethical values (see above) in society. These trends somehow define the general context in which research takes place.

Vibrant cultural industries versus lack of research funding

In the EU, a number of studies have recently addressed the economical situation of the creative and cultural industry to which music research is closely adhering. Similar studies, I believe, have been undertaken in Brazil. Interestingly, the EU-studies reveal that music represents one of the most vibrant cultural industries in the Europe. According to KEA (2006), music represents about 40% of the world-wide activity in this area. Europe employs about 650,000 workers in this sector, with an annual return of more than € 40 Billion (in Dutch: Miljard Euro). (Rough estimates for recorded music

8



sales are € 12.5 Billion; music publishing € 3.4 Billion; life concerts > € 12.5 Billion; collecting societies € 4.5 Billion). The study also claims that the cultural/creative industry to which music belongs is in total “twice as large as auto-mobile, and as large as ICT”, with remarkable growth figures of about 20% over the past 5 years.

Apart from being a vibrant economy, music is also a core component of the social and cultural cohesion of our society. As music has strong roots in the local cultures, it contributes to personal development and social cohesion/inclusion, self-respect and pride. In short, music communicates cultural values and stimulates self reflection.

Of all "content industries" (film, TV, art, heritage,…), music is the one which has been most affected by the digital revolution. Indeed, since the year 2000, the creation/production/distribution/consumption chain for music is almost entirely digital. Music is pushing broadband development (e.g. Napster and P2P) and mobile networks (GSM/GPRS, UMTS). Given this context, music has stimulated the uptake of broadband subscription and ICT by mass consumers (e.g. PCs, mobiles). Music stimulates e-business (e.g. iTunes), new management tools (e.g. Digital Rights Management, Audio-fingerprinting, Watermarking) and retrieval methods (Music information retrieval). Kusek and Leonhard (2005) predict that music industry will transform itself into an experience-based economy where musical audio will be freely distributed via large networks of ICT channels (broadband, mobile) but where services will provide an added economical and experiential value. The impact of music on media consumption has been huge in recent years and music has been a key driver for ICT uptake. Also in education, music has been a driver for young people to develop interest in science and ICT.

However, despite this large economical and social-cultural value of music, music research is still rather small-scale. Apart from a few exceptions (e.g. UPF-MTG in Barcelona, IRCAM in Paris), institutes working on music (in musicology, engineering, psychology, brain research) tend to be rather small (mean: 3 professors, 10 PhD students), although the number of doctoral dissertations and internationally peer reviewed papers has been growing over the past decennium (S2S2, 2007). The scientific organisation at EU level is based on changing coalition networks of partners specialized in niche areas related to sound and music computing. Both the KEA (2006) study and the S2S2 (2007) survey show that Europe’s potential power in this field, compared to US and Japan, is large.

However, sound and music computing research and its impact on industry is vulnerable due to unstable funding mechanisms. For example, the research at IPEM-UGent, Belgium, relies only on 1 single professor (tenure full-time), 1 guest-professor (30% temporal), 3 post-docs (temporal), and more than 12 PhD students. A failure in a research grant application can severely damage the know-how that was built up with great effort over a decade. As a side effect, the connection between academia and industry remains difficult and unexplored. It is known that the US is more versatile in terms of bringing a good idea into the market. The lack of continuity in research funding is one of the major difficulties of small scale institutes, especially when those

9


institutes are operating in humanistic disciplines rather than science or engineering.

To sum up, Europe has a flourishing music research space which may play an important role in the development of a new music industry. This research space is mainly organized as a network of small institutes, and consortia are formed by changing coalitions among the member of the pool that defines music research. However, this research space is also vulnerable, mainly due to discontinuous funding (which for small institutes is more dramatic than large institutes due the fact that core know-how is in the hands of a smaller number of people). Transdisciplinarity in this context implies that small institutes have to cross the boarders of their own institute and establish collaborations with other institutes. This can be done at home (own university) or abroad, possibly at an international level. Often the dynamics also implies that young researchers have to be flexible and change institutes throughout Europe according to the available opportunities.

Human semantic processing versus semantic gaps in technology, how embodiment can close the gap

Having specified a personal motivation for transdiciplinarity in music research, as well as a context in which transdisciplinarity operates within the European music research space, I now turn towards the core business of music research, namely, the research problem and the different approaches to handle these. Also at this level, it turns out that transdisciplinarity is a key aspect of the research approach.

One of the major problems to tackle in sound and music research can be identified as the semantic gap problem, that is, how people can access music (for retrieval) or produce music (for expression) with new digital technologies. This kind of access to music is a problem because the retrieval technologies are insufficiently taking into account the user’s search intentions, personal attitudes and social/cultural contexts. In a similar way, the production of music is a problem because electronics eliminates the direct causal transfer of bio-mechanical energy of the player into sound energy (as in acoustical instruments). As a result, the action-intended music control is rapidly lost. Music indeed implies meaning, experience and social practice. However, up to now, there is no satisfactory solution for the semantic gap problem.

In what follows, I first consider the classical approach to the semantic gap problem, which is object-based, and then formulate an alternative approach, which I call subject-based. The ultimate solution will consist of a combination of both.

The classical approach to the semantic gap problem

The classical approach to the semantic gap is object-oriented (in the sense of a material object), using feature extraction and classification as a means to transform physical energy into concepts which humans can mentally access. However, in his doctoral dissertation (partly executed at IPEM), Paivo (2006) demonstrated that the classical

10



bottom-up approach (he took the melody extraction from polyphonic audio as a case study, using state-of-the-art techniques in auditory modelling, pitch detection and frame-concatenation into music notes) has reached its performance platform. The engineering results are far from being sufficiently robust for use in practical applications. Similar observations have been made in rhythm and timbre recognition. The use of powerful stochastic and probabilistic modelling techniques (Hidden Markov Chains, Bayesian modelling, Support Vector Machines, Neural Networks) (see also http://www.ismir.net/ for publications) do not close this gap much further (De Mulder et al., 2006). The semantic gap problem turns out to be a hard problem. Among experts (see e.g. S2S2, 2007), there is a growing understanding that the techniques are excellent, but that the approach may be too narrow. Briefly listed, the approach is currently based on:

- Unimodality: the focus has been on musical audio exclusively, whereas humans process music in a multi-modal way, involving multiple senses (modalities) such as visual information and movement.- Structuralism: the focus has been on the extraction of structure from musical audio files (such as pitch, melody, harmony, tonality, rhythm) whereas humans tend to access music using subjective experiences (movement, imitation, expression, mood, affect, emotion).- Bottom-up: the focus has been on bottom-up (deterministic and learning) techniques whereas humans use a lot of top-down knowledge in signification practices. - Perception oriented: the focus has been on the modelling of perception and cognition whereas human perception is based on action-relevant values.- Object/Product-centred: research has focused on the features of the musical object (waveform), whereas the subjective factors and the social/cultural functional context in musical activities (e.g. gender, age, education, preferences, professional, amateur) have been largely ignored.

In short, the approach starts from the object, and does not really take into account the proper context, nor the subjective factors that define how users would like to access music.

Why human sciences are needed to solve the semantic gap problem

There is a growing awareness that more input should come from a better analysis of the subjective human being and its social/cultural context.

Such a subject-centred approach would involve:

- Multi-modality: the power of integrating and combining several senses that play a role in music such as auditory, visual, haptic, kinaesthetic sensing. Integration offers more than the sum of the contributing parts as it offers a reduction in variance of the final perceptual estimate. - Context-based: the study of the broader social, cultural and professional context and its effect on information processing. Indeed, the context is of great value for the

11


disambiguation of our perception. Similarly, the context largely determines the goals and intended musical actions (Lesaffre et al., 2007).- Top-down: knowledge of the music idiom to better extract higher-level descriptors from music so that users can have easier access to these descriptors. Traditionally, top-down knowledge has been conceived as a language model. However, language models may be extended with gesture models as a way to handle stimulus disambiguation. - Action: the action-oriented bias of humans, rather than the perception of structural form (or Gestalt) (Knoblich et al., 2006). In other words, one could say that people do not move just in response to the music they perceive, rather they move to disambiguate their perception of music, and by doing this, they signify music (see below).- User-oriented: research should involve the user in every phase of the research. It is very important to better understand the subjective factors that determine the behaviour of the user (Lesaffre, 2005).

It is of interest, perhaps, to note that the subject-centred approach can be based on an empirical and evidence-based methodology, rather than, or in addition to, a hermeneutic and introspective methodology. The subject-based approach starts from the human subject and its phenomenological experience of the world, rather than from the material object, but it adopts methods that aim at measurement, statistical data-analysis, and computer modelling of this experience. When considered from a historical perspective, this approach is in line with the tradition of systematic musicology (Leman and Schneider, 1997). It is understood that a serious contribution from the human sciences to the semantic gap problem is only possible provided that sufficient research potential is available, so that a critical mass of researchers can provide the necessary subject-centred data and develop this in collaboration with object-centred approaches.

The combination of subject-centred and object-centred approaches is indeed what is needed for tackling the semantic gap problem. As mentioned, I tend to consider this to be a core aspect of transdisciplinarity in music research. In addition, it is important to mention that the empirical and evidence-based methodology applied to the human subject and its musical environment is likely to have a high impact on the fast developing cultural/creative and ICT industries. Music technology may boom in this area, but it is likely that this will only happen when technology is developed in collaboration with users. This is another aspect of transdisciplinarity in music research, namely, that it starts from the user and aims at improving the conditions in which the user may interact with other users and machines so that new ways of music interaction can be explored.

Our solution: closing the semantic gap with embodiment

In my book (Leman, 2007), I propose a solution to the semantic gap problem in terms of embodiment. In particular, I consider the human body as a semantic mediator between musical meaning and sound energy. The core idea is that music semantics has a strong corporeal aspect and that this corporeal aspect is largely unexplored

12



until today. The natural tendency to move along with musical energy (called corporeal resonance), forms the basis for the notion of corporeal understanding, and ultimately, empathic engagement with music. As shown in Figure 1, it is assumed that the human body supports action causation and perception from musical goal to bio-mechanica/haptic/sonic/visual energy and back via processes called corporeal articulation and imitation. The approach is related with the modern perception theoretical notions of emulation and simulated perception (Berthoz, 1997; Wilson, 2005). Historical roots can be found in the work of Piaget, Apostel, Maturana and Varela (e.g. Maturana and Varela, 1980). Reference can also be made to ongoing European projects (ENACTIVE, ConGAS, EMCAP and others). Meanwhile, as mentioned, the approach has already proven to very inspiring for a number of doctoral research proposals at IPEM.

Figure1. Music communication model

The difference with the traditional musicological approaches (e.g. Hatten, 1994; Cumming, 2000) is that embodied music cognition uses a scientific apparatus in acquiring the necessary knowledge of the role of the human body in meaning formation. Yet, it is not the intention to reduce music to physics, nor to reduce gesture and embodied meaning to the biomechanics of the human body. Instead, the human body is considered to be a core component of the mediation between the mental and the physical world. In this context, transdisciplinarity means that disciplines which address the mental world, the physical environment and the human body should be involved.

13


The S2S2-project: a roadmap for transdisciplinary music research

On 16 April 2007, the consortium of the S2S2-project launched its roadmap on Sound and Music Computing in Brussels at the headquarters of the European Research Council. This roadmap is an ambitious document which aims at defining the major challenges for future music research. As a guide, it will have impact on the strategic planning for sound and music research within the European Union. In what follows, I will briefly introduce the rationale behind this roadmap and I will argue that transdisciplinary is a core aspect of the European music research as it is envisioned for the future.

The S2S2-project was an initiative of N. Bernardini who set up an interdisciplinary consortium of music research laboratories in Europe with the major task of writing a roadmap for sound and music computing6. Although some major European research centres were not present in the consortium, such as IRCAM, France, and partners from UK, contacts were established with both IRCAM and the UK-group during the course of the project (from 2004 to 2007). Indeed, both IRCAM and the UK-network were working on their own roadmap7. With the S2S2-project, the European Commission aimed at creating a new competing force grounded in a network of smaller EU-institutions. At this moment, the roadmaps are currently processed and a special issue of Journal of New Music Research will be devoted to them, including commentaries from researchers from the US and Japan.

Content of the S2S2-project

Apart form a number of initiatives at conferences and meetings, including public discussions and surveys, the S2S2-project has produced three major outcomes, namely a book containing the state-of-the-art in sound and music computing (currently

6. The consortium included: 1. Media Innovation Unit, Firenze Tecnologia, Firenze, Italy (N. Bernardini)2. Music Acoustics Group of the Kungliga Tekniska Högskolan in Stockholm, Sweden (R. Bresin)3. Music Technology Group of the Universitat Pompeu Fabra in Barcelona, Spain (X. Serra)4. CSC - Dept. of Information Engineering, University of Padova, Italy (G. De Poli)5. Austrian Research Institute for Arti_cial Intelligence of the Austrian Society for Cybernetic Studies in Vienna, Austria (G. Widmer)6. Département d’Etudes Cognitives of the Ecole Normale Supérieure in Paris, France (A. de Cheveigné)7. Laboratoire d’Etude de l’Apprentissage et du Développement of the Université de Bourgogne in Dijon, France (E. Bigand)8. Institute for Psychoacoustics and Electronic Music of the Universiteit Gent in Ghent, Belgium (M. Leman)9. Laboratory of Acoustics and Audio Signal Processing of the Helsinki University of Technology in Espoo, Finland (V. Välimäli)10. Vision, Image Processing & Sound Laboratory of the University of Verona, Italy (D. Rocchesso)11. Laboratorio di Informatica Musicale of the University of Genova, Italy (A. Camurri)

7. The latter presented their roadmap in December 2004 in London (see e.g. http://music.york.ac.uk/dmrn/roadmap/). IRCAM presented its view during the roadmap launch of the S2S2-project.

14



edited by D. Rocchesso, to be published by LOGOS-verlag), a series of summer schools (Barcelona 2004, Genova 2005, Barcelona 2006, Stockholm 2007), and of course, the roadmap itself, which is a text of about 100 pages (edited by X. Serra, M. Leman and G. Widmer). The latter material will be slightly polished and published in the special issue of the Journal of New Music Research (edited by N. Bernardini and G. De Poli).

The roadmap contains three parts, namely, (i) a description of the context and main trends in which music research operates, (ii) a state-of-the-art and identification of the research points and open issues, and (iii) a description of the research challenges.

(i) Context: This consists of the research context, the educational context, the industrial context, and the social/cultural context. These contexts tell us much about the main trends of the societal framework in which music research is currently operative. For example, it is revealed that transdisciplinarity is necessary in research, for industrial development and cultural applications, but rather difficult to implement in education.(ii) The state-of-the-art then focuses on the main open issues. A distinction is made between research that focuses on sound and research that focuses on music. In between, there is the interaction between the two. For each research field (sound, interaction, music), there is an analytic and a synthetic approach. The analytic approach goes from encoded physical (sound) energy to meaning (sense), whereas the synthetic approach goes in the opposite direction, from meaning (sense) to encoded physical (sound) energy. Accordingly, analytic approaches to sound and music pertain to analysis and understanding, whereas synthetic approaches pertain to generation and processing. In between sound and music, there are multi-faceted research fields that focus on interactional aspects. These are performance modelling and control, music interfaces, and sound interaction design. The nature of these distinctions reveals the inherent transdisciplinary character of the research field, as both the analytical (from sound to sense) and the synthetic (from sense to sound) approaches.(iii) The challenges part looks ahead and identifies the key challenges for music research together with the strategies with which to face them. These challenges fit with the open problems that were identified in part ii, and they are constrained by the contexts which were identified in part i.

Challenge 1: to design better sound objects and environments

- Strategy 1: Seek directions in which to extend the notion of musical instrument- Strategy 2: Improve technologies for pervasively producing, transforming and delivering sounds- Strategy 3: Intensify research in sound modelling that goes beyond imitation towards capturing the communicative potential of sound- Strategy 4: Promote research in fields involved in the shaping of natural, artificial and cultural acoustic ecosystems- Strategy 5: Promote research on the effect of environmental constraints on artificially diffused sound and music- Strategy 6: Promote studies aimed at reducing sound and music pollution in public

15


and private ecosystems

Challenge 2: to understand, model, and improve human interaction with sound and music

- Strategy 1: Promote computational modelling approaches in human auditory perception and cognition research- Strategy 2: Provide extensive augmented perception paradigms- Strategy 3: Intensify research on expressivity and communication in sound and music- Strategy 4: Develop an embodied, integrated approach to perception and action- Strategy 5: Intensify multimodal and multidisciplinary research on computational methods for bridging the semantic gap in music- Strategy 6: Intensify interaction with the arts

Challenge 3: to train multidisciplinary researchers in a multicultural society

- Strategy 1: Design appropriate multidisciplinary curricula for SMC- Strategy 2: Promote broader integration of Arts and Sciences- Strategy 3: Promote cross-cultural integration- Strategy 4: Promote better coordination in Higher Education- Strategy 5: Enhance education resources for Higher Education.- Strategy 6: Promote the dissemination of available Higher Education in SMC.

Challenge 4: to improve knowledge transfer

- Strategy 1: Promote dissemination of SMC research and objectives among the general public- Strategy 2: Promote projects containing artistic components- Strategy 3: Promote the awareness of the various models of IP protection of research results- Strategy 4: Promote venues for meeting industry experts- Strategy 5: Promote direct industrial exploitation of research results- Strategy 6: Promote academic quality standards.

Challenge 5: to address social concerns

- Strategy 1: Identify social needs relevant to SMC development; develop methods for the evaluation and assessment of SMC technologies in social contexts- Strategy 2: Expand existing SMC methodologies (currently targeted at individuals) to understand music in its social dimension- Strategy 3: Promote development of technologies and tools for broader collaboration, information and communication engagement; emphasise user-centred and group experience-centred research and development- Strategy 4: Exploit cross-fertilisation between human sciences, natural sciences, technology, and the arts

16



- Strategy 5: Expand the horizon of SMC research through a multi-cultural approach.

Challenge 2, Strategies 4-6 mention integration, multimodality, multidisciplinarity and interaction with arts. The notion of multidisciplinarity is taken up explicitly in Challenge 3, where the need for multidisciplinary curricula is addressed. In Challenge 4, the mentioning of cross-fertilisation between human sciences, natural sciences, technology, and the arts contains an explicit reference to transdisciplinarity. Reference to augmented perception, expressivity, embodiment and multimodality support the core challenges for music research. In Challenge 5, there is an explicit call to develop music technology in its social dimension. The latter aspect is not unimportant. After all, music is a very important aspect of all human cultures. Music gives meaning to life. It is a basic ingredient of cultural, group and personal idenfication and social bonding. Music affects the mental and bodily health of people.

Relevance to society

The music research policy thus draws upon a number of defined strategic objectives in which solidarity and security are basic topics of concern, and in line with the general Enlightenment ideas on which the European Union is based. These objectives are based on concepts such as a friendly business environment, the embracing of change, economic and social cohesion, responsibility for common values, justice and risk management. This approach can be adopted as a basic framework for the social and cultural values and goals of SMC research. It implies, among other things:

- respect for the diversity of social/cultural identity- the care of cultural heritage (preservation and archiving)- openness to cultural change and new forms of expression- democratic access to knowledge- a culture of participation and participation in culture.

The above research outline is not in contradiction with the idea of a creative and cultural industry, of which music forms an important part, and in which the sound and music computing research community has the core task to provide the basic research that underlies its development.

According to Kea (2006), cultural activities can be stimulated by both bottom-up, grass-roots initiatives and also the top-down initiatives of administrations and institutes. These social and cultural strategies are beneficial to the economic environment because they:

- reinforce social integration and help build an inclusive Europe- contribute to fostering territorial cohesion- contribute to reinforcing the self-condence of individuals and communities- participate in the expression of cultural diversity.

17


Creation and innovation form the motor of SMC research. Most interestingly, they are strongly driven by the context of artistic application. In that respect, it is of interest to mention that content-based music technology has roots in the particular cultural rationality of the 1950s. That rationality, heavily supported by European governments of the time, led to novel developments in electronic music, of which interactive multimedia is a recent outcome. In contrast, audio-recording technology had already begun by the early 20th Century and was driven by the logic of economic rationality and the free market. The trend of allying content-based music technology to economic rationality is new. But it is reasonable to assume that artistic creation remains a major factor in maintaining the former's innovative character. There are at least two reasons why art will continue to throw up innovating challenges to SMC research:

- First of all, there is the desire for expression. If tools are used to be expressive, then one is always inclined to go beyond that what is actually possible. Indeed, recent developments in SMC research have pushed back the frontiers of sensing, multi-modal multimedia processing and gesture-based control of technologies.- Secondly, there is the desire for social communication, and for technologies that enhance collaboration and exchange of information among communities at the semantic level. And indeed, recent developments in SMC research have pushed back the frontiers of networking into technologies that deal with semantics as well as new forms of human_human and human-machine interaction.

In short, the context of art application results in a constant drive towards human-friendly and expressive technologies of mediation.

Conclusion

In this contribution I have clarified the notion of transdisciplinarity in music research. In the first part, I have argued that a motivation for developing transdisciplinarity in music research may be grounded in ethical principles that foster progress and well-being of society. In the second part, I have shown that music research in Europe forms part of a research space which imposes particular constraints on how transdisciplinarity can be worked out. In the third part, I have argued that a core problem of music research is related to the semantic gap between and I proposed a combined object/subject-oriented methodology with particular attention to an action-based epistemology as a possible way for a solution. The latter implements transdisciplinarity as a new approach to research, one in which (i) the human subject is considered, no longer as a passive registration device, but as an active contributor to meaning, (ii) the development of technology is seen as improved extensions of the human body that serves to mediate between mind and environment. All of this was followed by a discussion of the S2S2-roadmap for future music research, where I showed that the notion of transdisciplinarity (or true interdisciplinarity) is a core feature of its methodology.

I have further pointed out that the European research space is characterized by diversity: one big and many small research centres that collaborate together in

18



changing alliances, forming together a rather impressive network of interconnected unities that focus on music. The research space is characterized by its potential key role in the development of a creative and culture sector, namely as a research foundation for a dynamics of mutual reinforcement between ICT-development and music. It has indeed been noticed that both ICT and music trigger each other and drive towards new forms of cultural content and application. It seems that the European industry has an unique opportunity in this because it draws on a long history in which culture and in particular music has been a key element in the expression of cultural and ethic values. However, the European music research space is also very vulnerable. Despite its economic potential, there is a lack of research funding for music research. Small institutes suffer from a lack of continued funding. Although this is partly compensated by the large number of research laboratories, it is music research in Europe is still missing the entrepreneurs that dare take up a good idea and develop it further in the context of creative and cultural activities.

As the entire musical economical chain has moved from the analogue and physical carriers (Vinyl, CD) to the digital and virtual environment (iTunes, PC), modern music research tends to focus on ways of accessing music using these new technologies. Pure engineering solutions are no longer sufficient and time has come to consider an transdisciplinary or true multidisciplinary approach based on the integration of bottom-up engineering solutions and top-down human science solutions. I have further argued that much progress can be made when the subject-based approach is based on the concept of embodied cognition, thereby expanding the empirical and evidence-based methodology of disembodied music cognition towards involving the human body.

By discussing the S2S2-roadmap, I intended to show how the above ideas, and in particular, the notion of transdisciplinarity, have been included within a core strategic initiative that may have a large impact on the future of music research in Europe. This roadmap initiative, as I tried to show, is fully in line with the philosophical, ethical, methodological and epistemological principles that underlay the kind of systematic musicology that I have been advocating elsewhere (Leman, 2007) and on different other occasions. It is now time to get at work and develop the concrete initiatives that will realize the goals.

I apologize for the fact that my contribution is centred on European research, and this is largely due to my ignorance of the Brazilian context. I hope that during the take up the IEAT-Fundação de Desenvolvimento da Pesquisa (FUNDEP) Chair of Humanities, I will be able to learn more about the Brazilian situation, with the hope to establish long-lasing collaborations. The start up of a centre for brain research on music is certainly an important step towards the achievement of a research space for music.

19


Acknowledgement

I wish to thank Professor Mauricio Alves Loureiro from the Universidade Federal de Minas Gerais (UFMG) and the Institute of Transdisciplinary Advanced Studies (IEAT) for the invitation to take up the IEAT-Fundação de Desenvolvimento da Pesquisa (FUNDEP) Chair of Humanities.

20



References

Adler, G. (1885). Umfang, Methode und Ziel der Musikwissenschaft. Vierteljahrsschrift für Musikwissenschaft, Volume 1.

Berthoz, A. (1997). Le Sens du Mouvement. Paris: Editions O. Jacob.Cumming, N. (2000). The Sonic Self: Musical Subjectivity and Signification. Bloomington: Indiana University Press.

De Mulder, T., Martens, J.-P., Pauws, S., Vignoli, F., Lesaffre, M., Leman, M., et al. (2006). Factors Affecting the Music Retrieval in Query-by-Melody. IEEE Transactions on Multimedia, 8(4), 728-739.

Elschek, O. (1992). Die Musikforschung der Gegenwart, ihre Systematik, Theorie und Entwicklung. Wien-Föhrenau: Dr. E. Stiglmayr.

Hatten, R. S. (1994). Musical Meaning in Beethoven: Markerdess, Correlation, and Interpretation. Bloomington: Indiana University Press.

KEA. (2006). The economy of culture in Europe.Unpublished manuscript.

Knoblich, G. (2006). Human body perception from the inside out. Oxford ; New York: Oxford University Press.

Kusek, D., Leonhard, G., & Lindsay, S. G. (2005). The future of music : manifesto for the digital music revolution. Boston: Berklee Press.

Leman, M. (2007). Embodied music cognition and mediation technology. Cambridge, MA: Massachusetts Institute of Technology.

Leman, M., & Schneider, A. (1997). Origin and nature of cognitive and systematic musicology: An introduction. In M. Leman (Ed.), Music, Gestalt, and Computing: Studies in Cognitive and Systematic Musicology (pp. 13-29). Berlin, Heidelberg: Springer-Verlag.

Lesaffre, M. (2005). Music Information Retrieval: Conceptual Framework, Annotation, and User Behavior (PhD Thesis). Ghent: Ghent University.

Lesaffre, M., De Voogdt, L., Leman, M., De Baets, B., De Meyer, H., & Martens, J.-P. (2007). How potential users of music search and retrieval systems describe the semantic quality of music. Journal of the American Society for Information Science and Technology.

Maturana, H. R., & Varela, F. J. (1980). Autopoiesis and Cognition: The Realization of the Living. Dordrecht, Boston: Reidel.

21


Nowotny, H., Scott, P., & Gibbons, M. (2001). Re-thinking science: knowledge and the public in an age of uncertainty. Cambridge: Polity Press.

Paivo, R. (2007). Melody Detection in Polyphonic Audio. University of Coimbra, Coimbra.S2S2. (2007). A roadmap for sound and music computing. Brussels: The S2S2 Consortium.

Wilson, M., & Knoblich, G. (2005). The case for motor involvement in perceiving conspecifics. Psychological Bulletin, 131(3), 460-473.

transdisciplinary music research at the cross-roads

Documents