Pushing Location Based Media Art to theLimit

Technical Challenges and Solutions

yvan vander sanden & Zimcke Van de Staey

Mute, Leuven, Belgium

contact: zimcke@mutecode.com — yvan@mutecode.com

Abstract

The current use of location based sound art for mobile devices is quite straightforward. Honourableexceptions not withstanding, projects often go no further than linking predefined zones to one ormore sounds. While this approach is certainly interesting because of its low entry barrier – eventhe most casual listener will instantly understand the correlation between position and sound – itwill not aid us in sustaining interest in this art form for very long.In our presentation we will discuss a few alternative approaches to sound / location mapping withmore surprising and varying results. This can be attained by indirect interpretation of the GPSsignal itself – as we have done in our LineWalk I application – and also by combining GPS withthe multitude of readily available sensors.Of course there are some hurdles to be expected: implementing sensor based interaction is nosmall task for the digital artist. And to add to our challenges, the mobile market is still highlyfragmented. It’s not enough to write your software just for Android or iOS. This is why we proposeto piggyback on the game industry: game developers use the same means for a different outcome.While more general software development continues to struggle with delivering Android, iOSand Windows phone versions of their software, cross platform mobile development in particular isalready quite mature within the gaming industry. And as far as sensors and interface programminggo, there are several game engines available that make it possible with ease. However, there is onething missing: imaginative audio support. It would be a shame to go the extra mile when it comesto being creative with interaction, while at the same time being limited by a sound interface thatwill do little more than play sounds at certain locations.In this matter, the options are limited. LibPD for one can be a useful tool, but it feels a bit likeprogramming your interface directly in openGL. A more high-level sound engine preferably onethat combines a 3D positioning system, DSP processing, software synths and composition toolswould be welcome. We will finish our presentation with a modest proposal in this direction.

Contents

Introduction 2

1 Sound to Location Mapping 2

1.1 Relative Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Data Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Not All is New . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1

2 Technical Challenges 5

2.1 Cross Platform Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6HTML and Javascript Based Frameworks • Precompiling Frameworks • Native Mode Frameworks

2.2 Game Design Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Esenthel Game Engine

3 Audio 9

3.1 Imagination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2 a Wishlist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.3 the YSE Sound Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Conclusion 13

Introduction

Over the past several years, we have seenmany new location based sound art for mo-bile devices. This led to an extensive re-search on the artistic possibilities of the ubiq-uitous sensors available on these devices.

Notwithstanding those efforts of integrat-ing the sensors in mobile music applicationsand the numerous writings on this subject,the current sound to location mapping ap-proaches are quite straightforward. The con-cept used by many applications often solelyconsists of linking predefined zones to one ormore sounds. We have made two such appli-cations for Musica1: Curvices2 (2013) andKlankroute Maasland3 (2014). And whilethis is a fair choice for projects in the touris-tic domain, there are numerous other designspossible.

1. Sound to Location Mapping

Linking sounds to zones has the advantagethat, from the perspective of the audience,it is very easy to understand what is goingon. Current GPS technology on mobile de-vices is also accurate enough to implementthis with reasonable detail.

On the other hand, the ‘direct’ approachdoes not give the user much of a challenge.There is no room for discovery or surprises,

both of which are an important part of artis-tic experiences.

1.1 Relative MappingMore innovative ways to map sounds can berealised by interpreting the data in a rela-tive, indirect manner. The GPS sensor canbe used to track the movement of the listenerinstead of focusing on the exact location. Asan experiment, we did develop an interac-tive application that is based on this con-cept: LineWalk I 4 (2013), as seen in figure3. The app is a seven- to ten-minute com-position, during which the listener can paceback and forth. He is supposed to “walk ona line”, such as when he or she is waiting forthe bus. The macro structure of the pieceis entirely precomposed, but new sounds willappear on different locations on the screen.The listener can decide whether or not tofollow these sounds.

To increase the interactive possibilities inLinewalk I, we implemented a small softwaresynthesizer. This greatly improved the flex-ibility of sound generation on a micro level.Because the idea was to experiment with dif-ferent types of user interaction, we imple-mented the following concepts throughoutthe three parts of the composition:

• pitches of continuous sounds can shift bywalking further from the starting point;

1http://www.musica.be/2http://www.musica.be/en/curvices-rozalie-hirs-music-poetry-voice-cox-grusenmeyer-design-animation3http://www.klankatlas.eu/routes/klankroutemaasland/4https://play.google.com/store/apps/details?id=com.mute.linewalkI

Figure 1: Screenshot from Curvices

• areas in which a specific sound is triggeredcan grow when the listener stays withinthat area;

• sounds can be changed or moved to an-other position when the listener has themactivated for a certain period in time;

• sound triggers can be numbered, therebyforcing the listener to engage them in aparticular order;

• different layers of the composition can beactivated by standing on a specific loca-tion;

• sounds can be placed in the stereo field ac-cording to their visual appearance on thescreen.

By combining all these concepts together,we were able to create a complex set of pos-sible interactions for the listener. But whilethis enlarges the reusability of the app, thereis the danger that the interaction will be un-

clear to the listener. We tried to counter thisin two ways:

1. Progress will never halt completely. Theuser cannot get ‘stuck’ when he does notunderstand the interaction. Instead themusic will continue and the interactionwill (hopefully) become clearer over time.Whenever the listener comes to under-stand a specific method of interaction,he will be encouraged to start the musicagain and this time have more impact onthe way the composition evolves.

2. All interactions are visualized in astraightforward manner because mostpeople are far better in interpreting im-ages than sounds. By ensuring a stronglink between the audible and the visualcontent, the listener is hinted at the pos-sible result of his decisions.

1.2 Data CommunicationOther ways of extending location based me-dia art can be realised by using the wireless

3

Figure 2: Screenshot from klankroute maasland

data network on mobile devices. Networkedapplications can exchange information witha server-side component and adapt their be-haviour accordingly. This gives artists theability to define flexible sound works insteadof pre-composed content. Above all this isan interesting way to create interactivity forthe listeners. An idea that we would like towork on is an app with movable sound ob-jects, where a listener can virtually “pick up”a sound from its original location and releaseit somewhere else. When locations are storedonline instead of on the listener’s device, thisapproach enables more lasting interactionswith the environment. And because theseinteractions are visible to all other listeners,they have more meaning. It does not haveto end with sound objects either: the sameapproach could be used for text, images oreven audio filters.

The approach above can be described asa closed circuit. A server on the networkinteracts with all client applications to ex-

change information. Networked applicationscan also make use of more general data thatis available on the internet. The obvious ex-ample would be the current weather condi-tions. Other ideas could be the location ofnearby landmarks, altitude differences, pop-ulation density, vegetation, or basically ev-ery kind of location based information thatis available online. For the most part, appli-cation programming interfaces for this kindof information already exist.

In the near feature, we think that workingwith so called ‘big data’ will also be achiev-able. Possibly an application could also usethe data about the user itself, which is al-ready gathered by companies like Google andFacebook5.

1.3 Not All is NewCreating a meaningful musical experiencewith this data might be a challenge. Butwhen we look for ways to interpret all thisdata, we’re not really entering uncharted ter-ritory. An inventory of the possibilities is be-

5After all, advertising companies do it, the NSA does it, our governments do it, so why shouldn’t we?

4

Figure 3: Screenshot from Linewalk I. The green dot symbolizes the position of the listener.

yond the scope of this paper, but they are ac-tually not that different from the more estab-lished practise where composers map datafrom (movement) sensors to musical expres-sion. As an example we’d like to refer to thework done by Kristof Lauwers and Godfried-Willem Raes at Logos foundation6.

2. Technical Challenges

Developing mobile applications often resultsin developing as many applications as yousupport platforms. Not only do these plat-forms use different programming languages,they also don’t always offer the same possi-bilities, at least not at programming level.

In Tonal Tools7, we needed to visualizethe notes played on a keyboard while playingback a midi file, as seen in figure 4. While theresult looks more or lest the same on bothiOS and Android, we needed a very differ-ent implementation. On Android, we coulduse a simple AudioPlayer (check this name)

object to play back a midi file. But onceit is playing, there is no way to keep trackon note on/off events. We had to write ourown midi file parser, extract all note on/offinformation and synchronise that with theAudioPlayer object.

IOS on the other hand provides us with acallback function which is called every timea note on / note off event happens. Thismakes the visualisation of that event veryeasy to implement. But setting up a systemto actually play the midi file is not straight-forward at all. There is no object like theAndroid AudioPlayer to just play the file.IOS does not even have default midi sounds:these have to be provided by the software.

This small example shows us just howdifferent both systems really are. It’s notenough to just translate code to another lan-guage: often it takes completely differentcode to create an application for both plat-forms.

But even within one platform things are

6http://logosfoundation.org/scores gwr/Namuda Links/namuda studies.html7An application for iOS and Android that we’ve created for Musica

5

Figure 4: Tonal Tools includes a keyboard display which is synchronized to a MIDI file player.

not always the same. One version might re-quire a piece of code to be different fromanother version. When Apple introducedretina displays, developers were required toadjust their interfaces to the new screensize.Android als made a lot of changes to theirgeolocation system in Kitkat. Most of themare backwards compatible, but not all. Atthe very least, your code will not be optimalif you don’t review those changes.

Supporting several platforms and keepingtrack of changes in all of them takes a lot oftime. Time we’d prefer to spend otherwise, Iam sure. Which is why Mute opted to use across-platform development framework. Weprefer that someone else keeps track of allthose differences and changes, so that we canfocus on the actual application development.

2.1 Cross Platform Frameworks

Given the difficulties and extra workload ittakes to develop applications for multiple

(mobile) platforms, it is no surprise thatcross-platform development frameworks aregetting a lot of attention nowadays. To givecomplete listing of available frameworks isbeyond the scope of this article. Also, our listwould surely be outdated in a few months.Still, a brief overview of the most importantoptions might be helpful.

2.1.1 HTML and Javascript Based Frame-works

With the advent of HTML5 and other recenttechnologies such as jQuery, modern web-sites can behave more and more like real ap-plications. This is why some cross-platformframeworks do not really create an applica-tion. All the functionality is created inside acollection of webpages is packaged so that itcan be viewed locally on a mobile platform.

While this can make development easier,it also has some important drawbacks. Gen-erally speaking, the available functionalityis quite limited compared to native applica-

6

tions. The application does not automati-cally have access to all the features of theplatform.

Equally important in the context of thispaper is that this approach creates reallyslow applications. This can be a problemif you want your audio to do more than sim-ply start and stop playing. (A slow applica-tion does not mean that your CPU is playinggolf half of the time, it means that it needsa lot more time to do calculations. There-fore, slow applications drain your batteriesfaster.)

2.1.2 Precompiling Frameworks

Another approach is to develop in a gener-alized environment which automatically gen-erates the correct code for several platforms.Once you compile that code into a program,you have an application for the supportedplatform. This has the advantage the resultis native code, which performs a lot betterthan JavaScript. The user interface elementswill often translate to an iOS element on iOSand an Android element on Android.

While this seems like a good idea there is,again, one drawback. If your interface ele-ments look like the native ones, users willalso expect the application to work like a na-tive application. But there are a lot moredifferences between platforms than just theshape of the buttons. With iPhone tabs goon the bottom, but Android tabs are on top.And Windows 8 scrolls everything horizon-tally instead of using tabs8. The implicationis that you will end up designing a separateinterface for every platform, which is exactlythe thing you’re trying to avoid.

Another drawback is that the not so ex-pensive options in this category often taketheir time when the target platform changesits requirements. This may leave you withno option to publish your app in the store.More expensive frameworks do not have thisproblem but are, well, expensive.

2.1.3 Native Mode Frameworks

Here’s when it gets confusing: mobile devel-opment uses the word ‘native’ in two differ-ent contexts. When we discussed the idea‘native development’ before, we meant de-velopment with the tools for that particu-lar platform, like x-code for iOS or eclipsefor Android. But both platforms also have a‘native mode’. This is the low level softwarelayer which operates below the high level ob-jective C (iOS) or Java (Android) interface.As an example we will briefly show the dif-ference with figure 5: a diagram of Android’sinternal design.

Android applications (in blue) are writ-ten in Java, but they rely on the applica-tion framework and libraries written in C++code. Native mode applications only havea placeholder in Java while the real core ofthe program sits on the same level as thelibrary (in green). While the native modedoesn’t give you easy access to the applica-tion framework provided by the SDK, it doeshave one advantage: you can use the sameprogramming language on both Android andiOS. This means a library can be writtenthat handles the internals differently whilestill giving the programmer a single interface,without any performance loss. And becausethe developer can use c++ to create the app,it is also easy to include other c++ libraries.

A simple function to get the current lon-gitude on your location might look like this:

double getLongitude() {

#if defined ANDROID

return AndroidsLongitudeFunction();

#else

5 return ApplesLongitudeFunction();

#endif

}

The application developer only has toknow that the function ‘getLongitude()’ ex-ists and that it will return the current longi-tude. Whatever happens internally is up to

8http://www.netwhisperer.com/2013/03/27/mobile-app-development-native-vs-cross-platform/

7

Figure 5: The Android architecture9

the maintainer of the library.

These native applications have no accessto the standard user interface objects in An-droid or iOS, but we think this is actuallyan advantage. A problem only arises whencross platform frameworks try to mimic thestandard interfaces, in which they never suc-ceed entirely. Doing so creates an expec-tation from the user: an application thatlooks like a standard application should be-have like one in all circumstances. But incontrast, if the user interface is entirely dif-ferent users will not expect it to behave likea standard application.

Both performance and flexibility are im-portant for the creation of mobile media art.This is why we think that the advantages ofusing a cross-platform native mode frame-work greatly outnumber the main disadvan-tage, namely that access to the applicationframework is more difficult. (Note that it isnot entirely impossible!)

2.2 Game Design FrameworksNative mode frameworks are especially pop-ular with game designers. This should comeas no surprise: only native mode applicationsare capable of accessing the graphics pro-cessing unit (GPU) directly. Therefore theyare the only way to create demanding vi-sual software, which is what games often are.Game engines have been supporting cross-platform mobile development for a few yearsnow. And while direct GPU access might notbe overly important to all media art, thereare several other advantages to game enginesfrom which we might benefit:

• Game development depends a lot of im-porting resources, like images and sounds.They mostly have a very efficient pipelinefor this which really saves time during de-velopment.

• Both Android and iOS support OpenSLES in native mode, which is a mobile so-

8

lution for playing and mixing sounds thatis not available through the Applicationframework.

• Because of the steep competition, gamedesigners really want to use the latest tech-nologies. While more general frameworksmight take a while to implement a new fea-tures, this is often done within a matter ofweeks by game engines.

• Game engines are a highly competitivemarket. Only a few games really generatea lot of income, and the engine developersknow this. While a cross-platform frame-work for business applications often cost alot of money, game engines are relativelyaffordable.

As sound artists we are naturally most oc-cupied with the audio side of what we create.But we should not forget that our product isalso a visual one. Whether we like it or not,the is this GUI, the graphical user interface.

Game designers, always creating a unique’feel’ for every game, have understood thisvery well. When using the standard Androidor iOS interface, it is easy to create an inter-face that looks OK. But when you want tocreate something unique, be it a game or anart product, OK is not enough. An appli-cation should engage a user not only withgame mechanics or sound, but also throughits interface. Neglecting the visual side of aproduct is a missed opportunity, we think.

2.2.1 Esenthel Game Engine

At Mute, we created quite a few mobile ap-plications with the Esenthel game engine10.There are other game engines which are bet-ter known, but we have quite a few reasonswhy we keep using this one:

• The Application Programming Interface isvery logical and well documented. Onceyou’re familiar with the basics, develop-ment in Esenthel is really fast because ofthis.

• The built-in code editor has some veryinteresting auto-complete functions andshows relevant documentation while youtype. It even creates your header files foryou, takes care of pointers, includes andexternal declarations. Again, this greatlyincreases our development speed.

• Everything you do is automatically syn-chronized over the network. This meansyou can work on a windows PC for hoursand after that open up your Macbook andtest your project on iPad in a matter ofminutes.

• Assets (sounds as well as images, videoand 3D models) can be imported with dragand drop. They will automatically be con-verted to a usable format. Once imported,they can even be dragged onto your codewherever you want to use them.

• Because code is written in C++, almostevery other C++ library can be added toyour project. If you have a source licenseyou can even add the functionality youneed directly to the Esenthel engine.

• Support is really good. Almost every timewe had a question, a solution was givenby the engine developer within a day. Re-quests for (small) new features are oftenhandled within a few weeks.

• The price is the lowest we could find. Acomplete source license costs only 13.75Euro a month. The binary license is only6.88 Euro a month.

3. Audio

A game engine is a good option for cross-platform mobile development. It even en-ables us to use the most flexible audio so-lution there is on mobile platforms: OpenSLES. Unfortunately, this most flexible solutionis not very flexible at all. Here are some lim-itations:

10http://www.esenthel.com

9

• A hard limit of 32 simultaneous sounds.This is not much once you start to usesounds as a software sampler. Even amonophonic instrument needs at least twoor three sounds to sound natural.

• Sound playback speed is limited from 0.5to 2.0. This means you cannot create asoftware sampler from one sound and haveit cover more than 2 octaves.

• Reverb is not available. We certainly don’twant to add a reverb to everything, but itcan be useful.

• No MIDI support. Although OpenSL ESalso defines MIDI support, the Androidimplementation lacks MIDI.

On top of that, the OpenSL ES interfaceis unwieldy at least: it doesn’t come withlots of examples and only has a primitive CAPI. The latter makes your code particularlyprone to errors. Because of this we wouldrecommend the same strategy as we usedfor targeting multiple platforms: let some-one else worry about it. Or at least we don’twant to worry about it while we’re workingon an application.

3.1 ImaginationIt is our view that art is the realisation ofour imagination. And while there will al-ways be limits to what is possible, we feelthat the current state of sound programminginterfaces really limits what we can realize.Hardware development has taken an enor-mous leap over that last years: we can domore on a recent phone than was possible ona computer fifteen years ago.

In general, software development kept upwith that. The game engines mentioned ear-lier in this article all provide an easy touse and flexible interface to create softwarethat uses graphics in new and fascinatingways. Unfortunately, sound doesn’t seem tobe very important to most companies. Un-derstandable too, because most sound artists

won’t request much more than playing pre-recorded audio tracks.

To improve the current situation, soundartists should think of new ways to work withsound. But as long as the technology doesn’tallow for real creativity, this won’t happen.A much needed improvement of audio pro-gramming possibilities should go hand inhand with sound artists working with thosenew possibilities.

3.2 a WishlistSo what exactly would be the ideal sound li-brary? What features should it contain? Asa programmer creating audio art, we thinkthat were in a good position to imagine whatwed like to work with.

Firstly, there are some general program-ming requirements:

• It should be fast. There is no use for asound engine which takes up half of theavailable CPU power. Preferably all DSPcalculations are divided over all availableprocessors. This speed requirement is im-portant, because it rules out all program-ming languages with automatic garbagecollections.

• It should work on all major platforms,without application code modifications.At the very least on Windows, Linux andAndroid. Mac and iOS would be nice also,for as long as they still have a marketshare.

• It should be understandable without look-ing at the documentation all the time. Ifyou look at a function’s name and it’s notclear what it’s supposed to do, the func-tion name is badly chosen. On the otherhand, documentation must be extensiveand there should be plenty of tutorials.

• Frequently used concepts should be readyfor use. I do not want to be able to createa looping sound object that can be playedat variable speed, I want to be there.

10

The same goes for mixing channels, reverband most established DSP functions. Youwould think this goes without saying, buteveryone who has used Pure Data knowsthat even for starting and pausing a sim-ple sound you have to put in an extra rampobject to avoid audio clicks. Since we allneed to do that, I think it would be betterto implement that sort of thing right away,

at the engine level.

• It should be easy to use right, and difficultto use wrong.

While the last one is an established pro-gramming mantra, Were still surprised howcomplicated most existing sound engines are.As an example, here is the recommended wayto initialize the FMOD sound engine:

UInt version;

_errorCheck(_system->getVersion(&version));

if (version < FMOD_VERSION) Exit("FMOD: outdated fmod dll.");

5 // get the number of available audio drivers

_errorCheck(_system->getNumDrivers(&_numDrivers));

if (_numDrivers == 0) {

// continue without sound

_errorCheck(_system->setOutput(FMOD_OUTPUTTYPE_NOSOUND));

10 } else {

// revert to default driver if the ID is not right

if (driverID > _numDrivers) driverID = 0;

// Get driver info

15 _errorCheck(_system->getDriverCaps(driverID, &_caps, 0, &_speakermode));

// Set speakermode

_errorCheck(_system->setSpeakerMode(_speakermode));

if (_caps & FMOD_CAPS_HARDWARE_EMULATED) {

20 _errorCheck(_system->setDSPBufferSize(1024, 10));

}

// something wrong with sigmatel audio drivers

_errorCheck(_system->getDriverInfo(driverID, _driverName, 256, 0));

25 if(strstr(_driverName, "SigmaTel")) {

_errorCheck(_system->setSoftwareFormat(48000, FMOD_SOUND_FORMAT_PCMFLOAT, 0, 0,

FMOD_DSP_RESAMPLER_LINEAR));

}

}

30 // try initializing the sound system

FMOD_RESULT r = _system->init(1000, FMOD_INIT_NORMAL, 0);

// if something went wrong, revert to play stereo mode

if (r == FMOD_ERR_OUTPUT_CREATEBUFFER) {

35 _errorCheck(_system->setSpeakerMode(FMOD_SPEAKERMODE_STEREO));

_errorCheck(_system->init(1000, FMOD_INIT_NORMAL, 0));

}

_system->set3DSettings(_dopplerscale, _distancefactor, _rolloffscale);

11

This should be enough to scare away ev-eryone but the most proficient programmers.And even those will have to program witha manual on a second screen. We think itshould be something like this:

SOUND::System.init();

If a programmer is not satisfied with thedefault options, he should be able to changethose. Reasonable defaults go a long waythough. If, like in the example above, some-thing is known to be wrong with sigmatelaudio drivers, this should be handled by theengine, not by the application programmer.

With these programming requirements inmind, we can move on to the audio features.In general, there are several categories ofsound manipulation that can contribute tothe way we’d like to work with audio:

• 3D sound positioning. We would like toposition and move sounds in a virtualworld. This can be interesting for loca-tion based sound art, but it also opens uppossibilities for many other concepts we’reworking on. In general, this is what soundengines for games use.

• Low level DSP functionality. We’d like tobe able to manipulate every audio streamat sample level. Pure Data is a good ex-ample of software to do this kind of thing.

• Software synths and MIDI support. Whilesound positioning and manipulation isvery interesting, we still believe in motives,melodies and chords. (Although not neces-sarily in a conventional way.) The idea ofa playing instrument which you can giveinstructions still has merit. Sequencersand dedicated software synths aside, thisfunctionality is non existent in most soundsoftware.

• Macro structure (or composition) func-tions. Most sound libraries are occupiedwith the ’here and now’. We think it couldbe very interesting to include functions

that define how sound evolves on a largerscale, directly into the sound library. Thiskind of functionality can be found in pro-grams like GMT and the Composers Desk-top Project.

Needless to say we have not found a soundlibrary that can do all of this. And whileit is not impossible to work with several li-braries together in your software, doing sois quite complicated. You will need to com-pile every library for all targeted platforms,compile their dependencies and write a lot offunctions just to pass data from one libraryto another.

Nonetheless, when new and innovative au-dio art is our goal, we will gain a lot from aneasy to use library which has all this to offer.

3.3 the YSE Sound EngineLast year we started development on a newsound engine: YSE. With the list above inmind, we think that even a partially com-pleted engine can help us create audio inways we are not able to do today.

Since we want a fast library, but with aneasy to use syntax, we didn’t write it in C.While C does the fastest code in most cir-cumstances, it isn’t the easiest language foreveryday use. As a compromise we usedC++11. This allows us to use C code for themost crucial parts of the engine, while stillproviding a solid and easy to use interface.

Because we are also working on the Attr-Xproject (a virtual online world about soundmanipulation) we started with 3D soundplacement and sound synthesis. At the endof the year we had a first stable version avail-able which is already implemented in Attr-X.

This first version of our library only sup-ports Windows, Mac and Linux. But nowthat we get more and more involved in mo-bile projects we made a lot of changes to haveit compile on Android and iOS. This stage isalso nearly done. We have not published astable release just yet, but that is a matterof a few months at most.

12

Right now, the following features are im-plemented:

• 3D positioning of all sound objects and ef-fects.

• Nearly all functionality provided bycommercial sounds engines for gaming.(Doppler effect, reverb, rolloff scaling,sound virtualization, etc.)

• Flexible speaker setup. Contrary to gam-ing solutions we are not only interested onplaying audio on a home surround system.Audio installation art can require any pos-sible speaker setup. With YSE you can ad-just all speaker positions. All 3D soundswill be outputted to the correct speakers,according to their position.

• Basic DSP functions, comparable to thevanilla version of Pure Data. DSP objectscan be used as a sound source in a 3D envi-ronment, as well as a sound filter attachedto a sound, a channel or a location in 3D.

• Full resource management. You don’thave to worry about acquiring and releas-ing memory. Also, sound files that are nolonger used will be erased from memory bythe engine.

• Audio calculations are spread out overmultiple cores. The library is completelylock free for the best performance.

Some stress tests were done on a Windows8 pc with a quad core i7 processor. Withthis set-up the engine can render about 2500sounds simultaneously11. With virtualiza-tion turned on, we can have 80.000 sounds inmemory, moving around in a 3D space andhave the engine play the 100 most importantones. On a Nexus 5 mobile phone with an-droid 4.4, we were still able to play about700 sounds at the same time.

While our implementation is far from com-plete we do think some of the hardest workis done now. We will continue to work onmore advanced functionality, adding soft-ware synths and macro functions, but at thesame time, we expect to release our first soft-ware based on the engine this year.

YSE is open source, released with theEclipse license. This means it can be used byother open source projects, as well as in com-mercial products. We hope that by sharingthe source, contributions and improvementswill follow over time.

4. Conclusion

Location based media art is emerging for afew years now. We think that projects oftenuse only a few of the available techniquesto create an engaging user experience. Ifwe want to keep our audience’s attention,we will have to keep innovating. But inte-grating new technologies, complicated inter-actions and engaging interfaces requires a lotmore programming. On top of that, the au-dience is divided over several platforms, themost important ones being Android and iOSright now. This too requires extra work fromthe programmer.

Because of these reasons we think it is bestto use an integrated, cross-platform toolkitfor development. And although none of themis specifically intended for art projects, gameengines offer much we can use.

If a game engine can be used in combina-tion with a sound engine that opens up newways of working with audio, artists will beable to create even more new and innovat-ing mobile art projects. And because suchan engine does not exist, we’ve set a modestfirst step in developing one, hoping to inspireothers to work with us.

11Of course we never intend to actually play that much sounds at the same time. This merely shows thatthere is plenty of headroom to do other calculations in our software.

13