1

AVUI:DesigningaToolkitforAudiovisualInterfaces

NunoN.Correia

Madeira-ITI,U.MadeiraFunchal,Portugal

nuno.correia@staff.uma.pt

AtauTanakaGoldsmiths,UniversityofLondon

London,UKa.tanaka@gold.ac.uk

AbstractThe combined use of sound and image has a rich history, from audiovisual artworks toresearch exploring the potential of data visualization and sonification. However, we lackstandard tools or guidelines for audiovisual (AV) interaction design, particularly for liveperformance.WeproposetheAVUI(AudioVisualUserInterface),wheresoundandimageareusedtogetherinacohesivewayintheinterface;andanenablingtechnology,theofxAVUItoolkit.AVUIguidelinesandofxAVUIweredevelopedinathree-stageprocess,togetherwithAVproducers:1)participatorydesignactivities;2)prototypedevelopment;3)encapsulationof prototype as a plug-in, evaluation, and roll out. Best practices identified include:reconfigurable interfaces andmappings; object-oriented packaging of AV and UI; diversesoundvisualization;flexiblemediamanipulationandmanagement.Thetoolkitandamobileapp developed using it have been released as open-source. Guidelines and toolkitdemonstrate the potential of AVUI and offer designers a convenient framework for AVinteractiondesign.

ACMClassificationKeywordsH.5.2UserInterfaces:Auditory(non-speech)feedback;H.5.2UserInterfaces:Graphicaluserinterfaces (GUI); H.5.2 User Interfaces: Prototyping; H.5.2 User Interfaces: User-centereddesign

AuthorKeywordsUser interface; toolkit; interfacebuilder; audiovisual; crossmodal interaction; prototyping;participatorydesign;interactiondesign;hackathons.

2

IntroductionThecombinationofaudiowith imagehasa long tradition, fromcolororgansusedby thecomposer Scriabin in the early 20th century [27] to the pioneering computer graphicsexplorationsofJohnWhitneyinthe1960s[26].Theadventofpowerfulpersonalcomputersfor media manipulation, from the 1990s, gave further impulse to audiovisual (AV)performance[32].These artistic explorations mirror how the brain deals with multi-sensorial information.Researchonsensorysubstitutionhasexploredhowthebrainreplacesfunctionsofonesensebyanother[1].Theperceptionofflashinglightscanbemanipulatedbysound:asingleflashof light canbe seenas consistingof two flashes ifdisplayed simultaneouslywithmultiplesoundsignals [34]. In theMcGurkeffect, theperceptionofanauditoryphonemechangesdependingon the image [24].An important factor forcrossmodality is congruency–non-arbitrary associations between differentmodalities. A congruent AV display can result inbetterperformanceandhigherengagementthanarbitraryassociationsbetweensoundandimage[25].Sound and image have been studied in HCI in different application areas, includingaccessibility in assistive displays [11], improvement of task accuracy in driving [31],enjoyabilityandperformanceingames[5,25].Despitethepotentialforfacilitatingusabilityandengagement,thereisa lackofdesignguidelinesandstandardtoolsforAVinteractiondesign.Specifically,currentsolutionsforAVperformancethatfacilitateUIintegrationwithcontentarelaborioustoimplement,andlackaestheticconcernsregardingcoherenceofUIand visuals. Interface design for AV performance is mostly subjective, and there are noestablished best practices. These best practices would benefit performers, audience,softwaredevelopers,interactiondesigners,researchersandstudents.WeproposetheAVUI(AudioVisualUserInterface)wheretheinteractionofsoundandimagein the interface extends the concept of GUIs. We seek to 1) leverage practices in AVperformanceforsketchesandprototypes,usingparticipatorydesignmethods;2)implementbestpracticesintoaconsolidatedprototype;3)proposeguidelinesandasoftwaretoolkittoallowdesignerstoeasilyintegratesoundandimageintheUIoffuturesystemsandproducts.This paper reports on the multi-stage design, development, release and evaluation of asoftwaretoolkit,ofxAVUI.Wepresentrelatedwork;theparticipatorydesignandqualitativemethods used; and their results; followed by AVUI guidelines proposed, discussion andconclusions.

RelatedWorkAVPerformanceandToolsAVperformancecombineslivemanipulationofsoundandimage[4].ThisdistinguishesitfromVJing(VideoJockeyperformance),whereavisualperformeraccompaniesamusician[8].A

3

numberofartistsareconcernedwithcreatinginterfacesandsystemsforAVperformance.Levin developed painterly interfaces for audiovisual performance [20]. Magnusson usesabstractGUIstorepresentmusicalstructures[23].Iwaicreatesplayfulpieces,crossinggenresbetweengame,installationandperformance[29].Most commercial VJ software, such as Modul8 (http://www.modul8.ch), focus on videoplaybackandmanipulation,withlimitedgenerativegraphicscapabilities,andonly“fairlylow-levelmusicalfeatures”[33].AVperformers“oftenrelyonbuildingtheirownsystems”[33]with coding frameworks such as openFrameworks (OF) (http://openframeworks.cc).Therefore,animportantelementofVJingandAVperformanceistheuseofDoItYourself(DIY)tools[32].Thisrequires“ahighleveloftechnicalabilityonthepartoftheuser”[33].Solutions such as Processing (http://processing.org) offer sound toolkits, which containvisualization modules (ex: Minim), and also offer GUI modules (ex: Control P5). Similarexamplescouldbegivenforothercreativecodingenvironments.ButalthoughtheseUIandsound/visualizationmodulescanbecombined,thisislaborious,andtheyarenotintegratedout of the box, neither functionally nor aesthetically. They do not offer easy ways toimplementan“AV+UI”solution,noraGUIdesignedtobeintegratedwithvisuals.Weaimtoprovideaneaseofimplementation,andhighlevelofintegration.AVSystemsandInteractionDesignSchofield et al. created Cinejack for “directing narrative video through live musicalperformance” [33], in collaborationwith artists. InMelodicBrush, a user-centered designapproach is adopted to design a tabletop AV system linking calligraphy and music [15].Wiethoff and Gehring created an interactive media façade system through an iterativeapproach:keydatacollection;userresearch;dataanalysis;designconcepts;andexperienceconcepts [35]. These studies have usedmulti-stage, user-centered approaches,whichweadoptedinourwork.However,theyeachonlyevaluatedonesystem,makinggeneralizationdifficult.Weapplythesemethodstoevaluatealargenumberofprojectsinordertogleandesigninsightsacrossmultiplesystems.TheNewInterfacesforMusicalExpression(NIME)communityhasbeenactiveinstudyingthecombinationofvisuals,soundandinteractiondesign.TheworkofJordà,suchasFMOLandReactable, isrelevantduetotheinterplayofinteractionandsoundvisualizationstrategies[17].TheauthorsofresidUUm,oneoftheprototypesresultingfromthecurrentresearch,havepresentedtheirperformanceapproachesforAVgenerationandaudiencevisibility[30].RouagesisanotherAVprojectthatisconcernedwithaudienceunderstanding[2].HookhasdevelopedaninterfaceforVJperformances,Waves[14].HealsostudiedVJingfromtheviewpointoftheperformer[13]andidentifiedthreemainthemes:aspirational,liveandinteraction. Within interaction, he identifies the following sub-themes: constraininginteractions; haptically direct; parallel interaction; immediacy; manipulable media;reconfigurable interfaces; and visible interaction. While he focuses on video content, herecognizestheneedforgenerativemediatools.WewilluseHook'sframeworkforourstudy,strengthening thepotential generalizabilityof theworkbybuildinguponpriorqualitativefindings.

4

MethodsTheresearchtookplaceovera2-yearperiodandinvolvedapreliminarystudyandthreemainstages (Figure 1): Pre) scoping interviews andbrainstormingworkshop; 1) hackathons forprototypedevelopment;2)a“consolidationprototype”gatheringbestpracticesandexpertevaluation;3)creationofasoftwaredevelopment toolkit for facilitating integrationofAVwithUI.Thetoolkitwasreleasedasopen-source, tested inahackathonandwith internetusers,andfurthertestedbyrebuildingthefinalprototypefromStage2asapubliclyreleasedproduct using the toolkit. The iterative cycle enabled “a dynamic process of invention,distributedacrossevents”[9].Hackathonsandinterviews(andsubsequentthematicanalysis)wereusedasmethodsthroughoutthestudies.

Figure1.Summaryofthefourcyclesoftheresearch:preliminarystudy;stages1,2and3.Workshopsandhackathonsinblue,interviewsinwhiteanddevelopmentphasesinorange.Stage1hadtwoiterations.

HackathonsHackathons are coding events in DIY communities where “small teams produce workingsoftwareprototypesinashorttimeperiod”andtheseeventsareoftencenteredaroundacommon theme or technology [18]. The hackathon challenge is an important part of themethod.Itsetsacommontaskinamotivatingwaytoparticipants,makingahackathon“amoment of design” [16]. In a hackathon, solutions are “conceived in response to thosechallenges” [21]. These elements make it a fun, easy to understand technique forparticipatorydesignandcodedevelopment.ThematicAnalysisWeconductedthematicanalysesofinterviewsinthethreestudies,basedontechniquesin[3].Wecodedtheresponsesbasedonemergingpatternsandissuesarising,thencollatedthecodes into potential themes. We used Hook's themes as a basis for our coding. Wecomplementedthistheoreticanalysisapproachwithaninductiveanalysisindependentofanypre-existingcodingframe.Thisallowedustobuildonpriorwork,contributeournewinsights,andachieveabalanced,thoroughandinclusivestructureofmainthemesandsub-themes.

PreliminaryStudy:Interviews,BrainstormingInourpreliminarystudy[7],weconductedinterviewswith12audiovisualperformers,askingthemabouttheirpractice,theirtools,andtheirneedsanddesiresasperformers.Theanalysisof the interviews brought forth a series of key issues: modularity, flexibility andreconfigurability; ease of hardware/software integration; instrument-like expressivity andfluidity; integration of environmental elements; generative capabilities and diversity;

5

communication of process to the audience; reliability and speed. These concepts on thewhole match and confirm the issues identified by Hook under the theme interaction.Generativecapabilitiesanddiversityconnectstotheirforwardlookingthemeofaspiration,andtheneedforavisualequivalenttosoundsynthesizers.The12intervieweesprovideduswith a group of experts that we would consult throughout the different studies of theresearch–theevaluatorsofStages1and2werefromthissamegroup.Theideasfromtheinterviewstheninformedabrainstormingworkshop,with19participants(including two from the previous interview stage). The one-day workshop structure wascomprisedoftwoparts:thefirstoneadoptingthe“bootlegging”ideagenerationtechnique[12].Forpart2,weextendedthiswithamorefocused,structuredre-examinationofideasfrompart1,whichwecalled“Re-boot”.Thefivebreakoutgroupsproducedfivesketches(storyboardsandwireframes)ofproceduralaudiovisual performance tools. Two sketches, Gestural Touchscreen andMeta/Vis, wereparticularlysuccessfulinaddressingthechallengessetoutintheworkshop.Bothrelyontheexpressive potential of multitouch interaction, employing different solutions forreconfigurability: the formerallows for loadingandmanipulatingvectorgraphics,and thelatteradoptsasimplifieddata-flowmechanism.Projectdescriptionsandsketchesareseenathttp://www.gen-av.org/sketches/.

Stage1:ParticipatoryDesignofPrototypesHackathonandHackChallengeUsingthekeythemesandthesketchesfromthepreliminarystudyasinputandinspiration,werantwohackathonsinaniterativecycle,Gen.AV1andGen.AV2.TheobjectivewastoleverageknowledgefromAVperformersintoprototypescombiningAVandUI,wherebestpracticescouldbeadoptedinafutureAVUItoolkit.Wesentoutacallforparticipation,withcodingknowledgeasprerequisite, and interviewedapplicants.Eachhackathon tookplaceovertwodays.BothGen.AV1and2 followed the samestructure:1) Introduction:apresentationon thepreviousstagesofthestudyandresultsachievedsofar,goalsandstructureoftheworkshop;2)Conceptualizationandsketching;and3)Softwaredevelopment.23participantstookpartinGen.AV1 (five femaleand18male).Gen.AV2had13participants (two femaleand11male),threeofwhomhadtakenpartinGen.AV1.Wedividedparticipantsintofive(Gen.AV1)andsix(Gen.AV2)groups,distributingpriorprogrammingexperienceevenlyacrossgroups.Wecreatedhackchallengesbasedonkeyconclusions(inparenthesis)fromthepreliminarystudy[7].Theywere:1)computer-generatedsoundandvisuals(generativecapabilitiesanddiversity); 2) powerful and fluid manipulation – “like an instrument” (instrument-likeexpressivityandfluidity);3)single-screen–whattheperformersees iswhattheaudiencesees (communication of process to the audience); and 4) possibility to reconfigure theinterface(modularity,flexibilityandreconfigurability).Theresultingprojectswerepresented

6

intwopublicperformances.FiveprojectswereshowcasedintheGen.AV1performance,andsixinGen.AV2.Eachgroupproduceda10minuteperformance.ProjectsWepresentthefiveprojectsfromGen.AV1.ABPisananimationengineandsoundvisualizer,wheretheusercandefinecolor,geometryandanimationparameters.IndrawSynth,aGUIallowscontrollingsoundandimage–userscandrawvectorshapesandselectcolors,whicharesonifiedbyasynthesisengine.EsoterionUniverseconsistsofa3Dspacethatcanbefilledwithplanet-likeaudiovisualobjects,eachcontainingaGUItomodifytheirvisualandsonicproperties.GS.aviisaninstrumentthatgeneratescontinuousspatialvisualizationsandmusicfromthegesturalinputofaperformer.Modulantallowsfordrawingimages,usingpaintbrushtypeoftools,whicharethensonified.SixprojectswerebuiltduringGen.AV2.Butterflyisanaudiovisualizerwhichallowsforthecombination and control of four audio synthesizers, by means of manipulating iconsdistributedinfourXYpadsonthescreen.CantorDustgenerates,displays,andsonifiesCantorsettypefractalsassoundandvisuals.EUGfurtherdevelopsEsoterionUniversefromGen.AV1,adding3DgesturalcontrolwithaLeapmotionsensor.OnTheTapplayswiththetactile,analogfeeloftappingsurfacesasinteractioninput,capturedasaudio.residUUmallowsforthe creation and manipulation of AV particles, with a variable lifespan, by clicking anddraggingonthescreen.Watcreatesachaotic3Dtexturebasedoncellularautomata(Figure2). All the projects were uploaded to GitHub for download or source-codemodification,accessiblefromhttp://www.gen-av.org.Inadditiontothecode,theprojectdescriptionsareavailablefromthesamelink,facilitatingrunningandreplicatingtheprojects.

Figure2.Stage1:Projectsfromthehackathons–lefttoright,thentoptobottom:hackathon,ABP,drawSynth,EsoterionUniverse,GS.avi,Modulant,Butterfly,CantorDust,EUG,OnTheTap,residUUm,Wat.

7

ExpertInterviewsAftertheperformances,theprojectsweretestedforeaseofinstallationandrobustness.Sixprojectswerechosen:EsoterionUniverse,GS.aviandModulantfromGen.AV1;andButterfly,residUUmandWatfromGen.AV2.Thesewereevaluatedbyexpertreviewers,establishedaudiovisualartistswhohadtakenpartinthepreliminarystudyinterviews.Eachexpertwasgiventwoprojectsforreview,andatleastoneweektimetopracticewiththesoftware.Thus,eachprojectwasevaluatedtwice(projectevaluatorsE1andE2).Wethenconductedsemi-structuredinterviewswiththereviewers,lastinganaverageof15minutesperproject.Theyservedtofollowuponthekey issuesemergingfromthepreliminarystage.Thequestionsaddressed:1)theAVcontentandtherelationshipbetweensoundandimage;2)easeofuseofthesoftware;3)fluidity,AVmanipulabilityandbehaviorasan“instrument”;4)flexibilityandreconfigurabilityoftheinterface;5)potentialusefulnessforotherartistsandperformers.Weconductedathematicanalysisoftheinterviews.Fromthisanalysis,threemainthemesemerged:Experience, InterfacesandMedia.Weretainedthesethemesforouranalysesofthe different stages of the research. Our starting point were Hook's themes related tointeraction:constraininginteractions(importanceofconstraintsandfocus);hapticallydirect(physical connection); parallel interaction (simultaneous control of multiple parameters);immediacy (immediate response from the software); manipulable media (powerful andvaried manipulation of media); reconfigurable interfaces (reorganize controls to fit aparticularperformance);andvisibleinteraction(makeinteractionvisibletoanaudience)[13].Immediacywasapre-requisitefortheselectionoftheprojects.Constraininginteractionwasnotdetected.Wedecidedtomergehapticallydirectandparallelinteraction,astheywouldappearcombinedinourdata.Wesplitinteractionintotwonewmainthemes:experienceandinterface.Additionalthemesrelatedtocontentemerged,originatingthenewmainthememedia.ExperienceVisible interaction:Twoof theevaluators, in theirownpractice, prefernottoconveytheinterfacetotheaudience,andwishtohaveaseparatescreenwiththeGUIfortheperformer(GS.avi,E1;Modulant,E2),forthreemainreasons:1)thevisualoutputcouldbere-routedwithoutGUI toother software foradditionalmanipulation (GS.avi, E2);2) the interface issomethingtheaudiencemaynotwanttosee;and3)havingaseparatescreenwouldallowforamorecomplexinterfacefortheperformer(Wat,E2).AnotherevaluatorisinterestedinshowingtheUItotheaudienceand“conveyingtheperformer'scontrolon-screen”aspartoftheexperience(residUUm,E2).Takingthatapproachwouldallowfor“visuallyreflectingthatagencyontothescreen”,makingitunderstandableforanaudience(Butterfly,E2).Haptic and parallel: The reviewers confirmed the desire for interaction that “provides asensationakintobeingindirectcontactortouchingandmoldingmedia”[13].Theprojectsdid not allow for either haptic or parallel interaction. Multitouch tablets and hardwarecontrollerswerementionedasmeans toachieveaparallel interaction.Compatibilitywithhardwarecontrollersandtabletsisdesiredtoachieveparallelinteraction:controllerswouldallow for the physicality and “flexibility of an instrument” (Esoterion Universe, E1) and

8

interaction“inatrackingpadonthecomputerisconfusing”,withatablet-basedapproachbeingsuggested(GS.avi,E2).Object-oriented:Whenauditoryandvisualdomainsarecombined,audiovisualobjectscanemerge,ifsimultaneityandaplausiblecommoncauseoccur[19].Threeprojectsfollowedanobject-orientedapproach,bygroupingaudioandvisualcontentintodistinguishableentities(EsoterionUniverse,ButterflyandresidUUm).Inthefirsttwocases,aGUIwasoverlaidonthevisuals for continuousmanipulation. In the thirdproject theopportunity formanipulationoccursonlyatthegenesisoftheobject.Theobjectapproachwasconsideredasbeingfruitful(EsoterionUniverse,E2).InterfaceReconfigurable interfaces: The reconfigurationofUIbecomespossibleonlybyediting thecode,which requiresspecific technicalknowledge (EsoterionUniverse,E1).Someprojectsorganizedthecodeinordertomakeiteasiertoreconfigure:“it'sveryeasytoaddyourownsynths”(Butterfly,E1).Interfacemappings:Insomeinstances,acomplexone-to-manymappingofinterfacetomediaparameterswasconsideredsuccessful(Butterfly,E2).EvaluatorsfeltthatmoreparametersshouldbecontrolledfromtheUI,resultingininsufficientmapping(Butterfly,E2;Wat,E2).Scalabilityoflayoutisdesired,asitwouldallowforadditionalUIelements(Butterfly,E1).Interface clarity: The lack of a parameter space in UI elements – an indication of theparameter range, and the current status – was considered problematic in some projects(residUUm,E2).Interfaceaesthetics:Thegesturalaspectofoneprojectwasseenasinnovativeandappealing(GS.avi,E1).Itwassuggestedthatitcouldbecomemoreintegratedinthevisualsbyvisualizingthegestures(GS.avi,E1).Thevisualdesignofthe interface isconsideredtobeevenmoreimportantwhenconveyedtoanaudience.Insomecases,thisdesignwasconsideredtobeunappealingtobeshown(Modulant,E2).OnereviewerconsidersthattheUIitselfshouldbedynamic,animatedinresponsetothesound(Butterfly,E2).Oneoftheprojectsadoptsalogicof interactive quadrants with XY pads, which was considered to be original and clear(Butterfly,E1).MediaManipulablemedia: Some projectswere considered to produce outcomeswith a narrowrangeofdiversity(EsoterionUniverse,E2;Butterfly,E1).Inseveralcases,theprojectsrelyon2Dor3Dspatialmetaphors.Thereisadesireforanexpandablecanvasorscenewherethemediacanbepresentedinandnavigatedthrough.Thisisconsideredtobemissingononeproject (Modulant, E2) and praised for its implementation on another (Wat, E1, E2). Thesatisfactory manipulation possibilities of some projects lead them to be considered“instruments”becauseoftheirfluidity(Wat,E1;Modulant,E1).

9

Generativemedia:Differentevaluatorsappreciated differentdegreesofrandomness.Thegenerativeaspectofsomeprojectswasconsideredtobetoochaotic(GS.avi,E1;residUUm,E2).Inthebalancebetweengenerativeelementsandcontrol,thelatterisseenasthepriority.Butacertaindegreeofrandomnessisdesirable,andconsideredtobemissinginsomecases(Modulant,E1).Mediamanagement:Theoptiontoloadfilesinsomeoftheprojectsisappreciated(GS.avi,E2;Modulant,E1).Runtimeloadingofcontent isdesired(GS.avi,E1).Real-timesharingofmedia between applications in the same device, using utilities such as Syphon(http://syphon.v002.info)iswishedfor(EsoterionUniverse,E1).Thepossibilityofaccessingnetworkedcontentisalsosuggested(EsoterionUniverse,E2).AudienceStudyIn order to study audience understanding of the performers' actions,we asked audiencemembersofthetwoperformancestofillinaquestionnaireaboutthedifferentprojectsfromGen.AV1and2(withtheexceptionofDrawSynth,alastminuteaddition).Respectively45and34respondentsansweredthequestionnaire.Thequestionaskedwas:“Didyoufindtheconnectionbetweentheperformer'sactionsandtheaudiovisualresultunderstandable?”,onascaleof1to5.ProjectsEsoterionUniverseandModulantfromGen.AV1,andButterfly,EUGandresidUUmfromGen.AV2obtainedthebestresults(Modulantwithamedianof5,theotherswithamedianof4).Thefiveprojectsthatachievedthebestresultsmakevisibleboththeinterfaceandtheparameterspace.CantorDustandOnTheTap,bothwithamedianof3,implementonlyoneoftheseaspects(visibilityofparameterspace inCantorDust)oronlytemporarily show them (OnTheTap). The remaining projects,with amedian of 2, employneither.Theseobservationsinformedourdesignprinciples.

Stage2:AVZonesConsolidationPrototypePrototypeDesignTheresultsfromStage1fedintodesignguidelinesforafinalprototype,aniPadappforAVperformance entitled AV Zones. It has been released as open-source(https://github.com/AVUIs/AVZones-beta).Itadoptstheobject-orientedconceptof“zones”:rectangular areas that incorporateUI elements producing andmanipulating sound, and avisualizationofthatsound.Theapphasthreeverticalzones,eachwiththreeXYpadsforaudiomanipulation,controlling:pitchshift,delayandfilter.Eachzonehasasequencer,whichcanrecordtouchinformationandvisualizeit.Thereareninesoundsavailableperzone,whichcanbe switchedat runtime, and replaced in the code.Different touch inputs createdifferentresults: tapping for triggering sounds; touchmovement formanipulating the sound; two-fingertaptoswitchonandoff;anddoubletaptotriggerspecialfunction–menuorsequencer(Figure3).TheappwasdevelopedusingtheOFenvironmentandtheMaximilianaudiolibrary(https://github.com/micknoise/Maximilian).Bothareopen-sourceandcross-platform.

10

Figure3.Stage2:AVZonesprototype.InitialTests-PerformancesWe testedAVZones in“realworld” settings: fourpublicperformancesand twodemos inconferences.Inaperformance,onlyaniPadisusedforaudiovisuals.Whattheperformerseesisalsowhatisprojectedtotheaudience.Theinterfaceisshownonthescreen,withtouchpoints being represented by white circles. We made minor improvements betweenperformances. For example, a sequencer was added due to the difficulty of interactingsimultaneouslywiththethreezones,andtheneedtoautomatesomeoftheprocessesbyrecordingthem.ExpertInterviewsWefollowedasimilarevaluationprocedurethaninStage1:weinstalledAVZonesiniPadsand handed them to three evaluators (E1, E2, E3) from our initial expert group. Theinterviewees tried theapp forat leastoneweek.Wethen ransemi-structured interviewslastingonaverage30minutes.WeusedthesamequestionsasStage1.Weranathematicanalysis,maintaining the threemain themes:media, interface andexperience.More sub-themesemerged:soundvisualization(withinmedia);constraininginteraction–athemethathad existed in Hook's analysis but had not appeared in Stage 1; and playfulness (withinexperiencemaintheme).ExperienceConstraining interaction: Two evaluators were satisfied with the prototype's designconstraints andminimalism, stating that “it's nice tohave limitations”, having a “minimalsimplicity”waspleasing,andits“reducednature”madeit“veryappropriateforalivetool”(E1,E3).Visibleinteraction:OneevaluatorwouldliketobeabletohidetheUI,completelyorpartially,andaddedthatbyseparatingwhattheaudienceandtheperformersee,moreUIelementscouldbeaddedontheperformerside(E1).Anotherrespondentissatisfiedthattheaudiencecanseewhattheperformerisdoing,astouchpointsarehighlightedwithwhitecircles,andwouldliketoseemoreinteractionsvisualized,suchassoundeffectmanipulation(E2).

11

Haptic and parallel: One respondent was satisfied with the number of zones andsimultaneouscontrolelements(E2),whereasotherswouldliketoaddanexternalhardwareMIDIcontroller(E1)oranothertablet,creatingadualsetup(E3).Object-oriented:One respondentwaspleasedwith thenotionof zonesand theway theyoperate,statingthatitwasagoodconceptanddesign(E2).Playfulness: The application was considered to be playful – one respondent mentionedmultipletimesthatheenjoyedplayingwithit,thatitwas“fun”(E1).InterfaceInterface aesthetics: One evaluatorwas very pleasedwith the interface aesthetics of thesequencerfunctionality,statingthatitlookslikea“visualmusiccomposition”(E1).Anotherrespondent considers that more work could be done in terms of visualizing additionalprocessesinthesoftware,suchasloadingorchoosingsounds(E2).Interface clarity: The prototype suffers from hidden discoverability issues. Several of thefunctionalitiesareactivatedbydifferenttypesoftouchinteraction,notapparentintheUI.Testers had problems activating these, despite the documentation provided (E1, E2, E3).Evaluatorscomplainedthatitwashardtounderstandwhattodonext(E2)leadingtogettingoccasionallystuck(E1).Reconfigurable interfaces: One respondent in particular was interested in addingreconfigurability options, such as allowing for extending functionalities with software“plugins”thatotherscouldbuild.Anothersuggestionwashavingthepossibilityofgroupingzones and nesting them - thiswould facilitate scaling of zoneswithout overcrowding thescreen (E1). One of the respondents suggested adding a back end with substantialconfigurationoptions(E3).MediaManipulablemedia:Onerespondentconsidersthatthesoftwareis“areallyusefullivetool”(E3).Thetwoaudioeffects,delayandfilter,wereconsideredwellchosen,andhavingthreesimultaneous sounds allows for “enough scope” to maintain a performance (E3). Thesequencer isconsideredan importantelementforthis,as itallowstoautomateonezonewhileinteractingwithothers(E1,E2,E3).Theprototypeisconsideredtoallowfor“adifferentwayofapproachingsound”,less“musical”and“kindofweird”(E1).Tohaveabroaderandmoremusical appeal, two evaluators consider that a stricter timing or “clock” would beimportant(E1,E3).Having“moreauthorshipandasenseofcontrol”(E1)overthesoundisdesired.Onthevisualside, respondentswouldalso liketohavegreatercontrol.Onlyonevisualization type, with “very little visual configurability” (E3) is considered insufficient.Evaluatorswouldliketobeabletohaveothervisualizationsandbeabletomakemorechoicesaboutthem(E1,E2).

12

Mediamanagement:Allevaluatorswouldliketobeabletoloadsoundfiles.Althoughthisispossible,itrequiresmodifyingthecodeandre-installingtheapp,whichisinconvenient.Thepossibilitytorecordsoundsisalsodesired(E2).Oneevaluatorwouldliketobeabletoroutethevisualstoothersoftwareforfurtherprocessing(E1).Sound visualization: One evaluator considers that the approach followed, to visualize theamplitudelevelsoftheaudiobuffer,was“fascinating”and“veryresponsive”,particularlyatslowerspeeds(E1).Theothertwoconsiderthisapproachtobesimplistic,asitdoesnothelpto “understand anything about the sound” (E2). They would rather have a “perceptuallymotivatedapproach”thatwouldbring itclosertothestateoftheart (E2)andatwo-wayinteractionbetweensoundandimage:notjustsoundvisualization,butalsovisualsonification(E3).

Stage3:TheofxAVUIToolkitTookitDesignandDevelopmentTo assist in making the development of AV work more streamlined, to better integrateinterfacesinAVperformances,andtomakeinteractionmoreunderstandableforaudiences,wehavedevelopedatoolkitforcombiningUIwithAVcontent.Wegeneralizedknowledgegained from the previous stages in the development of ofxAVUI, a modular, reusablesoftwaretoolkittofacilitatetheproductionofaudiovisualuserinterfaces.Theevaluationofthe11Stage1prototypesandtheappfromStage2weredistilledintoasetofdesigninsights,which in turn contributed to thedefinitionof the feature set of our toolkit. This led to atechnical specification and software architecture. The design specifications for the toolkitwere, divided bymain themes: 1) experience – allow for parallel and visible interaction;integratesound,imageandUIfollowinganobject-orientedapproach;2)interface–enablereconfigurableinterfaces,withflexiblemappings;ensurebothclarityandaestheticappealofinterface,harmonizedwithvisuals;3)media–allowforpowerfulmediamanipulation,withproceduralcontent;andadoptaflexiblemediamanagement.Fulldesignspecifications,andtheir connection with previous stages of the research can be found at http://www.gen-av.org/avui-design-tables/.Forthedevelopmentofthetoolkit,weagainusedOFandMaximilian.Weorganizedthecodeintothreegroupsofclassfiles:audio,visualsandUI.Eachofthethreegroupshasabaseclass,makingiteasytoextendandtocreateanewaudioprocess,anewvisualizationandanewUItype. The style of the UI is centralized in one class, facilitating the customization of itsappearance.Itwasreleasedasan“add-on”(plug-inforOF),allowingtobeintegratedinotherOFprojectsbydevelopers.Wereleasedtheadd-oninversionsforpersonalcomputerandmobilemultitouchdevices.ofxAVUIwasreleasedasopensource inourGitHubrepository(https://github.com/AVUIs/ofxAVUI). As is customary with OF add-ons, we includedexamples,extensivelycommentedthesourcecode,andadoptedthe“ofx”prefix.Itisnowpart of the main directory for OF add-ons, in the UI category:http://ofxaddons.com/categories/1-gui.

13

Wekept theobject-orientednotionof zones from the app in Stage2, as anorganizationstructureforcombiningAVandUI.Eachzonehasonlyonesoundandonevisualization,toreinforceitsindividualityanditsobjecthood.DifferentUIelementscanbeaddedtoazone:buttons,toggles,XYpads,sliders,rangesliders,drop-downmenusandlabels.Thenumberofzones can be defined, as well as their: size; position; color palette; UI elements. AnyparameterfromtheUIcanbereroutedtoanyaudiofeatureofthezone,oranyotheraspectofthesoftware(forexample,anygraphiconthescreen).WekepttheminimalUIaestheticsoftheprototype.Visualizationisanimportantlinkbetweensoundandimage,thereforeweaddedtwovisualizations,withmoreconfigurationoptions.Wealsofacilitatedthecreationofnewvisualizations,makingthevisualizationmoduleextensible.WeincorporatedtheSyphonprotocol,sothatmediacouldbechanneled,withorwithoutUI,tootherapplications(Figure4).Thesedesignelements,coretothedefinitionofanAVUI,areexposedtotheOFdeveloperthroughhighlevelfunctioncalls,makingintegrationintoanOFprojectstraightforward.

Figure4.Stage3:ExampleofAVUIbuiltwiththetoolkit,usingthreezones,withdifferentUIelementsandvisualizations(explanatorylabelsinwhite).

AsanexampleofeaseofofxAVUI implementation,onlythreelinesofcodeareneededtocreate and configure an AV zone with a button that triggers a sound and associatedvisualization.UIandvisualizationinherittheaestheticpropertiesconfiguredforthezone.EvaluationForafirst, internalvalidationofofxAVUI,webuiltageneralreleaseversionofourStage2prototypeusingtheadd-on.ThisnewversionallowedustoaddressareastoimproveinAVZones identified during Stage 2: interface clarity, media manipulation, and mediamanagement.Intermsofinterfaceclarity,weseparatedthemultiplefunctionsoftheXYpadintodedicatedtogglesandbuttons(on/offtoggle,sequencertoggle,soundfiledrop-downmenu).Regardingmediamanipulation,thesequencercannowrecordandvisualizeadditionalinteractions.Asformediamanagement,userscanaddandmanagesoundfilesviatheAppleiTunesinterface,astandardforiOSapps.ThisfinalversionofAVZoneshasbeenreleasedasopen source on GitHub (https://github.com/AVUIs/AVZones-ofxAVUI), and can be loadedintoaniOSdeviceusingApple'sXcodesoftware.TheofxAVUIadd-onallowedustoeasilyandquicklyredevelopourprototypeandsolveissuesdetectedinStage2(Figure5).

14

Figure5.Stage3:AVZones,rebuiltwithofxAVUI.Toevaluatetheadd-onwithotherdevelopers,weorganizedaone-dayhackathontolookatits ease of use and effectiveness of development. A call was circulated using the samechannelsastheStage1hackathons.Eightparticipantstookpartinthehackathon(fivemale,threefemale).Theirprofilewassimilartothepreviousparticipants:audiovisualperformersanddevelopers.Fouroftheparticipantsmanagedtocompleteasmallprojectduringtheone-day event. The projectswere: FFT/MFCC, audio frequency analyzers and visualizers; StepSequencer for creating rhythmicpatterns; Background Image, for customizing zones; andLisajous and Grid, two additional visualizers. These projects expand the toolkit andwereaddedtotheofxAVUIonlinerepository(Figure6).

Figure6.Stage3:ProjectsfromtheofxAVUIhackathon(firstfour)andonlineevaluators(lasttwo)–lefttoright,thentoptobottom:BackgroundImage,FFT/MFCC,LisajousandGrid,StepSequencer,Multisamplerand

ShaderUI.Inordertoobtainfurtherfeedback,wereachedouttoofxAVUIusersonGitHub.Althoughsoftwaredownloadsareanonymous,12individualshad“starred”therepository–aformoffollowing the repository and its updates on GitHub. Of those 12, eight had contact

15

informationintheirGitHubprofilesandwerecontactedbyus.Wesentanemailaskingiftheywouldliketoparticipateinastudy.Weobtainedfourreplies,andtwodevelopersagreedtoparticipate. They developed two projects: a four-zone Multisampler and ShaderUI, animplementationofsound-responsiveshaders.Theywerealsoaddedtoourrepository.We conducted face-to-face interviewswith the participants in the hackathon (E1-8), andSkypeinterviewswiththetwoonlinedevelopers(E9,E10).Interviewslastedonaverage30minutes.Thesemi-structuredinterviewsaddressed:1)easeofdevelopmentwithofxAVUI;2)itsusefulness;3)theappealofitsdesign;4)resultsachievedandsatisfactionwiththose;5)potentialforfutureuseoftheadd-on.Inourthematicanalysisoftheinterviews,anewmainthemeemerged:development,relatedtoobservationsonprogrammingandcode.Fromthis, threesub-themeswereconsidered:organizationandarchitecture of code, speedand ease of development, andpatchingandbuilding.Thismainthemepre-emptedthereconfigurableinterfacesthemeofthepreviousstudies.Oneadditionalthemeemerged:scenarios,undertheexperiencemaintheme.DevelopmentOrganizationandarchitectureofcode:Mostrespondentsconsideredthatthecodewaswellorganized,with“everythingnicelyintheirrespectivecategories”(E1),andthatitwaseasytosee“howtheobjectsrelatedtoeachother”(E9).Someevaluatorsmentionedthatthecodewaseasytoextend,aseverycategoryhasabaseclass(E3,E10).Onerespondenthighlightedthe flexibility inmapping UI parameters to other zones. Different from other UI toolkits,ofxAVUIis,accordingtoourusers,designed“toreusebitsandpiecesinparticularwaystoinventnewstuff”(E10).Threerespondentswantedmoreabstractioninthecode(E2,E8,E9):flexibilityregardingmultipletypesofinput(forexample,touchscreenorsensors),andtobeableto“switchaudioenginesatwill”(E9).OneevaluatorcouldnotfinishtheprojectontimebecausethetoolkitdidnotsupportPureData(https://puredata.info)asaudioengine(E8).TwoevaluatorsfeltthattheUIcouldhavetakenadvantagefromtheexistingofxGUItoolkit(E1,E3).Speedandeaseofdevelopment:ofxAVUIwasconsideredeasytoworkwithbyrespondents(E3,E4,E7,E9,E10),notjustbecauseofitsorganizationandarchitecture,butalsobecause“it already has the minimum package of sound, UI and visual” built in (E3). One of therespondentsconsidersofxAVUIeasiertousethanthetwomainUItoolkitforOF,ofxUIandofxGUI(E10).Thisevaluatorconsidersiteasierandfastertoprototypewiththanusingrelatedtools,andthat“itfillsagap”,providing“interestingopportunitiesthatwouldnotbeaseasilypossiblepreviously”.Tworespondentsthoughtthattheadd-oncouldbebetterdocumented(E1,E10).Patchingandbuilding:Threerespondentswould liketohaveatop-levelenvironmentthatfacilitatesthecreationofUIs,withasimplifiedcodinglanguage(E1,E3,E7),orbydrag-and-drop, as in visual interface builders (E1). Two evaluators expressed interest in having a“mastercontroller” (E5,E10) thatcouldswitchzonesonandoff,andreroute informationbetween them. Anotherwas interested in integrating zones and patching them (E3). Yetanother suggested having multiple visualizations per zone, stacked in layers, visualizing

16

differentparameters(E5).TworespondentsareinterestedintheimplementationoftheOSCprotocol(http://opensoundcontrol.org)tocontrolotherapplicationsanddevices(E6,E10),withonestatingthatitwasfastertobuildaOSCcontrollerwithofxAVUIthanwithLemur,apopulartabletcontrollerbuilderapp(E10).ExperienceVisible interaction: Inausecase that involves showingofxAVUI toanaudience, suchasaperformanceorademo,mostrespondents(E1,E3,E5,E6,E8,E10)considerthatrevealingtheUIisimportant,makingthesoftware“engaging”and“easytounderstand”.Itcreates“amorecohesiveexperience”,byshowing“thebeautyoftheinternalsofthesystem”(E3).OnerespondentconsidersthatshowinganUIwoulddependontheusecase,andthatrevealingit“challengeshowyou interact” inorderto“findwaystomakeaudienceawareofwhat'shappening”(E5).Scenarios:TwoevaluatorsconsideredofxAVUIwellsuitedtoteachsoundandvisualization(E2,E8).OnerespondentconsideredofxAVUIadequateforgamedevelopment,particularlypedagogical games (E2). He also suggested that ofxAVUI could be used in more genericapplications,forhighlightingimportanttasks.AnotherevaluatorstatedthattheadoptionofOFmakesthetoolkitmoresuitedtoartisticapplications,butifmademore“portable,ornotrelyingonOF”itcouldbeusedformore“day-to-daysoftwaredevelopment”(E9).InterfaceInterfaceclarity:TwoevaluatorsrequestedmorevisualfeedbackforchangesofstateintheUI(E2,E6),suchashovering.Interfaceaesthetics:Severalrespondentslikedtheminimalist“barebones”aestheticsoftheUIelements(E1,E3,E9).OneevaluatorwantedtocustomizeUIelementsandimplementUI“themes”, and developed a project for adding background images to zones (E1). OnerespondentmentionedthatthelargesizeoftheUIelements“seemsmoreapplicabletoatouchinterfacethanamouseinterface”(E10).MediaSoundvisualization:Onerespondentwantedtohavemorepossibilitiesforaudioanalysis,andcreatedaprojectinthatdirection,basedonfrequencyvisualizers.Hesuggestedthatmoreaudio information retrieval techniques and 3D visualizations could be added (E2). Onerespondentwanted tohavenotonly sound visualization, but also visual sonification (E1).Anotherconsideredthatthevisualizationsshouldbeusednotjustforsound,buttovisualizeotherdata(E10).Mediamanipulation:Oneevaluator(E4)showedinterestinhavingliveaudioinput.Anotherrespondent(E5)wantedtosynthesizesoundfromanimageanditscolorinformation.Onerespondentwasinterestedinrhythmicalandquantizedaspectsofsound,anddevelopedastepsequencer(E3).

17

AVUIGuidelinesThe best practices identified in our research allow us to propose the following designguidelinesforAVUIs,forusebydesignerswhowishtoimplementAVUIs,eitherusingofxAVUIoradifferentapproach.Theymaybeusefulfordesignerswhowishtousesoundandimagetogetherintheinterface,eitherbyusingofxAVUIorbyusingdifferenttechnologies.Theseguidelines are divided into three topics, whichmatch the threemain themes across thedifferentstages:1)MaximizingAVExperiencea) Develop AVUIs that can be implemented across multiple platforms and interactionmodalities–multitouchinteractionseemsparticularlysuited,asitallowsforthesynaestheticillusionoftouchingandmoldingtheaudiovisuals;b) Consider the potential of AVUIs for facilitating visualization of interaction whensharing/showingascreen;c) Adopt an object-oriented approach, for a harmonious, coherent and interrelatedconvergenceofaudio,imageandUI;d) Facilitate different types of display, allowing for different performer-audience displayconfigurationsandhardware.2)OptimizingInterfaceFunctionalityandAestheticsa)Usereconfigurableinterfaces,possiblywithaback-end,thatallowtore-mapelementsoftheUItodifferentsonicfeaturesandvisualproperties;thatcanalsochangehowthesoundis visualized; and that can have an extensible architecture in order to better allow forcustomization;b) Explore not simply one-to-one but also one-to-manymappings betweenUI, audio andvisualfeatures;c) Adopt a minimalist interface aesthetics that integrates well with the visuals, namelyregardingcolor,shapeandmovement,andthatdoesnotdetractfromthevisuals;d)ReinforceinterfaceclaritybyensuringvisibilityofallUIelements,theirstate,parameterspaceandcurrentpositiontoit;andverifythatthevisualizationsdonotdetractfromthis;e) Allow for hierarchical interfaces, with the possibility of a master control, andcommunicationbetweenmodules.3)MediaStrategiesa)Allowforpowerfulmanipulationofsoundandimage:differentformsofmediageneration,suchasdifferentformsofsoundandvisualsynthesisandsampling;multipleaudioandvisualeffects; and experiment with mappings between UI, audio and visuals across differentproperties;b)Makeuseofgenerativemedia,duetoitsvariety,flexibilityandeconomyofresources;c) Try different visualization and sonification approaches, using information retrievaltechniquesfromaudioandimage;

18

d)VisualizationshouldreflectnotonlyaudiobutalsothemultipleinteractionsaffordedbytheUI;e)Leveragepowerfulmediamanagementfeatures,suchasnetworkedcontent(forexample,streamedaudioorvisuals),audioandvisualinput,andcontentsharingbetweenapplicationsinthesamedevice.

DiscussionOurmulti-stageresearchproducedarangeofconcreteoutcomes:prototypesbyusaswellasbyparticipants;asoftwareplug-intoolkit;andanappbuiltwiththatplug-in.Thisprocessallowedus,inaniterativeuser-centricmanner,togaininsightonAVUIdesign,summarizedasasetofAVUIdesignguidelinesabove,withimplicationsfordesigndiscussedhere.ComparisonwithExistingSolutionsSinceofxAVUIisbuiltinC++andwiththepopularopenFrameworkstoolkit,itcanbeeasilyadoptedbydigitalartistsanddesigners,andembeddedinotherC++code(withouthavingtoresorttoOSC,althoughitalsosupportsit).Stage3evaluatorswereverypositiveregardingspeedandeaseofdevelopmentwithofxAVUI,comparedtoothersolutions(suchasofxUIandofxGUI),andmostofthemconsideriteasytouse.Oneofthemaintrade-offsofofxAVUIversusothersolutionsis,duetoitsinherentpre-packagingofAVandUI,itisnotasflexibleasusingassortedgraphicsandsoundtoolkitstobuildasolutionfromscratch.Ontheotherhand,itisquicker,andalreadyestablishesaharmonizationofUIwithcontent.Object-orientedIntegrationofInterfacewithAVInanAVUIaswepropose,sound,visualizationanduserinterfaceareintegrated,functionallyandaesthetically,intothesameentity.Thisrelatestotheconceptofaudiovisualobjectsincognitivescience[19].Theresultsfromourstudiesconfirmtheappealofanobject-orientedapproachtoAVinteractiondesign[6].Byanalyzingsoundandrepresentingitvisuallyinreal-time,soundandimageareharmonized,synchronousandcoherent.Audiovisualcongruencyisensured,makinguseof the identifiedbenefitsofcrossmodalcongruencyregardingtaskaccuracyandengagement[5,11,25,31].Thisobject-orientedapproachtoAVUIdesignisaptto situationswhere thedisplayand interactionplaneare fused,as inmultitouchdisplays,allowingfor“asensationakintobeingindirectcontactortouchingandmoldingmedia”[13].VisibilityofInteractionThevisibilityofinteractionisinherentinanAVUI:aninteractiontriggerseitheravisualorasound,whichisvisualized.AnAVUIcanbeparticularlysuitedforusecaseswheretherearebenefitsfromrepresentinguserinteraction,suchas:performances(ourcase-study);remotecollaboration and telepresence; presentations and demos. However, we have detecteddifferentprofilesofusersregardingvisibleinteraction:someprefertovisuallyreflectagencyontothescreen,“makingitunderstandableforanaudience”(Butterfly,E2),othersconsiderthataudiencesdonotnecessarilywanttoseeagencyonthescreen.Infunctionalapplications,

19

thiswillbedeterminedbythetaskathand.Forcreativeapplications,thiscanbeamatteroftaste,aswenotedwithourexpertevaluatorsinStage1.Ineithercase,theAVUIallowsthedevelopertomergevisualcontentwithinterfaceelements.ReconfigurableInterfacesandFlexibleMappingsOneoftheidentifiedstrongpointsofofxAVUIwasitsmodularityandreconfigurability:itisdesigned to recombineUI andmedia in different configurations andmappings, providing“interesting opportunities that would not be as easily possible previously” (user E10). InofxAVUI,beingable toeasily routeanyUIparameter toanyaspectwithina zone (sound,visuals,otherUIelements)oroutsidethezone(forexample,anygraphiconthescreen)wasconsideredbyourusersasinnovativewithrespecttootherUItoolkits.FlexibilityofmappingsbetweenUI,soundandimage,andwaystomanagethatflexibility,arefundamentalqualitiesof an AVUI, and important features in enabling technologies for them to be useful tointeractiondesignersacrossarangeofapplicationdomains.DesignConstraintsandSpeedofDevelopmentUserE3appreciatedthatofxAVUIprovidesthecorenecessaryfunctionalityinsound,UI,andvisuals. This is considered an advantage for some: one evaluator considers that he can“prototypeacertainpartofmyprocessfaster”withofxAVUI,andthatitiseasiertousethanotherUItoolkits(E10).Forothers,ofxAVUIistooconstrainedpreciselybecauseitistiedtospecificpackagesofsound,UIandvisuals.Theseevaluatorswouldlikemoreabstraction,tobeabletoreplacecertainelementsofthetoolkit(forexample,theaudiolibrary).SomeevaluatorswouldlikeofxAVUItobesimplertouse,forexamplebyaddingaGUIlayerthatwouldallowuserstobuildAVUIsbydragginganddroppingelements,asintraditionalinterfacebuilders.Therefore,thereseemstobeadesireforbothhigherleveleaseofuse,andlowerlevelflexibility.Abetterbalancecouldbepursuedinthefuturebetweeneaseofusebypre-packagingelements,andallowingformorearchitecturalflexibility.HavingbeenbuiltwithOFisaconstraintinitself–OFispopularinmediaartanddesign,butnotusedasmuchformoregenericdevelopment.Thetoolkitcouldbeportedtootherframeworkstofacilitateitsadoptionformoregenericusecases.ParticipatoryDesignandHackathonsOurparticipatorydesignapproachenabledustoleverageartisticknowledgeinaudiovisualperformancefromarangeofpractitionersintoagenericsoftwaretoolkit.AVperformersarespecialistsinsoundvisualizationandvisualsonification–audiovisualcrossmodalinteractions.WebelievethattheirAVdesignskillswereanimportantcontributiontothisresearchthatbenefitedthedesignoftheofxAVUItechnology.Additionally,theseusersmakehighdemandsfor an AV system in terms of media manipulation and interaction design: they can beconsidered super-users, who are regularly performing in front of an audience, and needpowerful,fluidandresponsivemanipulationofAVmediathrougharobustinterface.

20

Weusedhackathonsasamotivating,productivewaytoconnectwithourusers.Hackathonswereemployedfromtwodifferentperspectives.InStage1,twohackathonswereusedfortherapidprototypingofAVperformancesystemsbyAVartists. InStage3,thehackathonaimedtotestinashortperiodoftimetheeaseofimplementationofourtoolkit.Participantswereaskedtodevelopaprojectwithit,whichcouldbeaddedtoourtoolkitrepository,asanextensionofitsfunctionalityorademonstrationofanewusecase.Wealsoreachedouttothe community of interaction designers and developers following our GitHub repositorywebsite.Inthissense,wecomplementedthe“local”perspectiveofthehackathonswiththe“global” community of GitHub users, adapting, albeit on a smaller scale, the approachfollowedin[28].Multi-stageApproachInformedbyrelatedmulti-stagestudies[33,35],weadoptedathree-stageapproachtothedevelopment of the ofxAVUI toolkit. This could be used for the development of othertechnologies.Itcanbesummarizedasexplore-consolidate-abstractapproach:1)exploreandgathermultipleviewsviaparticipatorydesignprocess,andevaluateresultswithotherusers;2)designaprototypethatconsolidatesbestpracticesdetected inthepreviousstudy,andevaluate with users; 3) develop a general technology based on the evaluation of theprototype, convertandabstractpositiveaspectsof it intoa toolkit,and runanadditionalparticipatorydesignandevaluationsessionfortesting.Thisapproach,withaparticipatorystagebasedonhackathons,allowedustoiterativelydevelopbothourAVUIGuidelinesandtheofxAVUItoolkit.Conclusionsfromeachstudywereconvertedintodesignspecificationsfor the following one. Conclusions from each stage fed into design specifications for thefollowingone,withthelaststageinformingasetofgeneraldesignguidelines.AVUIasParalleltoCrossmodalityintheRealWorldOur interactions with the world are multi-sensorial. Opening a door handles producesauditoryandvisualfeedback.Someoftheseinteractions,suchaspouringwaterintoaglass,giveus audiovisual feedback regardingdimensional data. In these interactions, audio andvisualinformationarerelatedinacongruentway.Theconceptofauditoryiconsaimsto“touse sound in away that is analogous to the use of visual icons to provide information,”providingaanaturalwaytorepresentdimensionaldataaswellasconceptualobjects[10].AVUI extends Gaver's pragmatic concept by proposing a crossmodal approach thatincorporatesUIelements,sonicfeedbackandcongruentvisualizationinawaythataestheticcontentandinterfacebecomeone.TheintegratedaudioandvisualizationreflectthestatusofUIelements,recallingafunctionalsimplicityofthesortencouragedbyJohnMaeda[22].

ConclusionWehave introduced theconceptofAudioVisualUser Interface (AVUI),a typeof interfacewhereUI,audioandvisualizationareinterconnectedandintegrated.BycombiningUIwithinterrelated sound and image, the proposed concept of AVUI (and ofxAVUI toolkit inparticular) canhelp leverage the identifiedbenefitsofaudiovisual crossmodal interaction,suchasimprovementsinperformanceandengagement.

21

We presented an iterative multi-stage process of design, prototyping, development andevaluationofofxAVUI,anenablingsoftwaretoolkittofacilitatedevelopmentofAVUIs.Thetoolkit has been released as open-source, and is multi-platform, aiming to facilitate itsadoption.Participatorydesignmethodswereused,centeredaroundthreehackathons.ThisprocessalsoallowedustoincorporateexpertandpractitionerinsightintoaseriesofgenericguidelinesforthedesignofAVUIs.Thetoolkitandguidelineswillbeofinteresttointeractiondesignerswhowishtocreatecompellingproductsandsystemswhichintegratesoundandimageintheinterface.ByextendingHook'sexistingtheoreticalframeworktostudyalargenumberofAVsystems,webelievethatthefindingshaveastronggeneralizabilitythatthepreviousstudiesdonot.WebelievethattheAVUIconceptandtheofxAVUItechnologyhavepotentialforapplicationinanumberofusecaseswhereascreenandinteractionisshowntoanaudience,andforandmultimodalinteraction.ThecrossmodallinkagesthatanAVUIfacilitatecouldbeusefulforengagement inVR andAR interactive environments. This formof interactionwhich fusessensingmodalities,functionandcontent,canbecompellingforanumberofdomains:notonly areas where engagement is important, such as art, education and games, but alsoassistiveandaccessibletechnologies.

AcknowledgementsWe would like to thank all who collaborated in this research: workshop and hackathonparticipants;evaluatorsandinterviewees;colleagues;andinparticularBorutKumperščakforprogramming assistance and Alessandro Altavilla for documentation. This work wassupported by the EU Marie Curie fellowship FP7 REA grant 627922 and also partly byFCT/MCTESLARSyS(UID/EEA/50009/2013(2015-2017)).

References

1. Paul Bach-y Rita. 1987. Brain Plasticity as a Basis of Sensory Substitution.Neurorehabilitation and Neural Repair 1, 2 (June 1987), 67–71. DOI:http://dx.doi.org/10.1177/136140968700100202

2. FlorentBerthaut,MarkT.Marshall,SriramSubramanian,andMartinHachet.2013.Rouages:RevealingtheMechanismsofDigitalMusicalInstrumentstotheAudience.In Proceedings of New Interfaces for Musical Expression (NIME). Daejeon, Seoul.http://nime.org/proceedings/2013/nime2013_51.pdf

3. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.Qualitative Research in Psychology 3, 2 (2006), 77–101. DOI:http://dx.doi.org/10.1191/1478088706qp063oa

4. Ana Carvalho and Cornelia Lund (Eds.). 2015. The Audiovisual Breakthrough.Fluctuating Images, Berlin. http://www.ephemeral-expanded.net/audiovisualbreakthrough/

22

5. KarenCollinsandPeter J.Taillon.2012.Visualizedsoundeffect icons for improvedmultimediaaccessibility:Apilotstudy.EntertainmentComputing3,1(Jan.2012),11–17.DOI:http://dx.doi.org/10.1016/j.entcom.2011.09.002

6. NunoN.Correia.2013. InteractiveAudiovisualObjects.Doctoral thesis.AaltoARTSBooks, Helsinki. https://www.taik.fi/kirjakauppa/product_info.php?cPath=26&products_id=269&language=en

7. NunoN.CorreiaandAtauTanaka.2014.User-CenteredDesignofaToolforInteractiveComputer-Generated Audiovisuals. In Proc. 2nd ICLI. Lisbon.http://www.liveinterfaces.org/proceedings2014.html

8. BramCrevits.2006.TheRootsofVJing.InVJ:Audio-VisualArtandVJCulture,MichaelFaulkner(Ed.).LaurenceKingPublishing,London,14–19.

9. Carl DiSalvo and Melissa Gregg. 2013. Issue-Oriented Hackathons. (2013). http://carldisalvo.com/posts/issue-oriented-hackathons/

10. WilliamW.Gaver.1986.AuditoryIcons:UsingSoundinComputerInterfaces.Hum.-Comput. Interact. 2, 2 (June 1986), 167–177. DOI:http://dx.doi.org/10.1207/s15327051hci0202_3

11. F.Wai-lingHo-Ching,JenniferMankoff,andJamesA.Landay.2003.Canyouseewhatihear?:thedesignandevaluationofaperipheralsounddisplayforthedeaf.InProc.CHI’03 (CHI ’03). ACM, New York, NY, USA, 161–168.DOI:http://dx.doi.org/10.1145/642611.642641

12. LarsErikHolmquist.2008.Bootlegging:MultidisciplinaryBrainstormingwithCut-ups.InProceedingsoftheTenthAnniversaryConferenceonParticipatoryDesign2008(PDC’08). Indiana University, Indianapolis, IN, USA, 158–161.http://dl.acm.org/citation.cfm?id=1795234.1795260

13. JonathanHook,DavidGreen,JohnMcCarthy,StuartTaylor,PeterWright,andPatrickOlivier.2011.AVJcenteredexplorationofexpressiveinteraction.InProc.CHI’11(CHI’11). ACM, New York, NY, USA, 1265–1274. DOI:http://dx.doi.org/10.1145/1978942.1979130

14. Jonathan Hook, John McCarthy, Peter Wright, and Patrick Olivier. 2013. Waves:exploring idiographic design for live performance. In Proceedings of the SIGCHIConferenceonHumanFactorsinComputingSystems(CHI’13).ACM,NewYork,NY,USA,2969–2978.DOI:http://dx.doi.org/10.1145/2470654.2481412

15. MichaelXuelinHuang,WillW.W.Tang,KennethW.K.Lo,C.K.Lau,GraceNgai,andStephen Chan. 2012. MelodicBrush: A Novel System for Cross-modal Digital ArtCreationLinkingCalligraphyandMusic. InProceedingsof theDesigning InteractiveSystems Conference (DIS ’12). ACM, New York, NY, USA, 418–427. DOI:http://dx.doi.org/10.1145/2317956.2318018

16. LillyIrani.2015.HackathonsandtheMakingofEntrepreneurialCitizenship.Science,Technology & Human Values (April 2015), 1–26. DOI:http://dx.doi.org/10.1177/0162243915578486

17. Sergi Jordà. 2003. Sonigraphical Instruments From FMOL to the reacTable*. InNIME03.Montreal.http://www.nime.org/proceedings/2003/nime2003_070.pdf

18. M.Komssi,D.Pichlis,M.Raatikainen,K.Kindstrom,andJ.Jarvinen.2015.WhatareHackathons for? IEEE Software 32, 5 (Sept. 2015), 60–67. DOI:http://dx.doi.org/10.1109/MS.2014.78

23

19. MichaelKubovyandMichaelSchutz.2010.Audio-VisualObjects.ReviewofPhilosophyand Psychology 1, 1 (March 2010), 41–61. DOI: http://dx.doi.org/10.1007/s13164-009-0004-5

20. Golan Levin. 2000. Painterly Interfaces for Audiovisual Performance. Ph.D.Dissertation.M.Sc.Dissertation.Massachusetts InstituteofTechnology,Cambridge,MA.http://acg.media.mit.edu/people/golan/thesis/thesis300.pdf

21. ThomasJamesLodatoandCarlDiSalvo.2016.Issue-orientedhackathonsasmaterialparticipation. New Media & Society (Feb. 2016), 1461444816629467. DOI:http://dx.doi.org/10.1177/1461444816629467

22. JohnMaeda.2006.TheLawsofSimplicity.TheMITPress,Cambridge,MA.23. Thor Magnusson. 2006. Affordances and Constraints in Screen-based Musical

Instruments. In Proceedings of the 4th Nordic Conference on Human-computerInteraction:ChangingRoles(NordiCHI’06).ACM,NewYork,NY,USA,441–444.DOI:http://dx.doi.org/10.1145/1182475.1182532

24. HarryMcgurkandJohnMacdonald.1976.Hearinglipsandseeingvoices.Nature264,5588(Dec.1976),746–748.DOI:http://dx.doi.org/10.1038/264746a0

25. Oussama Metatla, Nuno N. Correia, Fiore Martin, Nick Bryan-Kinns, and TonyStockman. 2016. Tap the ShapeTones: Exploring the Effects of CrossmodalCongruenceinanAudio-VisualInterface.InProceedingsofthe2016CHIConferenceonHumanFactorsinComputingSystems(CHI’16).ACM,NewYork,NY,USA,1055–1066.DOI:http://dx.doi.org/10.1145/2858036.2858456

26. William Moritz. 1997a. Digital Harmony: The Life of John Whitney, ComputerAnimationPioneer.2.5(Aug.1997).http://www.awn.com/mag/issue2.5/2.5pages/2.5moritzwhitney.html

27. William Moritz. 1997b. The Dream of Color Music, And Machines That Made itPossible. Animation World Magazine 2.1 (April 1997). http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

28. AlistairMorrison, DonaldMcMillan, Stuart Reeves, Scott Sherwood, andMatthewChalmers.2012.AHybridMassParticipationApproachtoMobileSoftwareTrials.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems(CHI’12). ACM, New York, NY, USA, 1311–1320. DOI:http://dx.doi.org/10.1145/2207676.2208588

29. Paul Nagle. 2008. Yamaha Tenori-On Review. Sound On Sound Feb. 2008 (2008).http://www.soundonsound.com/sos/feb08/articles/yamahatenorion.htm

30. I.Olowe,G.Moro,andM.Barthet.2016.residUUm:usermappingandperformancestrategiesformultilayeredliveaudiovisualgeneration.InProceedingsofInternationalConference on New Interfaces for Musical Expression (NIME). Brisbane. http://www.nime.org/proceedings/2016/nime2016_paper0053.pdf

31. Shin’ichi Onimaru, Taro Uraoka, NaoyukiMatsuzaki, andMichiteru Kitazaki. 2008.Cross-modal information display to improve driving performance. In Proc. ACMsymposiumonVirtualrealitysoftwareandtechnology(VRST’08).ACM,NewYork,NY,USA,281–282.DOI:http://dx.doi.org/10.1145/1450579.1450653

32. Chris Salter. 2010.Entangled: Technology and the Transformation of Performance.MITPress,Massachusetts.

33. GuySchofield,DavidGreen,ThomasSmith,PeterWright,andPatrickOlivier.2014.Cinejack:UsingLiveMusic toControlNarrativeVisuals. InProceedingsof the2014

24

ConferenceonDesigningInteractiveSystems(DIS’14).ACM,NewYork,NY,USA,209–218.DOI:http://dx.doi.org/10.1145/2598510.2598520

34. LadanShams,YukiyasuKamitani,andShinsukeShimojo.2000.Illusions:Whatyouseeis what you hear. Nature 408, 6814 (Dec. 2000), 788–788. DOI:http://dx.doi.org/10.1038/35048669

35. Alexander Wiethoff and Sven Gehring. 2012. Designing Interaction with MediaFaçades:ACaseStudy.InProceedingsoftheDesigningInteractiveSystemsConference(DIS ’12). ACM, New York, NY, USA, 308–317. DOI:http://dx.doi.org/10.1145/2317956.2318004