Taptop:UsingtheLaptopChassisasaMusicalControllerOliverThompson1andChristopherHarte2

1DepartmentofElectronics,UniversityofYork,UKollythompson94@gmail.com

2MelodientLimited,Leeds,UKchris@melodient.com

Abstract:ThispaperpresentsTaptop,anewgesturecontrolapplicationdesignedforcontrollingdrummachines,samplersandothersoftwarevirtualinstrumentsrunningonlaptopcomputers.Withoutrequiringexternalelectronichardware,Taptopturnsthechassisofthelaptopintoavelocitysensitivecontrolsurfacefortriggeringsamplesbycombiningdatacollectedfromthebuilt-inwebcamandmicrophonetoproduceMIDImessages.Theusertapsontheunusedspaceeithersideoftheirlaptopmousetrackpadwhichisdividedintoanumberofvirtual‘drumpads’.EachvirtualpadcausesadifferentMIDInotetobeoutputwhentapped,withthevelocityofthatnotedeterminedbytheforceofthetap.ThiscontrolsolutionaimstoprovideamoreaffordableandportablealternativetothehardwareMIDIcontroller,andincreasethepotentialofthecomputerasastandaloneperformanceinstrument.

Keywords:GestureControl,DrumPads,VelocitySensitive,LivePerformance,MusicalInterface

IntroductionSoftwaredrummachinesandsamplersareanimportantpartoftheworkflowofmanyelectronicmusicproducers.Yet,despitetheprominentroleoftheseinstrumentsinthemodernmusicstudio,theoptionsforcontrollingthemremainrestrictedtoalimitednumberofmethods.Thisisanissuenotonlyassociatedwithsoftwaredrummachinesandsamplers,butothervirtualinstrumentstoo.Whilesignificantadvancescontinuetobemadeindevelopingmorepowerfulandflexiblevirtualinstruments,thewaysinwhichmusiciansinteractwiththoseinstrumentshasremainedlargelythesamesincetheinceptionoftheDigitalAudioWorkstation(DAW)Theberge(1997).Whencomposingwithavirtualinstrument,amusicianwilloftenarrangesequencesofnotesviatheirDAWpianoroll.Thismethodofcontrolobscuresthemusician’sinteractionwiththeirinstrumentbehindsoftware,incontrasttoanacousticinstrumentwherethephysicalgesturesthatproducethesoundareimmediatelyapparenttotheaudience(Dobrian2001).Wellthoughtoutmappingofphysicalgesturestomusicalparameterscouldthereforeprovideamoreexpressivemethodofcontrolforsoftwareinstruments,andhelptoreintroducesomeoftheperformanceelementthatisoftenlostwhenmakingmusicwithacomputer(Bown,Bell,andParkinson2014).

Forthepurposesofmusicalinteraction,itisimportantthatagesturecontrolsystemprovidesthemusicianwiththeopportunitytodeveloparepertoireandadegreeofmusicalityovertimethroughpractice,aswouldbethecasewithamoretraditionalinstrument.Whenaskedwhatinstrumenttheyplay,fewcomputermusicianswouldrespondwith‘Iplaythecomputer’.Thisislargelyduetothefactthatthemajorityofhuman-computerinteractioninterfacesleavelittleroomforprogressionorvirtuosity(WesselandWright2002).Thefamiliarmethodofcontrollingacomputerthroughakeyboardandmouseortrackpadmaybeintuitive,butitisalsoincrediblyrestrictiveasamusicalinterface(Pavlovic,Huang,andSharma1997).Suchdevicesareunabletocapturevelocityinformationandhencelimitmusicalexpression.However,thekeyboardandmousearenottheonlysensorsavailabletousonamodernlaptop,indeedFiebrinketalhaveinvestigatedtheuseofvariousbuilt-insensorsincludingmicrophones,accelerometers,andwebcamsamongothersforcontrollingmusic(Fiebrink,Wang,andCook2007).HongandYeousedacousticfeedbackbetweenthecomputerspeakeranditsownmicrophonetodriveaudioDSPforexperimentalaudio-visualperformance(HongandYeo2013).

Figure1.Taptopinuse.Anangledmirrorallowsthesystemtodetecttheuser’shandmotioninthedrumpadareausingthebuiltinwebcam.Theuserhasachoiceofdividingthelaptopchassisinto2,4or8drumpads,selectedviaaseriesofbuttonsontheGUI.Thedrumpadsincreaseinwidthasfewerpadsareselected,sothattheentiretyofthelaptopchassiswidthisalwaysbeingutilised.Theangleofthemirrorattachmentisadjustablesothatitcanbeused

withavarietyoflaptopmodels.

ExternalMIDIcontrollersgosomewayinenhancingthemusicalperformancecapabilitiesofthecomputer,buttheseareoftenexpensivedevicesandconstituteanotherpieceofequipmentforamusiciantotransportoraccommodateintheirstudio(GillianandNicholls2012).AsystemthatprovidesthesamefunctionalityasaMIDIcontrollerwithouttherequirementforexternalelectronichardwarewouldthereforepossesssignificantadvantages.Tothisend,wehavedevelopedTaptop,agesturecontrolsoftwareapplicationthatusesthelaptop’sownmicrophoneandcameratodetecttheuser’sactions.Usingthesebuilt-insensors,weallowunusedspaceonthelaptopchassisitselftobeusedasawellestablishedinterfaceforcontrollingvirtualdrummachinesandsamplers,namelyabankofdrumpads.Drumpadsareanintuitiveinterfacethatanyonecanusetotriggersamples,buttheyalsoallowaperformertodeveloptheirskillssufficientlyenoughovertimetobeconsideredanexpert(Zamborlinetal.2014);theelectronicmusicianJeremyEllisbeingagoodexample(Ellis2011).

Theconceptofusingmicrophonestoturnagenericsurfaceintoaninterfaceforcontrollingdigitalaudioisnotnew.Anumberofsystemshavealreadybeensuccessfullyimplementedusingthisidea,themostwidelyknownprobablybeing‘Mogees’(Zamborlin2016).Mogeesaresmalldevicescontainingacontactmicrophonewhich,whenconnectedtoasmartphoneorcomputer,allowtheusertodriveasynthesisengineinaccompanyingsoftwaresimplybyhittingorrubbingasurface.Asinglemicrophoneallowsthesystemtocapturetimbreandtransientinformationbutsomeotherapproacheshaveusedmultipletransducerstoallowpositioninformationtobecapturedaswell.ForexampleBisbyetal.(2014)developedaninstrumentusinganarrayofpiezostoachieveaccuratelocalisationofsoundsourcesonasurface,whileNovelloandRaijekoff(2015)utilisedapairofcontactmicrophonestolocatethegesturesoftheuseronasurface,mappingthepositionofmovementtothepitchoftheinstrument.

SystemOverviewTaptopwasimplementedinthePureDatagraphicalprogramminglanguage,makingextensiveuseoftheimageandvideoprocessingobjectsthatcomprisetheGEMlibrary.Insteadofphysicaldrumpads,theusertapsintheunusedspaceeithersideoftheirlaptopmousetrackpadwhichisdividedintoanumberofhorizontallyarrangedvirtualpads,representedbycolouredboxesinfigure1.Thelaptopwebcamfieldofviewispositionedoverthisareabymeansofamirrorattachment;theonlyexternalaccessoryrequiredtouseTaptop.Toassisttheuserinpositioningthemirrorattachment,thewebcamstreamisdisplayedinawindowoverlaidwithablackrectanglethatindicateswherethemousetrackpadshouldbelocated.

TaptopcontrolsvirtualinstrumentsviaMIDI.InordertotriggeraMIDInote,twocriteriamustbemet.Thisprocessisdescribedintheflowchartinfigure2.Firstly,theaudibletapoftheuser’sfingeronthechassisofthelaptopmustbedetectedthroughanalysisofthelaptopmicrophonesignal.ThelevelofthistapcorrespondstothevelocityofanyMIDInotesubsequentlytriggered.Secondly,movementmustbefoundtobeoccurringinatleastoneofthevirtualdrumpadsbyamotiondetectionalgorithmappliedtothewebcamsignal.EachofthesedrumpadscorrespondstoadifferentMIDInotenumber.Whenatapisdetectedinthemicrophonesignal,MIDInoteonmessagesaresenttothevirtualinstrumentbeingcontrolledwithnotenumbersdeterminedbythemovementoccurringinthedrumpads.

Figure2.Dataflowinthesystem.TaptoptranslatesdatafromthelaptopwebcamandmicrophonetoproduceMIDImessageswhicharethenusedtocontrolavirtualinstrument.

DetectingMovement

Taptopcropsthelaptopwebcamsignalintoanumberofregions,eachrepresentingavirtualdrumpad.Itisthemovementoftheuser’sfingersintheseregionsthatdeterminewhichnotenumbersareencodedintotheMIDInoteonmessagesthataresent.Movementisdetectedbycomparingthecurrentandpreviousframesofeachregiontodeterminewhichpixelshavesignificantlychanged.ThevideostreamisconvertedfromanRGBtoanRGBAcolourspacesothatpixelsthatdemonstrateachangeinthesesubsequentframescanbestoredintheemptyalphachannel.Ifthenumberofpixelsstoredinthealphachannelisaboveasetthreshold,thesystemassumesthatmovementisoccurringinthatdrumpad.

DetectingTransients

Todetecttheaudibletapoftheuser’sfingeronthelaptopchassis,weusedamethodbasedonthe‘Bonk’PureDataobject(Puckette1997).Bonkisatransientdetectorthathastheabilitytodeterminewhichinstrumentwasresponsibleforaparticularattackbycomparingspectralchangestoasetofstoredtemplates.Tapsonthelaptopchassiswererecordedatavarietyofvelocities,thenstoredastemplatessothatBonkcouldbetaughttoignoreotheraudiobeingpickedupbythemicrophone.Thishelpedaddressthepossibleissueoffeedbackinthesystem,whereplayingbackdrumsamplesthroughloudspeakerscouldcausetheapplicationtotriggeritself.Whilesomedrumhitsmayhaveasimilarenvelopetothetemplates,thespectraldifferencesbetweenthetwoshouldcausethemtobeignoredbyBonkinmostcases.

Transient detection

Motion detection

Motion detection

Motion detection

Partition and crop

Pad 1

Pad 2

Pad n

Note x

Note y

Note z

Video

Audio Events with velocity

MIDI output

Virtual instrument

MIDI Note On

Pad→note mapping

Pad→note mapping

Pad→note mapping

Mirror

User taps chassis

Web cam

Microphone

UserTestingAseriesofusertestswerecarriedoutinordertodeterminethesuccessofTaptopasamusicalinterface.Tenparticipantswereselectedwhoeachhadbackgroundsinelectronicmusicproduction.Thiswastoensurethatthosetakingpartinthetestswouldhavetherelevantknowledgetoprovideinformedopinionsontheperformanceandfeaturesoftheapplication.Twolinesofinvestigationwereemployed;afocusgrouptoestablishiftheconceptwaspopularamongmusicproducers,followedbymoredetailedbetatestingtoascertainhowproducersutilisedTaptopoveranextendedperiodoftime.Thefocusgroupandbetatestinvolvedfiveparticipantseach.

Methodology

Inthefocusgroup,participantswerefirstgivenanintroductiontotheconceptbehindTaptopandthecurrentissuesinvirtualinstrumentcontrolthatitwasdesignedtoaddress.Theyweretheninstructedonhowtousetheapplicationbeforebeinggiventenminutesalonetotryitoutforthemselvescontrollingavirtualinstrument.Thetestswerecarriedoutundercontrolledconditionsinaroomwheretheleveloflightingcouldbeadjusted.ThismeantthattheperformanceofTaptopcouldbeevaluatedinbright,dark,andoptimallightingtoreflectthedifferentenvironmentsthatitmaybeusedin.Afterthetestswerecompleted,theparticipantseachansweredaquestionnaireabouttheirexperienceswiththeapplication.

Forbetatesting,theparticipantswereeachprovidedwithacopyofTaptoptotryintheirownstudiosetups.Overtheperiodofaweek,theycouldusetheapplicationinwhateverwaytheychose.Afterthistimehadelapsed,theparticipantssubmittedfeedbackonhowithadperformed.ThetwokeythingstoestablishfromthiswerethevariouswaysinwhichtheparticipantsutilisedTaptop,andiftheywouldthenconsideradoptingtheapplicationovertheirusualmethodofcontrolattheendofthetrialperiod.

Results

Inthefocusgroupquestionnaireresults,thetwomostfrequentanswersgivenbytheparticipantsforthefeaturesofTaptoptheylikedthemostwereitsself-contained,‘inthebox’format,anditsabilitytocontroltheplaybackvelocityofsamplesthroughaphysicalaction.TheconceptofamusicalinterfacebeingcompletelyintegratedintothesamedeviceasthesoundsourceprovedattractivetothoseparticipantswhomakeuseofhardwareMIDIcontrollers.Althoughmanufacturersputeffortintomakingsuchdevicesmoreportable,nothavingtoaccommodateoneatallwouldbedesirabletoamusicianwhohastotravelorwhohaslimitedstudiospace.ForthoseparticipantswhouseaDAWpianorolltocontroltheirvirtualinstruments,theinputofnotevelocitythroughgesturewasseenasamoreexpressivealternativetoprogrammingvelocityautomationwithinaDAW.

ThemostcommonfeatureofTaptopthattheparticipantsdislikedwasitsreliability.Iflightingwasnotatanoptimumlevelthentheapplicationhaddifficultyindetectingtheusersmovement,meaningthatsamplesweresometimesnottriggeredwhentheyweresupposedtobe.Sincetheenvironmentsthatelectronicmusicisperformedinaregenerallydimlylitwithsporadicburstsofverybrightlight,thisisasignificantproblemthatneedstobeaddressed.Anotherconcernraisedbytheparticipantswasthattappingalaptoptoohardinanareawheretheharddriveislocatedmayresultinthedrivebeingdamaged.Thisisespeciallyrelevantformagneticdiskdriveswhicharemoresusceptibletoimpactdamagethansolid-statestorage(Pinheiro,Weber,andBarroso2007).Ithasyettobeestablishedwhetherornotlongtermuseoftheapplicationhasanaffectontheharddriveofthelaptopthatitisrunon.

TheresultsofthebetatestingaredisplayedinTable1.Feedbackwasgenerallypositivewith3outofthe5participantsstatingthattheywouldconsiderusingTaptopinplaceoftheirusualmethodofcontrol.Participant2encounteredissueswithPureDatabeingunabletostreamvideofromthelaptopwebcam,thoughttobecausedbyacompatibilityproblembetweenthe‘pix_video’PureDataobjectandcertainversionsoftheMacOSXoperatingsystem.Forthisreason,theywereunabletogettheapplicationworkingintheirstudiosetupandthereforestatedthattheypreferredtheirusualmethodofcontroltoTaptop.

Table1.Resultsofbetatesting.FeedbackwasgenerallypositivewiththreecandidateswillingtoconsiderusingTaptoptoreplacetheircurrentcontrolmethod.*Participant2wasunabletorunthesystemduetoacompatibility

problemwithPureDataontheirlaptop.

Participant5alsostatedthattheypreferredtheirusualmethodofcontrol,aMIDIkeyboard,toTaptop.Whiletheythoughtthattheapplicationwasusefulforgeneratingideasonthemove,theyenjoyedthetactilefeedbackofkeysbeingdepressedwhenperformingonahardwaredevice.ItisinterestingtonotethatParticipant4usedTaptopinconjunctionwithsoftwaresynthesisersratherthanwithsoftwaredrummachinesandsamplers,aswasitsintendeduse.Theycommentedthattheapplicationprovidedamoreimmediatewayofinterfacingwithsynthesiserswhenauditioningpatchesthantheirusualmethodofcontrol,aDAWpianoroll.Participants1and3usedTaptopforitsoriginalpurpose,controllingdrummachineandsamplervirtualinstruments.TheybothstatedthattheypreferredTaptopovertheirusualhardwaresetupsduetotheeaseofportabilitythatitaffordedthem.

ConclusionWehavedemonstratedareal-timemethodofvirtualinstrumentcontrolthatusesonlythebuilt-inwebcamandmicrophonethatcomeasstandardwithmostlaptopcomputers.Thedevelopmentofthisapplicationsoughttoaddressanumberoftheissuesassociatedwithcurrentmethodsofvirtualinstrumentcontrol,particularlyconcerningsoftwaredrummachinesandsamplers.OneobjectivewastoreleasemusiciansfromtheburdenoftransportingandaccommodatinghardwareMIDIcontrollersthroughagesturecontrolsystemthatpossessedthesamefunctionality.Thiswasachievedbyexclusivelyutilisingthebuilt-infeaturesofalaptopcomputertocontrolthevirtualinstrument;theonlyexternalcomponentrequiredbeingasmallmirrorattachment.BybasingTaptop’scontrolmetaphoronabankofdrumpads,thefunctionalityoftheapplicationislargelyidenticaltoanalreadywellestablishedmusiccontrolinterface.

Anotherobjectivewastoexplorethepotentialofthecomputerasaperformanceinstrumentthroughthedevelopmentofanexpressivemethodofcontrolforvirtualinstruments.Normally,withouttheuseofanexternaldevice,musiciansarerestrictedtocomposingonacomputerviatheirDAWpianoroll.Sincethismethodofcontrolisincapableoffacilitatingreal-timeperformance,itisunderstandablewhymanymusicianshesitatetoclassthecomputerasamusicalinstrument(WesselandWright2002).However,withsuchawealthofmusicnowbeingmadeexclusivelyusingcomputers,itseemsstrangethatthisshouldbethecase.Taptopexpandsthepotentialofthecomputerasamusicalinstrumentbyenablingalaptoptobeusedforstandaloneperformance,wheretheaudiencecandirectlyrelatethephysicalgesturesoftheperformertothemusictheyarehearing(CicilianiandMojsysz2014).

FurtherworkneedstobedoneinimprovingTaptop’sreliability,particularlywhenbeingusedinadverselightingconditions.Ifthelightingistoobrightortoodark,themotiondetectionalgorithmislesslikelytobeabletodetectusermovement.Makingthemotiondetectionalgorithmadaptivetodifferentlightingconditionscouldbetheanswertothis,sothatindifficultenvironmentsthesensitivityisautomaticallyloweredincreasingthechancesofusermovementbeingdetected.

ReferencesBisby,Helena,AdamCooper,StephenJames,AlecRobertson,andKiaNg.2014.“ATactileAudioVisualInstrumentUsingSoundSourceLocalisation.”InProceedingsofthe2014ElectronicVisualisationandtheArtsConference,200–206.

Bown,Oliver,RenickBell,andAdamParkinson.2014.“ExaminingthePerceptionofLivenessandActivityinLaptopMusic:ListenerInferenceAboutWhatthePerformerIsDoingfromtheAudioAlone.”InProceedingsofthe2014InternationalConferenceonNewInterfacesforMusicalExpression,13–18.

Ciciliani,Marko,andZenonMojsysz.2014.“EvaluatingaMethodfortheAnalysisofPerformancePracticesinElectronicMusic.”InProceedingsofthe2014InternationalConferenceonLiveInterfaces,45–61.

Dobrian,Christopher.2001.“AestheticConsiderationsintheUseofVirtualMusicInstruments.”JournaloftheSocietyforElectro-AcousticMusicintheUnitedStates16(2):23–33.

Ellis,Jeremy.2011.“JeremyEllisPerformingonMaschine-UnlikeAnyOther.”https://www.youtube.com/watch?v=FcJCxe1VSLA[Accessed:21Feb.2016].

Fiebrink,Rebecca,GeWang,andPerryCook.2007.“Don’tForgettheLaptop:UsingNativeInputCapabilitiesforExpressiveMusicalControl.”InProceedingsofthe2007InternationalConferenceonNewInterfacesforMusicalExpression,164–67.

Gillian,Nicholas,andSarahNicholls.2012.“AGesturallyControlledImprovisationSystemforPiano.”InProceedingsofthe2012InternationalConferenceonLiveInterfaces.

Hong,DaeRyong,andWoonSeungYeo.2013.“Laptap:LaptopComputerasaMusicalInstrumentUsingAudioFeedback.”InProceedingsofthe2013InternationalConferenceonNewInterfacesforMusicalExpression,233–36.

Novello,Alberto,andAntonyRaijekoff.2015.“APrototypeforPitchedGesturalSonificationofSurfacesUsingTwoContactMicrophones.”InProceedingsofthe2015InternationalConferenceonNewInterfacesforMusicalExpression,170–73.

Pavlovic,Vladimir,ThomasHuang,andRajeevSharma.1997.“VisualInterpretationofHandGesturesforHuman-ComputerInteraction:AReview.”IEEETransactionsonPatternAnalysisandMachineIntelligence19(7):677–95.

Pinheiro,Eduardo,WolfWeber,andLuizBarroso.2007.“FailureTrendsinaLargeDiskDrivePopulation.”InProceedingsofthe5thUSENIXConferenceonFileandStorageTechnologies,17–18.

Puckette,Miller.1997.“PureData:RecentProgress.”InProceedingsoftheThirdIntercollegeComputerMusicFestival,1–4.

Theberge,Paul.1997.AnySoundYouCanImagine:MakingMusic,ConsumingTechnology.Hanover,NewHampshire:UniversityPressofNewEngland.

Wessel,David,andMatthewWright.2002.“ProblemsandProspectsforIntimateMusicalControlofComputers.”ComputerMusicJournal26(3):11–22.

Zamborlin,Bruno.2016.“Mogees.”http://mogees.co.uk[Accessed:28Apr.2016].

Zamborlin,Bruno,FredericBelivacqua,MarcoGillies,andMarkd’Iverno.2014.“FluidGestureInteractionDesign:ApplicationsofContinuousRecognitionfortheDesignofModernGesturalInterfaces.”ACMTransactionsonInteractiveIntelligentSystems3(4):30–45.