‘Vivisecting’tempo-spatialsemanticsinimmersiveenvironmentsandhybridevents:

fewmethodologiesfor’3DVJs’and’VRconductors’JānisGarančs

MPLab(ArtReaserchLab),LiepajaUniversity,Latviajg@mplab.lv

Abstract.Thisresearch(doctoralprojectinitsbeginningphase)investigatespotentialsoflivestagingofimmersivemulti-mediaproductionswithself-adoptinginterfaceelementsforauthor(s)andparticipants.Theaimistoproposenovelconceptsforahybridinteractionsystemthatcanenhanceartisticexpressionwithincomplexscenarios,andalsofunctionasapartofembeddedinfrastructuresforadaptedvenues.Whilefocusingoninterfacescontrollingspatialdynamicsof3D/stereoscopicvisualsandmulti-channelaudio,researchwouldproposefewguidelinesformorecomprehensivemetadataandsignalexchangeframework,whichwouldcomprisenotonlyformalstructureofmultimediaevent,butalsoincludespatialarrangementsandsignalfeedbackwithine.g.exhibitionorperformancelocations.Theframeworkisenvisionedasahybridinteractionsystem,consistingofnetworkedhardwarespecificationsandcustomdevelopedsoftwaremodulesforadaptedvenues;participationandinvolvementroles,andconventions(dynamicsetofprotocols)forreal-timeimprovisationandcollaborationamongauthors,co-developers,producersandaudience.

Keywords:immersive,stereoscopic,synaesthesia,VR

IntroductionThe(working)titlereferstothetraditionsoforchestra/choralconductorrole,themorerecentphenomenonofVJ(VisualJockey)movementinaudio-guidedaudio-visualperformance,andusesametaphorofthehistoricmedicaltermforcuttingonlivingtissue(coinedbyanti-vivisectionistsinthe19thcentury).Italsopoeticallyhintstoexistenceofspecific,‘metaphysic’tempo-spatialqualities,characteristictoseveraltime-basedgenresandeventtypes,whereillusoryandphysicalspacesaremerging,transformingand,sometimes,colliding.

Havingmyowneducationalbackgroundandcreativepracticepaththatmadeevolutionfromclassicalartsandmusic,toimmersive3Denvironments,mymotivationistocombinesomeaccumulatedtacitknowledgewithsurveyofrelevanttechnologicaloutlooks(e.g.infieldofaugmentedreality,real-timestructurescanning,gesturecapture)andcriticalreviewsaswellasrecenttheories(e.g.conceptofideasthesia),inordertoproposenovelformatsofstagingimmersivetime-basedcreativepractices.

Thevision(hypothesis)ofthisresearchis,thatfunction-adoptinginterfaces,constructedinVirtualandAugmentedRealitycanenhance‘overview’:rangingfromtraditionalfixedscore,torepresentationofanaleatoric,generativeaudio-visualcomposition,orothertypeofsequentialornon-lineardramaturgyand‘control’–efficientmanipulationofelementsandtheirrelationshipsinmulti-modalperformancebypre-definedandimprovisedmappingofparameters,dynamicallylocatedinillusoryperceivablespaces.

BriefdescriptionofthepracticalresearchtopicsTheaimistoinvestigate,proposeanddesignandevaluatenumberofmethodsandtoolprototypesforartisticandcreativeprojects,tocreateandperformoriginalartworksandalsotousethesystemasmoregenericinteractionorevenexhibitionplatform.Itwouldfocusoncapacitiesofaffectionontheuserandthevarietyofinteractionwithparticularcommunities.Theresearchtopicsfocusonpracticaldiscourseandtechnologicalapplicationandusetheirproactiveinterconnectionforpotentialnewoutcomes.

Followingtopicswillbeinvestigatedpractically:

• combinedcharacteristicsofinteractiveinstallationprototypesthatcanbe:

- improvised(performedlive)byartist(author)as‘artist'ssignaturestyle’expression- self-running,autonomous(permanentortemporary)structurewithinexhibition,stage(orother

type)ofvenue,processingdatafromexternalorsimulatedprocesses,exploredbyvisitors- extendedforotheralternativepresentationformats,addingvariousadditionalcontextswithin

largerexhibitionorevent-potentiallycreatinglinkwithotherartworksorevents

• conceptsof‘tempo-spatialobjects’(e.g.unityofspatial,time-basedaudio-visualelements)-bothasartisticmetaphorandauthoringinterfacethatleadstoefficientcontrolofpowerfulimmersive(hybridreality)experiences-alsoas‘artificialsynaesthesia’

• individualandcollaborativeauthoringinterfaceconcepts,thatcaninvolvegesture,body,brain-orothertypeofbio-feedback,thatenhanceslivecontrolofcomplex‘tempo-spatialobjects’

Oneoftheaspectsistoproposea‘meta-framework’,comprisingpre-categorisedandlive-extractedartworkstructure,exhibitionspacesensorreadingsandaudienceresponse.Thus,theframeworkwoulddescribenotonlyartwork’sparameters(content,formalstructureandunlimitedabstractdata),butmightincludealsoe.g.spatialarrangementswithinexhibition/performancevenueandparticipation/involvementroles.Theframeworkisenvisionedasself-updatingenvironmentinstallations(softwareandnetworkedhardwareandarchitecturalelements),foradaptedvenues;andconventions(dynamicsetofprotocols)forreal-timeimprovisationandcollaborationamongauthors,co-developers,producersandaudiences.Inaddition,explorationofpotentialfutureorganizationalandbusinessmodelsforsucheventswouldbeinvestigatedandproposed.

BriefdescriptionoftheresearchmethodologyAseriesofexperimentalmoduleswouldbedeveloped,totestaspectsofthesystemforrolesofartworkauthor,activeparticipantandpassivespectator.Theywillexploreperceptualpotentialsofpanoramictechnologiesandlookattheroleofsoundandmotionintheseparticularenvironments.Thiswillimplicateexperimentalstrategieswithuncertaincapacitiesintheeffectivenessofinteraction,qualityofpublicresponseandimpactofaudio-visuallanguage.Thesefactorswillbehandledwithusabilityandlikabilitytestingwhenparticularsystemstructuresmightbesetupandtestedforpublicresponse.Iwilluseaudienceresearchinbothqualitative(specificgroupsande.g.testteams)andquantitativemode–analysisofanonymousinteractionstatisticsonsite(withinlargeevent)andthroughonlineinterfaces.DuringtheresearchprojecttimeIwillcreateatleasttwodifferent,specificartworksthatwouldillustratethefindingstheresearchedconcepts.

Figure1.Interfacestudy

FewmetaphoricconceptsFollowingarefewmetaphors,frommyrecentandupcomingartworks–plannedforthepracticalresearchsessions:

‘Spatialsubversion’

Setupusesmultipleviewrepresentationofinteriorandvisitorpresence,capturedbystereoscopiccameras,structureanddepthsensors,withappliedspatialdepthextraction,edge-detectfilters,‘pointcloud’,etc.-thatallowillusoryswappingthesolidobjectsbytheir‘hollowversions’–tobesuperimposedwithadditionallymapped,‘solid’3Dobjects.

‘Vivisection’(fromLatin,meaning–vivus:aliveandsectio:cutting)

Inthiscaseis‘virtualmanipulation’–animagined‘surgery’onabstractedmultiplelayersofaudio-visualillusion,representedby3Dstereoscopicimage(AugmentedRealityHMDor3Dprojection)andspatialsound(binauralsimulationormultichannelsetup).Inadditionto‘spatialsubversion’elements,tappingintosignalcircuitsofequipmentdisplayingaudio-visualmaterial(e.g.specificartwork)and‘augmenting’thescenewithlayersofextractedfeatures,andestablishingalternativecircuits(echos,delaysofvariousvalue,feedbackloops,reverberation,filtering,etc.)Theaestheticaimistocreate‘denseartificialatmosphere’,thatissuperimposedwithabstract,graphicallyvisibleinterfacethatvisuallydivides,cuts,mixes,recombinesspace,exaggeratedbystereoscopicimagingandimmersiveaudioeffects.The‘surgery’ismanipulatedbyapairofXYZ/6-DOFsensors(e.g.Razor/HydraorSixense/STEM),coupledwithlaser-pointers(totrackandmapprojectedelementsinexhibitionspace).Intheprocess,theinterfacetotheobject/space(usedbythesubjects:authorandviewer)becomespartoftheobservedobject.

‘VRConductor’

Inmanymusicalgenres,theconductorprovidestempo,dynamics,articulation,phrasingandcueingtootherperformersbyspecificgesturesandevenbymimicsandgaze.Frommid-19thcenturyconductingtechniquebecameincreasinglystandardised.Intherecentdecades,therehasbeenanumberofexperimentalprojectstotrackconductor’sarm,handandbatonmovements(bywiderangeoftechnologies),toextractvariousaspectsofexpressionandconvertittocode,thancanbepassedfurthertocontrolsomeparametersofelectro-mechanicordigitalinstruments.Themostrecentadvancesinpopularlivetrackingtechnologies(e.g.KinectSensor,LeapMotionetc.)allowautomatedcalibration,whole-body-,arm-,hand-andfingertracking,and‘smarter’parametermappingtoadopttoindividual,uniquegesturestyles(notrelatedtotheclassicalconductortradition).Forthisresearch,theseorsimilartechnologieswillbeintegratedwithreal-time3Dgraphicsengine,whichgeneratesinterfaceelementsthatcancontroltempo,flow,movementdirection,pickingelements,adjustingparametersin3Dspace.

‘3DVJ’

TheconceptofVJ-ing–asaccompanyingvisualentertainment,characterisedbyshortvideo-loops,live-cutandmixedtorhythmicalmusic–hasgrownintoaspecificartform,whichismergingwithgenresofstage-design,performanceandlivecinema.Someofthesynestheticmethodologiesandtechnologieswillbeused-includingalgorithmicmusic,beat,pitch,volume,frequencyspectrumdetection,MIDI/DMX/OSCsyncedequipmentandsoftware,2Dand3Dprojections(using3Dstereoscopicsourcematerialandrendering)mappedonalmostanykindofsurface.

ConclusionTheresearchwillparticularlyexaminecreativeinterfacesolutionsthatcanengagebroaderaudiencesbyproposingnewauthoringmethodstosolvethecomplexrequirementsofcontentmanagement.Aproofofconceptwouldbearrangingaspecificculturalevent(incl.performance,exhibition,conference),connectingatleasttwovenuesbythedesignedsystemandoriginalartwork–usingthesystemmodules.Afterevaluationandadaptation,thesystemprototypeshouldhaveascalablecapabilitytofunctionasprogressing,temporary(orpartiallysustained)‘infrastructure’formoreculturaleventsandvenues.Thisresearchwillproduceacriticallyinformedpracticeinwhichtheoryandpracticebuildabalancedpartinthefinaloutcome.