experiential audio
DESCRIPTION
Audio Spatialization Effects with the Web Audio APITRANSCRIPT
audio
Audio Spatialization Effects in the BrowserJanessa Det - WEBHACKER @TWITTERNYC
experiential
summary
nodeS2noTE& PANNING
START & resourcesTHE WEB AUDIO API
APPLICATIONSMOTIVATION
sound&space CONVOLUTION DSP COMPONENTS
SPATIALIZATION experiential
audio
primer SIMULATING
ENVIRONMENTS
whoami.experientialaudio
janessa det (@jandet)
@twitternycSoftware engineer - webDiscovery Front-End
thefuture.fmlead front-end devDJ Defined Radio
columbia universityM.S. Computer scienceGraphics & Vision > UX/HCI
summary
sound&space.sound in the digital age
experientialaudiojanessa det (@jandet)
time vs. frequency
digitalization
browsers are 2d
<audio></audio>
soundhas become
flatsummary
sound&space.sound defines a space
experientialaudiojanessa det (@jandet)
summary
Sound interacts with surrounding space
3d space
environment
Environments perceived through sound
human perception
One of the key ways weperceive our surroundings
Janet CardiffThe Forty Part Motet
[1]
primer.alas! the web audio api
experientialaudiojanessa det (@jandet)
compatibilityfeatures
• audio signals processing for your browser
• easy to create, process, and control audio within web applications
• complements webgl
• chrome 10.0-24.0
• safari 6.0
• safari ios 6.0
• firefox NOOOOPE (BUT PLANNED)• AUDIO DATA API
summary
audio context
primer.audio routing graph
experientialaudiojanessa det (@jandet)
summary
audioDESTINATION
gain(volume)
audiosource
convolverfilter(s)audio
source
if ('AudioContext' in window) { context = new AudioContext();} else if ('webkitAudioContext' in window) { context = new webkitAudioContext();} else { throw new Error('AudioContext not supported!');}
nodes2note.types of audio nodes
experientialaudiojanessa det (@jandet)
• raw audio data into the system • final node to connect to
• outputs to lineout• speakers/headphones
AUDIO SOURCE AUDIO DESTINATION
var request = new XMLHttpRequest(); request.open("GET", options.path, true); request.responseType = "arraybuffer";
request.onload = function() {createSoundSource(request.response);
};request.send();
soundSource.connect(context.destination);
function createSoundSource(audioData) {soundSource = context.createBufferSource();soundSource.buffer =
context.createBuffer(audioData, false);soundSource.loop = true;
};summary
nodes2note.types of audio nodes
experientialaudiojanessa det (@jandet)
• apply gain to change volume • 0: lowpass• 1: highpass• 2: bandpass• 3: lowshelf
• 4: highshelf• 5: peaking• 6: notch• 7: allpass
filterNode = context.createBiquadFilter();filterNode.type = 0;filterNode.frequency.value = 220;
volumeNode = context.createGainNode();volumeNode.gain.value = 0.5;
summary
gain (volume) filters
types of audio nodes
experientialaudiojanessa det (@jandet)
summary
• delay audio for specified duration
nodes2note.
delay oscillator
delayNode = context.createDelayNode();delayNode.delayTime.value = 5;
• 0: sine• 1: square• 2: sawtooth
• 3: triangle• 4: custom
oscillatorNode = context.createOscillatorNode();oscillatorNode.type = 0;
types of audio nodes
experientialaudiojanessa det (@jandet)
summary
• scriptprocessornode
• synthesize and process audio directly with javascript
• custom DSP effects
• exposing internals of dsp to web applications
nodes2note.
fancypants jsJS
http://www.w3.org/TR/webaudio/
audio context
audio routing graph
experientialaudiojanessa det (@jandet)
summary
audioDESTINATION
gain(volume)filter
audiosource
nodes2note.
// initialize nodes...
sourceNode.connect(filterNode);
filterNode.connect(volumeNode);
volumeNode.connect(context.destination);
spatialization.INTRODUCTION TO panning
experientialaudiojanessa det (@jandet)
panning algorithm
• distance effects• sounds softer farther away
• orientation• what direction are you facing
• channels• adjust volume to l & R outputs
• mimick human perception of space[1]
summary
USING PANNING nodes
experientialaudiojanessa det (@jandet)
summary
• connect to an audio source
• makes sound source positional
• sound cones• inner angle• outer angle• outer gain
• note: velocity!
• one listener per context
• position (x, y, z)
• orientation• front vector• up vector
spatialization.
panning node listener: )
pannerNode = context.createPanner();pannerNode.setPosition(x, y, z);pannerNode.coneInnerAngle = 30;pannerNode.coneOuterAngle = 40;pannerNode.coneOuterGain = 0.5;
context.listener.setPosition(x, y, z);var front = getFrontVector(angle);var up = [0, 0, 1];context.listener.setOrientation(
front.x, front.y, front.z, up.x, up.y, up.z);
convolution.INTRODUCTION TO convolution
experientialaudiojanessa det (@jandet)
what is convolution?
h(t)x(t) y(t) = h(t) x(t)
• “cross-multiplying” 2 signals
• LTI system theory
• impulse response
summary
USING convolver nodes
experientialaudiojanessa det (@jandet)
summary
• set buffer as impulse response
• connect sound source to convolver node
• convolves input through impulse response
• emulate sound in any environment
convolution.
convolver nodevar request = new XMLHttpRequest(); request.open("GET", "sounds/ir/church.m4a", true); request.responseType = "arraybuffer"; request.onload = function () {
createConvolverNode(request.response);}request.send();
function createConvolverNode(audioData) {convolverNode = context.createConvolver();convolverNode.buffer
= context.createBuffer(audioData, false);}
position orientationaudio context
audio routing graph
experientialaudiojanessa det (@jandet)
summary
audioDESTINATION
convolution.
// initialize nodes...
sourceNode.connect(convolverNode);
convolverNode.connect(pannerNode);
pannerNode.connect(context.destination);
pannerconvolveraudio
source
demotime.experientialaudio
janessa det (@jandet)
interaction controls
START sound&space nodeS2noTE SPATIALIZATION CONVOLUTION APPLICATIONSprimer MOTIVATION THE WEB AUDIO API DSP COMPONENTS & PANNING SIMULATING
ENVIRONMENTS& resourcesexperiential
audio
summary
• player• listener• panning
sound
• sound• panning
sound
• panning sound• panner node• sound node
classes
• wasd - movement• panning sound
• q/e - rotate ccw/cw
• x - make tweeting sound
• expansions to come
panner-demo.jit.su
applications.experientialaudio
janessa det (@jandet)
gamingenriched
uxmobile web
(ios6)augmented
reality
summary
experientialaudiojanessa det (@jandet)
summary
future workareas to explore
• VISUAL INTERFACE & TOOLS TO BUILD AUDIO ROUTING GRAPHS
• INTERFACING WITH WEBGLexplore full 3d orientation & positioning
• INVESTIGATION OF AUDIO DATA APIcloser to the actual dsp implementations
• WEB AUDIO API ON IOS6 SAFARIdifferent ui, applications, and event handling
resources.
• rtfmhttp://www.w3.org/TR/webaudio/
• safari developer libraryhttp://bit.ly/RcVHxY
• DAFX (ref [1])
• JOHN WILEY & SONS• MAR 8, 2011
• impulse responseshttp://fokkie.home.xs4all.nl/IR.htm
SUMMARYPRESENTATION OVERVIEW
GOBACKORCHOOSEAsection
applications& resources
spatialization& Panning
simulating environmentsconvolution
nodes2notedsp components
START experiential audio
sound&space motivation
the web audio apiprimer