positional audio and window highlighting for the access grid robert putnam ([email protected]) boston...

Positional Audio and Positional Audio and Window Highlighting Window Highlighting

for the Access Gridfor the Access Grid

Robert Putnam([email protected])Boston University

Scientific Computing and Visualization

Boston University Scientific Computing and Visualization Group 2

AgendaAgenda

The problem Prototype positional audio Prototype window highlighting Distribution/installation Discussion


Who's talking now?Who's talking now?

In some AG contexts, it is not always clear which video window contains the current speaker. The problem is exacerbated by:– Multiple, concurrent speakers– Users unfamiliar with the AG– Users unfamiliar with each other

Remedy: look for familiar face or moving lips.


What’s that noise?What’s that noise?

In some AG contexts, it is not always clear who is transmitting echo or noise.

Remedies:– Broadcast appeal– Operator mutes RAT streams one at a time until

offender is found.


Solution #1: Positional audioSolution #1: Positional audio

Use location of site's video window(s) to determine its audio position in stereo field.


Solution #1: Positional audioSolution #1: Positional audio

Preliminary system design– Components

• (Minimally modified) vic and RAT• “Agent” which communicates with vic and RAT

– Sequence of events• Operator moves window• Agent gets report of window movement from vic• Agent sends audio panning command to RAT

– Need: communications & audio localization layers.

agentRAT

?vic

? ?


Positional audioPositional audio

RAT has built-in support for “3D audio”* RAT’s localization methods:

– Interaural Intensity Difference (IID)• Difference in sound intensity between near and far ear.

– Interaural Time Difference (ITD)• Difference in time of arrival between near and far ear

– Head Related Transfer Function (HRTF)• 3D-location-specific filtering effect of ears, head, and

torso.

*Actually “1D,” since there’s only a single degree of freedom.



Revised system design:

What about communications? Next slide…

agentRAT

3D audiovic

? ?


Message BusMessage Bus

RAT’s control, user interface and engine already communicate via the Message Bus (“mbus*”):

So…

rat-ui rat-engine

mbus

*see www.mbus.org

rat-control



Revised system design:

Next: what mbus message is used to enable 3D audio?

agentRAT

3D audiovic

mbus mbus


3D audio mbus message3D audio mbus message

3D audio mode is enabled for a given RAT source via an audio.3d.user.settings message:

rat-ui rat-enginembustcl

*see www.mbus.org

audio.3d.user.settings


Talking to RATTalking to RAT

So, agent sends audio.3d.user.settings message to rat-engine:

– ssrc = “Synchronization source” = RTCP stream id– azimuth = horizontal angular distance of listener

from sound source. (90 = panned left; -90 = panned right; 0 = panned center)

agent rat-engine

mbus

audio.3d.user.settings ssrc azimuth

vic

mbus

?


Listening to vicListening to vic

Next, create new mbus message– window.xy

• Normalized xpos (0 – 1.0)• Normalized ypos (0 – 1.0)• Name of video window (e.g., “LANL(Video)”)

Whenever video window is moved, resized, or renamed, send window.xy message to agent.

agent rat-engine

mbus

audio.3d.user.settingsvic

window.xy

mbus


Associating vic and RAT streamsAssociating vic and RAT streams

Agent responsible for determining which video windows correspond to which RAT streams (ssrc’s):

As of AGtk 2.4, it is assumed that vic and RAT streams have the same name – I.e., RAT “LANL” corresponds to vic “LANL(Video)”

agent rat-engine

04b6910a “LANL”

rtp.source.namevic

window.xy

“LANL(Video)”


Solution #2: Window highlightingSolution #2: Window highlighting

System shown at May 2003 AG retreat Feature request: visual indication of current

speaker– But, danger of distraction/fatigue (i.e. too much

visual clutter or “flashing”).


Window highlightingWindow highlighting

Change border color of site’s video windows when its audio rises above a threshold.

Two new mbus messages:– RAT: rat.source.amplitude(source_id amplitude)

• Report each stream’s peak amplitude 2-3X / second

– vic: highlight.window(window_id on_off)

agent rat-engine

rat.source.amplitude

audio.3d.user.settings

vic

window.xy

highlight.window



Window stays highlighted during brief pauses in audio unless another site begins transmitting.

Each site’s local video windows also highlight when RAT transmitting (“On Air” alert).

Window border highlight color and width is configurable in AG service.


Setting up the systemSetting up the system

Install stereo speakers/amplifiers Install stereo wiring to and from echo canceller

(aka “the Gentner”):

RATpc

Gentner amplifierline out l r l r l

r l r



Create left+right echo cancellation reference:



Remove or disable original video consumer and audio services and add two new ones:– HighlightVideoConsumerService.zip– PositionalAudioService.zip



Configure HighlightVideoConsumerService:

WindowsDesktop

Venue Client


What next?What next?

Ready for beta testers! Get it here:– http://scv.bu.edu/~putnam/ag-audio/services-2.4.zip– Zip file contains services for Windows / Linux, PIG /

multi-computer configurations.

Request features / report bugs here:– [email protected]

positional audio and window highlighting for the access grid robert putnam ([email protected]) boston...

Documents

positional audio rat

visualization slide

visualization group

slide agent rat

d audio vic mbus slide

rat agent

d audio mode

d audio mbus message