positional audio and window highlighting for the access grid
Post on 17-Jan-2016
23 Views
Preview:
DESCRIPTION
TRANSCRIPT
Positional Audio and Positional Audio and Window Highlighting Window Highlighting
for the Access Gridfor the Access Grid
Robert Putnam(putnam@bu.edu)Boston University
Scientific Computing and Visualization
Boston University Scientific Computing and Visualization Group 2
AgendaAgenda
The problem Prototype positional audio Prototype window highlighting Distribution/installation Discussion
Boston University Scientific Computing and Visualization Group 3
Who's talking now?Who's talking now?
In some AG contexts, it is not always clear which video window contains the current speaker. The problem is exacerbated by:– Multiple, concurrent speakers– Users unfamiliar with the AG– Users unfamiliar with each other
Remedy: look for familiar face or moving lips.
Boston University Scientific Computing and Visualization Group 4
What’s that noise?What’s that noise?
In some AG contexts, it is not always clear who is transmitting echo or noise.
Remedies:– Broadcast appeal– Operator mutes RAT streams one at a time until
offender is found.
Boston University Scientific Computing and Visualization Group 5
Solution #1: Positional audioSolution #1: Positional audio
Use location of site's video window(s) to determine its audio position in stereo field.
Boston University Scientific Computing and Visualization Group 6
Solution #1: Positional audioSolution #1: Positional audio
Preliminary system design– Components
• (Minimally modified) vic and RAT• “Agent” which communicates with vic and RAT
– Sequence of events• Operator moves window• Agent gets report of window movement from vic• Agent sends audio panning command to RAT
– Need: communications & audio localization layers.
agentRAT
?vic
? ?
Boston University Scientific Computing and Visualization Group 7
Positional audioPositional audio
RAT has built-in support for “3D audio”* RAT’s localization methods:
– Interaural Intensity Difference (IID)• Difference in sound intensity between near and far ear.
– Interaural Time Difference (ITD)• Difference in time of arrival between near and far ear
– Head Related Transfer Function (HRTF)• 3D-location-specific filtering effect of ears, head, and
torso.
*Actually “1D,” since there’s only a single degree of freedom.
Boston University Scientific Computing and Visualization Group 8
Positional audioPositional audio
Revised system design:
What about communications? Next slide…
agentRAT
3D audiovic
? ?
Boston University Scientific Computing and Visualization Group 9
Message BusMessage Bus
RAT’s control, user interface and engine already communicate via the Message Bus (“mbus*”):
So…
rat-ui rat-engine
mbus
*see www.mbus.org
rat-control
Boston University Scientific Computing and Visualization Group 10
Positional audioPositional audio
Revised system design:
Next: what mbus message is used to enable 3D audio?
agentRAT
3D audiovic
mbus mbus
Boston University Scientific Computing and Visualization Group 11
3D audio mbus message3D audio mbus message
3D audio mode is enabled for a given RAT source via an audio.3d.user.settings message:
rat-ui rat-enginembustcl
*see www.mbus.org
audio.3d.user.settings
Boston University Scientific Computing and Visualization Group 12
Talking to RATTalking to RAT
So, agent sends audio.3d.user.settings message to rat-engine:
– ssrc = “Synchronization source” = RTCP stream id– azimuth = horizontal angular distance of listener
from sound source. (90 = panned left; -90 = panned right; 0 = panned center)
agent rat-engine
mbus
audio.3d.user.settings ssrc azimuth
vic
mbus
?
Boston University Scientific Computing and Visualization Group 13
Listening to vicListening to vic
Next, create new mbus message– window.xy
• Normalized xpos (0 – 1.0)• Normalized ypos (0 – 1.0)• Name of video window (e.g., “LANL(Video)”)
Whenever video window is moved, resized, or renamed, send window.xy message to agent.
agent rat-engine
mbus
audio.3d.user.settingsvic
window.xy
mbus
Boston University Scientific Computing and Visualization Group 14
Associating vic and RAT streamsAssociating vic and RAT streams
Agent responsible for determining which video windows correspond to which RAT streams (ssrc’s):
As of AGtk 2.4, it is assumed that vic and RAT streams have the same name – I.e., RAT “LANL” corresponds to vic “LANL(Video)”
agent rat-engine
04b6910a “LANL”
rtp.source.namevic
window.xy
“LANL(Video)”
Boston University Scientific Computing and Visualization Group 15
Solution #2: Window highlightingSolution #2: Window highlighting
System shown at May 2003 AG retreat Feature request: visual indication of current
speaker– But, danger of distraction/fatigue (i.e. too much
visual clutter or “flashing”).
Boston University Scientific Computing and Visualization Group 16
Window highlightingWindow highlighting
Change border color of site’s video windows when its audio rises above a threshold.
Two new mbus messages:– RAT: rat.source.amplitude(source_id amplitude)
• Report each stream’s peak amplitude 2-3X / second
– vic: highlight.window(window_id on_off)
agent rat-engine
rat.source.amplitude
audio.3d.user.settings
vic
window.xy
highlight.window
Boston University Scientific Computing and Visualization Group 17
Window highlightingWindow highlighting
Boston University Scientific Computing and Visualization Group 18
Window highlightingWindow highlighting
Window stays highlighted during brief pauses in audio unless another site begins transmitting.
Each site’s local video windows also highlight when RAT transmitting (“On Air” alert).
Window border highlight color and width is configurable in AG service.
Boston University Scientific Computing and Visualization Group 19
Setting up the systemSetting up the system
Install stereo speakers/amplifiers Install stereo wiring to and from echo canceller
(aka “the Gentner”):
RATpc
Gentner amplifierline out l r l r l
r l r
Boston University Scientific Computing and Visualization Group 20
Setting up the systemSetting up the system
Create left+right echo cancellation reference:
Boston University Scientific Computing and Visualization Group 21
Setting up the systemSetting up the system
Remove or disable original video consumer and audio services and add two new ones:– HighlightVideoConsumerService.zip– PositionalAudioService.zip
Boston University Scientific Computing and Visualization Group 22
Setting up the systemSetting up the system
Configure HighlightVideoConsumerService:
WindowsDesktop
Venue Client
Boston University Scientific Computing and Visualization Group 23
What next?What next?
Ready for beta testers! Get it here:– http://scv.bu.edu/~putnam/ag-audio/services-2.4.zip– Zip file contains services for Windows / Linux, PIG /
multi-computer configurations.
Request features / report bugs here:– putnam@bu.edu
top related