Real-Time Auralization Real-Time Auralization of Sound in Virtual 3D of Sound in Virtual 3D

EnvironmentsEnvironments

by Scott McDermottby Scott McDermott

sdm1718@louisiana.edusdm1718@louisiana.edu

Overview & ObjectiveOverview & Objective

Develop an adaptive virtual environment that Develop an adaptive virtual environment that simulates real-time generation of 3D sound.simulates real-time generation of 3D sound.

Design algorithms to efficiently and effectively Design algorithms to efficiently and effectively compute realistic 3D sounds in this environment.compute realistic 3D sounds in this environment.

Apply these techniques to various applications, Apply these techniques to various applications, including simulations, virtual reality, gaming, and including simulations, virtual reality, gaming, and modeling.modeling.

OutlineOutline

Sound PerceptionSound Perception

Digital Sound and ComputersDigital Sound and Computers

3D Sound Approximations3D Sound Approximations

True 3D SoundTrue 3D Sound

““Surround” Sounds (Stereo Expansion) Approach Surround” Sounds (Stereo Expansion) Approach

Head Response Transfer Function (HRTF) ApproachHead Response Transfer Function (HRTF) Approach

Beam Tracing ApproachBeam Tracing Approach

The Graphics AnalogyThe Graphics Analogy

Sound PerceptionSound Perception

When we hear a sound, we automatically obtain When we hear a sound, we automatically obtain certain information about the source:certain information about the source:

DirectionDirection

DistanceDistance

ElevationElevation

Environmental conditionsEnvironmental conditions

Status of sourceStatus of source

Sound PerceptionSound Perception

Interaural Delay Time Interaural Delay Time (direction)(direction)

delay between time arrives at each ear (0 to 0.63 ms)delay between time arrives at each ear (0 to 0.63 ms)

Head Shadow Head Shadow (direction and distance)(direction and distance)

difference in volume from one ear to the other (up to 9 dB)difference in volume from one ear to the other (up to 9 dB)

Pinna Response Pinna Response (direction and elevation)(direction and elevation)

outer ear filters sound, compare between two earsouter ear filters sound, compare between two ears

Shoulder Response Shoulder Response (elevation and direction)(elevation and direction) reflections off upper body (1-3 kHz)reflections off upper body (1-3 kHz)

8 types of cues for sound spatialization [1]:8 types of cues for sound spatialization [1]:

Head MotionHead Motion move head to re-evaluate these filtersmove head to re-evaluate these filters

VisionVision ignore audio cues if different from visualignore audio cues if different from visual

Early Echo Response Early Echo Response (distance and direction)(distance and direction)

echos from environment (50 to 100 ms)echos from environment (50 to 100 ms)

Reverberation Reverberation (distance and direction)(distance and direction)

late dense echos from environment (> 100 ms)late dense echos from environment (> 100 ms)

Sound PerceptionSound Perception

An environment with true 3D sound will An environment with true 3D sound will need to take all of these into account.need to take all of these into account.

It must also be able to perform It must also be able to perform calculations and apply filters in real-time.calculations and apply filters in real-time.

The result must be convincing to the The result must be convincing to the listener and enhance the virtual listener and enhance the virtual experience.experience.

Sound in the Digital WorldSound in the Digital World

Sound in the physical world exists as Sound in the physical world exists as waves of pressure changes.waves of pressure changes.

A microphone converts pressure changes A microphone converts pressure changes to changes in voltage.to changes in voltage.

Blah blah blah…

An analog to digital convert changes these An analog to digital convert changes these voltage signals to discrete digital signals.voltage signals to discrete digital signals.

A computer stores, manipulates, and A computer stores, manipulates, and retransmits these abstractions of sound.retransmits these abstractions of sound.

The sounds can be stored in various The sounds can be stored in various formats and qualities (such as mono or formats and qualities (such as mono or stereo, 8 or 16 bit, 11 or 44 kHz).stereo, 8 or 16 bit, 11 or 44 kHz).

The reverse of this process allows the The reverse of this process allows the computer to re-generate the sound.computer to re-generate the sound.

3D Sound, The Basics…3D Sound, The Basics…

In a virtual 3D environment, sound can In a virtual 3D environment, sound can originate from an infinite number of originate from an infinite number of locations relative to the observer.locations relative to the observer.

Ideally, when the observer hears the Ideally, when the observer hears the sound it should take into account the sound it should take into account the environment.environment.

Specifically:Specifically:

Distance:Distance:Causes sound to arrive at different times.Causes sound to arrive at different times.

Reflection & Reverberation:Reflection & Reverberation:Causes “copies” of the sound to arrive at different times.Causes “copies” of the sound to arrive at different times.

Diffraction & Refraction:Diffraction & Refraction:Causes sound to bend around objects or arrive at Causes sound to bend around objects or arrive at different times.different times.

Absorption & Attenuation:Absorption & Attenuation:Causes the sound to be weaker when it arrives.Causes the sound to be weaker when it arrives.

3D Sound Approximations: 3D Sound Approximations: Surround SoundSurround Sound

Surround sound uses various filters to Surround sound uses various filters to simulate the effects of sound simulate the effects of sound spatialization.spatialization.These filters create effects such as These filters create effects such as reverberation, localization, and reverberation, localization, and attenuation.attenuation.Sound paths are not calculated.Sound paths are not calculated.Used in most theaters and home Used in most theaters and home entertainment units.entertainment units.

Surround SoundSurround SoundThe user is situated with a set of speakers around him.The user is situated with a set of speakers around him.To simulate 3D localization, sound is played louder, out To simulate 3D localization, sound is played louder, out of phase, and/or at slightly different times from each of phase, and/or at slightly different times from each speaker.speaker.Comes in a variety speaker placement setups [8]:Comes in a variety speaker placement setups [8]:Bark bark

bark!!

Dolby 5.1Two Speaker StereoHeadphonesQuadraphonic

3D Sound Approximations:3D Sound Approximations:Head Related Transfer FunctionsHead Related Transfer FunctionsUsed in conjunction with surround sound Used in conjunction with surround sound to create better 3D approximations.to create better 3D approximations.Microphones record sound from within the Microphones record sound from within the ear of a person or a model.ear of a person or a model.Differences between original sound and Differences between original sound and recordings are used to create filters.recordings are used to create filters.These filters are applied to generated These filters are applied to generated sounds to create the illusion of sounds to create the illusion of dimensionality. dimensionality.

Surround Sound &Surround Sound &Head Related Transfer FunctionsHead Related Transfer Functions

Pros:Pros:

Cons:Cons:

Relatively cheap.Relatively cheap.Effective.Effective.Makes sense.Makes sense.

Many different approaches (non-standard).Many different approaches (non-standard).Works only with limited speaker positions.Works only with limited speaker positions.Not entirely generic.Not entirely generic.Still not pure 3D sound.Still not pure 3D sound.

True 3D SoundTrue 3D Sound

3D graphical environments already exist.3D graphical environments already exist.

Light paths traverse the scene and surface Light paths traverse the scene and surface intensities are calculated.intensities are calculated.

Currently, sound paths are at most Currently, sound paths are at most superficially computed.superficially computed.

Yet, programmers already have a wealth Yet, programmers already have a wealth of environmental data.of environmental data.

Various possible approaches…Various possible approaches…

True 3D SoundTrue 3D Sound Beam Tracing Beam Tracing

Approach:Approach:

Source: Real-Time Acoustic Modeling forDistributed Virtual Environments [4]

Divide the environment into Divide the environment into cells or regions.cells or regions.

Precompute and store beam Precompute and store beam paths from various source paths from various source locations.locations.

Lookup, in real-time, reverberation paths from the avatar Lookup, in real-time, reverberation paths from the avatar to the source.to the source.

Use these paths to calculate delay and attenuation from Use these paths to calculate delay and attenuation from the original, anechoic, audio signal for each of the the original, anechoic, audio signal for each of the echoes.echoes.

Beam Tracing Beam Tracing

Quick and effective (with a good data structure).Quick and effective (with a good data structure).Intuitive.Intuitive.Scalable for large environments.Scalable for large environments.

Needs offline computations.Needs offline computations.Assumes sources are stationary.Assumes sources are stationary.Assumes source locations are finite.Assumes source locations are finite.

Pros:Pros:

Cons:Cons:

True 3D SoundTrue 3D Sound

On a basic level, we can determine sound On a basic level, we can determine sound propagation similar to how light travels through a propagation similar to how light travels through a 3D environment.3D environment.One simple, but computationally intensive One simple, but computationally intensive method would be similar to method would be similar to ray tracingray tracing..Ray tracing algorithms are generally very Ray tracing algorithms are generally very effective but also extremely slow and prone to effective but also extremely slow and prone to sampling errors.sampling errors.Most real-time algorithms for graphical Most real-time algorithms for graphical computers make various assumptions:computers make various assumptions:

True 3D SoundTrue 3D SoundThe Graphics AnalogyThe Graphics Analogy

Objects are made from geometric primitives composed of points.Objects are made from geometric primitives composed of points.These vertices are transformed to be relative to the camera.These vertices are transformed to be relative to the camera.Objects outside of the viewing field are clipped.Objects outside of the viewing field are clipped.Rays are sent from the camera, through each point on the projection Rays are sent from the camera, through each point on the projection plane, and into the scene.plane, and into the scene.Corresponding pixel values in the viewport are calculated from Corresponding pixel values in the viewport are calculated from these rays.these rays.

The 3D Graphics Pipeline:The 3D Graphics Pipeline:

True 3D SoundTrue 3D SoundThe Graphics AnalogyThe Graphics Analogy

Objects are made from geometric primitives (triangles, Objects are made from geometric primitives (triangles, rectangles) composed of points.rectangles) composed of points.

Light intensities are calculated based on surface normals Light intensities are calculated based on surface normals of these points.of these points.

These intensities are fed into the graphics pipeline.These intensities are fed into the graphics pipeline.

True 3D SoundTrue 3D Sound The Graphics Analogy The Graphics Analogy

Many of these computations are forwarded Many of these computations are forwarded to optimized 3D graphics cards.to optimized 3D graphics cards.

Many of these same techniques could be Many of these same techniques could be employed for generating realistic 3D employed for generating realistic 3D sounds.sounds.

We would need to develop and design 3D We would need to develop and design 3D sound cards and appropriate algorithms.sound cards and appropriate algorithms.

ConclusionConclusion

3D graphics and many other components 3D graphics and many other components of today’s computer systems have been of today’s computer systems have been almost thoroughly developed.almost thoroughly developed.

3D sound is still in the infancy stage.3D sound is still in the infancy stage.

This field has a great deal of research This field has a great deal of research potential.potential.

ReferencesReferences[1][1] Burgress, David, A. Burgress, David, A. Techniques for Low Cost Spatial AudioTechniques for Low Cost Spatial Audio. ACM UIST, pages 53-59, 1992.. ACM UIST, pages 53-59, 1992.

[2][2] Ellis, Sean. Ellis, Sean. Towards More Realistic Sound in VRMLTowards More Realistic Sound in VRML. ACM Virtual Reality and Modeling, . ACM Virtual Reality and Modeling, pages 95-100, 1998.pages 95-100, 1998.

[3][3] Flaherty, Nick. Flaherty, Nick. 3D audio: new directions in rendering realistic sound3D audio: new directions in rendering realistic sound . Electronic Engineering, . Electronic Engineering, pages 49, 52, 55, & 56, 1998.pages 49, 52, 55, & 56, 1998.

[4][4] Funkhouser, Thomas, A. , Patrick Min, and Ingrid Carlbom. Funkhouser, Thomas, A. , Patrick Min, and Ingrid Carlbom. Real-time Acoustic Modeling for Real-time Acoustic Modeling for Distributed Virtual EnvironmentsDistributed Virtual Environments. SIGGRAPH, pages 365-374, 1999.. SIGGRAPH, pages 365-374, 1999.

[5][5] Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. A A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual EnvironmentsBeam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments ..

[6][6] Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. Interactive Acoustic Modeling of Complex EnvironmentsInteractive Acoustic Modeling of Complex Environments . Acoustical Society of America, 1999.. Acoustical Society of America, 1999.

[7][7] Min, Patrick, and Thomas A. Funkhouser. Min, Patrick, and Thomas A. Funkhouser. Priority-Driven Acoustic Modeling for Virtual Priority-Driven Acoustic Modeling for Virtual Environments. EUROGRAPHICSEnvironments. EUROGRAPHICS, 2000., 2000.

[8][8] Tsingos, Nicolas, Thomas A. Funkhouser, Addy Ngan, and Ingrid Carlbom. Tsingos, Nicolas, Thomas A. Funkhouser, Addy Ngan, and Ingrid Carlbom. Modeling Modeling Acoustics in Virtual Environments Using the Uniform Theory of Diffraction.Acoustics in Virtual Environments Using the Uniform Theory of Diffraction.

[9][9] Hull, Joseph. Hull, Joseph. Surround Sound Past, Present, and FutureSurround Sound Past, Present, and Future. Dolby Laboratories Inc. . Dolby Laboratories Inc. http://www.dolby.com/tech/.http://www.dolby.com/tech/.

[10][10] Suen, An-Nan, Jhing-Fa Wang, and Jia-Ching Wang. Suen, An-Nan, Jhing-Fa Wang, and Jia-Ching Wang. VLSI Implementation of 3-D Sound VLSI Implementation of 3-D Sound GeneratorGenerator. IEEE Transactions on Consumer Electronics, pages 679-688, 1997.. IEEE Transactions on Consumer Electronics, pages 679-688, 1997.

Real-Time Auralization Real-Time Auralization of Sound in Virtual 3D of Sound in Virtual 3D

EnvironmentsEnvironments

by Scott McDermottby Scott McDermott

sdm1718@louisiana.edusdm1718@louisiana.edu