a roadmap: beyond big api’s why software component frameworks may be interactive audio’s future...
TRANSCRIPT
A Roadmap:A Roadmap:Beyond Big API’sBeyond Big API’s
Why Software Component Frameworks Why Software Component Frameworks
May Be Interactive Audio’s FutureMay Be Interactive Audio’s Future
Chris Grigg [email protected]
ThesisThesisBig audio API’s compromiseBig audio API’s compromise
delivered audio qualitydelivered audio quality
• • To use them, you have to call their functions; To use them, you have to call their functions;
• • Every API uses a different interface;Every API uses a different interface;
• • So Mixing or Changing APIs is too hard.So Mixing or Changing APIs is too hard.
• • Many custom API’s & media types have no tools.Many custom API’s & media types have no tools.
This sucks because:This sucks because:
• • No single API does everything well, or runs on all platforms.No single API does everything well, or runs on all platforms.• Time invested in custom API features doesn’t port.• Time invested in custom API features doesn’t port.
• • Not about specific faults of particular API’sNot about specific faults of particular API’s
• • A Structural Problem of all big audio API’sA Structural Problem of all big audio API’s
• • Programming itselfProgramming itself the bottleneck. the bottleneck.
ThesisThesis
We want to be able to construct runtime We want to be able to construct runtime audio systems out of arbitrary functional audio systems out of arbitrary functional blocks from multiple vendorsblocks from multiple vendors
Our Studios Have Many Different Synthesizer BrandsOur Studios Have Many Different Synthesizer Brands
Because They’re All Good At Different Things –Because They’re All Good At Different Things –
Our Runtime Systems Need Similar DiversityOur Runtime Systems Need Similar Diversity
In this ‘What If’ world:In this ‘What If’ world:
• • Units of software organization become SmallerUnits of software organization become Smaller
• • They become ModularThey become Modular
• • Each Unit Takes On a Specific FunctionEach Unit Takes On a Specific Function
• • Units Have to be able to Talk to one anotherUnits Have to be able to Talk to one another
An Architecture Fundamentally DifferentAn Architecture Fundamentally Differentfrom our current big, do-everything from our current big, do-everything
Audio API development & use practices.Audio API development & use practices.
Things We NeedThings We NeedTo Get ThereTo Get There
11 Software Component Framework Software Component Framework with Object Model with Object Model
22 Audio Signal Interconnect Mechanism Audio Signal Interconnect Mechanism
33 Control Signal Interconnect Mechanism Control Signal Interconnect Mechanism
44 Scriptable Control Scriptable Control
11
A StandardizedA Standardized
Audio Software Component FrameworkAudio Software Component Framework
with Object Modelwith Object Model
If our audio blocks are components,If our audio blocks are components,
we need a framework to hold them.we need a framework to hold them.
Typical Component Typical Component FrameworkFramework
Framework
SoftwareComponent
A
Manager MaintainsManager MaintainsComponent InstancesComponent Instances& Directs Host Calls& Directs Host Calls
Instances of VariousInstances of VariousComponent ClassesComponent Classes
SoftwareComponent
B
SoftwareComponent
C
SoftwareComponent
D
Host Application
Framework Manager
GUI Authoring ToolGUI Authoring ToolUses Same Framework & Components As RuntimeUses Same Framework & Components As Runtime
Select, Configure, & Connect ComponentsSelect, Configure, & Connect ComponentsSave Configuration for Use at RuntimeSave Configuration for Use at Runtime
Having a Uniform,Having a Uniform,Implementation-IndependentImplementation-Independent
Object Model allows:Object Model allows:
• • Single, Universal Authoring ToolSingle, Universal Authoring Tool
• • Cross-Platform ContentCross-Platform Content
• • Cross-Platform Runtime Communication Cross-Platform Runtime Communication
2 & 3:2 & 3:InterconnectionInterconnection
Audio Signals & Control SignalsAudio Signals & Control Signals
Connecting Anything to AnythingConnecting Anything to Anything
Requires Control & Audio Interconnect Requires Control & Audio Interconnect StandardsStandards
Most Component FrameworksMost Component Frameworks
Don’t Have InterconnectionsDon’t Have Interconnections
to Deal withto Deal with
No off-the-shelf solution –No off-the-shelf solution –
We need a development effortWe need a development effort
2
Audio Interconnection MechanismAudio Interconnection Mechanism
Between ComponentsBetween Components
Audio Signal FlowAudio Signal Flow
Between ComponentsBetween Components
……Doesn’t Require Multiple BuffersDoesn’t Require Multiple Buffers
Audio Signal FlowAudio Signal Flow
Using Single BufferUsing Single Buffer
Multiplexed Operations on a Single BufferMultiplexed Operations on a Single Buffer
RepresentingRepresentingInterconnectionsInterconnections
in thein the
Framework Object ModelFramework Object Model
Block Diagram ViewBlock Diagram View
Object Model View:Object Model View:Components and ConnectionsComponents and Connections
Everything derives fromEverything derives fromclass Componentclass Component and and class Connectionclass Connection
Benefits I
• • Better Programmer/Soundmaker CommunicationBetter Programmer/Soundmaker Communication
• • Increases Clarity of PurposeIncreases Clarity of Purpose
• • Simpler, Faster Code DevelopmentSimpler, Faster Code Development
• • Promotes Audio API ProliferationPromotes Audio API Proliferation
• • Encourages View of Audio APIs as InterchangeableEncourages View of Audio APIs as Interchangeable
• • Simplifies Porting of Game Engines and TitlesSimplifies Porting of Game Engines and Titles
• • Encourages Porting Components to New PlatformsEncourages Porting Components to New Platforms
A Message-Based MechanismA Message-Based Mechanismfor Control Signal Interconnectionfor Control Signal Interconnection
Between Software ComponentsBetween Software Components
33
Uh… why “control signals”?Uh… why “control signals”?
To avoid messes like this.To avoid messes like this.
Rat’s Nest of Function CallsRat’s Nest of Function Calls
Like MIDI CablesLike MIDI Cables
for Software Componentsfor Software Components
Unified Control Signal – Single-Point ConnectionUnified Control Signal – Single-Point Connection
Assemble Controlled Sound SystemsAssemble Controlled Sound Systems
from Software Componentsfrom Software Components
Like a Recording Studio:Like a Recording Studio:Audio Connections & Control ConnectionsAudio Connections & Control Connections
Host Game UsesHost Game UsesSame Control MechanismSame Control Mechanism
Still Like a Recording Studio:Still Like a Recording Studio:Audio Connections & Control ConnectionsAudio Connections & Control Connections
Components Can ProcessComponents Can ProcessControl SignalsControl Signals
(not just audio)(not just audio)
Examples: Y-Cord, Merger, AdaptorExamples: Y-Cord, Merger, Adaptor
Easy Communication withEasy Communication withExternal Devices Using Same ModelExternal Devices Using Same Model
Consistent Binary Control Signal FormatConsistent Binary Control Signal FormatMakes This PossibleMakes This Possible
Multi-Component ‘Templates’Multi-Component ‘Templates’
Add Configuration PowerAdd Configuration Power
Example: Mixer Channel Strip TemplateExample: Mixer Channel Strip Template
Many Components Instantiated in Single StepMany Components Instantiated in Single Step
with Internal Routing, Configuration, Initial Settingswith Internal Routing, Configuration, Initial Settings
Control Signals Give Us KnobsControl Signals Give Us KnobsBoth the Programmer Both the Programmer andand the Soundmaker the Soundmaker
So what kinds of messagesSo what kinds of messagesare in the Control Signals?are in the Control Signals?
Standardize theStandardize theControl Signal ConnectionControl Signal Connection
But But NotNot the Command Set, the Command Set,Parameter Set, etc.Parameter Set, etc.
Arbitrary MessagesArbitrary MessagesOn The ConnectionOn The Connection
How Do You Know What MessagesHow Do You Know What Messagesa Component Accepts?a Component Accepts?
You Ask It.You Ask It.
Don’t Predefine the Message Vocabulary.Don’t Predefine the Message Vocabulary.
Let Every Component Define Its Own.Let Every Component Define Its Own.
Every Component HasEvery Component Has
Its Own Command SetIts Own Command Set
…And …And PublishesPublishes It It
This Is How The GUI Authoring ToolThis Is How The GUI Authoring ToolKnows What Each Component Can Do.Knows What Each Component Can Do.
Simple Piano ComponentSimple Piano ComponentTwo Control InputsTwo Control Inputs
Simple ExampleSimple Example
Simple Piano ComponentSimple Piano ComponentControl Interface DescriptionControl Interface Description
ControlInput1:Name: MIDI InProtocol: MIDI Messages
Interpretation: (MIDI Implementation Chart appears here) ControlInput2:
Name: Sustain Switch InProtocol: Boolean MessagesInterpretation: low bit 1 = pedal down, 0 = up
Ask the Component for its Interface Description and you get a Ask the Component for its Interface Description and you get a list of the Messages that its Control Inputs Understandlist of the Messages that its Control Inputs Understand
3D Panner Component3D Panner ComponentOne Control InputOne Control Input
More Complex ExampleMore Complex Example
3D Panner Component3D Panner ComponentControl Interface DescriptionControl Interface Description
Commands:SetVolume( Fixed16 ) -- Master out volumeMute( Boolean ) -- Mute (TRUE) or Pass audio
SetSpatialPosition( Std3DVector ) -- 3-space
The Messages that the Control Input UnderstandsThe Messages that the Control Input Understands
But Weren’t We Trying to Get AwayBut Weren’t We Trying to Get Away
from Function Calls?from Function Calls?Commands:
SetVolume( Fixed16 ) -- Master out volumeMute( Boolean ) -- Mute (TRUE) or Pass audio
SetSpatialPosition( Std3DVector ) -- 3-space
This looks just like a Big Audio API!This looks just like a Big Audio API!
True, this semantic is similar to the HLL function calls...True, this semantic is similar to the HLL function calls...
But it’s expressed differently at runtime.But it’s expressed differently at runtime.
SetVolume( Fixed16 ) Mute( Boolean )
SetSpatialPosition( Std3DVector )
Not as compile-based function calls, but Not as compile-based function calls, but rather inrather in Data-Based Messages Data-Based Messages that can be that can be
routed dynamically at runtime.routed dynamically at runtime.
How do Components Publish theirHow do Components Publish theirInterface DescriptionsInterface Descriptions
at Runtime and Authoring Time?at Runtime and Authoring Time?
Some Invention Is Required Here.Some Invention Is Required Here.
Learn from existing solutions to similar problems:Learn from existing solutions to similar problems:• • COMCOM
• • CORBA’s IDLCORBA’s IDL
• • Apple Event Terminology ResourcesApple Event Terminology Resources
• • XML’s DTDXML’s DTD
• • MAXMAX Objects Objects
44
Adding theAdding the
Scripting LayerScripting Layer
Scripted Control of allScripted Control of allAudio Components & MediaAudio Components & Media
With Enough Component Types and Media Types, With Enough Component Types and Media Types,
the ‘Ultimate Interactive Audio Architecture’the ‘Ultimate Interactive Audio Architecture’
Sound Dept. DeliversSound Dept. DeliversPlatform-IndependentPlatform-Independent
Media & ScriptsMedia & Scripts
Components, FrameworkComponents, Framework& Script Player Get& Script Player GetPorted to All PlatformsPorted to All Platforms
Game UsesGame UsesSame Sound Event CuesSame Sound Event CuesOn All PlatformsOn All Platforms
Benefits IIBenefits II
• • Simplified Control of Sound From GameSimplified Control of Sound From Game
• • Increased API Experimentation FlexibilityIncreased API Experimentation Flexibility
• • Flexible Patching of Control Sources & DestinationsFlexible Patching of Control Sources & Destinations
• • Encourages Plug-In Style DevelopmentEncourages Plug-In Style Development
• • Ability to Transform Game Control VariablesAbility to Transform Game Control Variables
• • Sound Designer Control over InteractivitySound Designer Control over Interactivity
• • Auto-Porting of Auto-Porting of InteractiveInteractive Audio Content Audio Content
ConclusionConclusionAn essentially non-commercial idea:An essentially non-commercial idea:
• • The limits of big audio APIs are structural.The limits of big audio APIs are structural.
• • They favor programmer contributions while They favor programmer contributions while marginalizing soundmaker contributions – through marginalizing soundmaker contributions – through poor or missing tools, but also by virtue of their poor or missing tools, but also by virtue of their fundamental compile-based architectures. fundamental compile-based architectures.
• • This dynamic hurts the delivered audio quality.This dynamic hurts the delivered audio quality.
• • Open standards for component software appear to Open standards for component software appear to offer a way out, and might serve as the basis of a offer a way out, and might serve as the basis of a generalized interactive audio methodology as generalized interactive audio methodology as universal and stable as MIDI has been. universal and stable as MIDI has been.
Because Media is a Harsh Master.Because Media is a Harsh Master.