chris grigg beatnik inc. chairman, mma tech board chrisg @ beatnik tom savell

New MIDI-Based Game Audio Techniques – Grigg / Savell

Chris GriggBeatnik Inc.Chairman, MMA Tech [email protected]

Tom SavellCreative Labs – Advanced Technology CenterMMA Tech [email protected]

New MIDI-Based Game Audio Techniques


What’s MIDI?• The Technology• The Standards Orgs

The New MIDI Standards1. iXMF: Interactive XMF2. 3D MIDI Controllers3. High Definition Protocol

for MIDI Devices

Questions

Session Overview


What’s MIDI?

What’s MIDI?


• MIDI 1.0 over 5-pin DIN• The Underlying Models• The Recommended Practices

GM, Show Control, MTC, MMC, GM2, SP-MIDI, etc.

• The File FormatsSMF, Extended RMID, XMF, Mobile XMF

• Support in Computer OS’sMac OS, Windows, BeOS, Linux etc.

• Support in Other Media StandardsMPEG-4 etc.

• Support in Audio Plug-In API’sVST, DX, AU etc.

• Support in Mobile Phones3GPP etc.

MIDI: The Technology


• MMA: MIDI Manufacturers Association (world except Japan)

• Tech Board

• Working Groups: Mobile WG XMF WG DLS WG Transport Layer WG High-Def WG

• IASIG: Interactive Audio SIG

• AMEI: Assoc. of Musical Electronic Industry (Japan only)

MIDI: The Standards Orgs


The New MIDI Standards

The New MIDI

Standards


iXMF: Interactive XMF

1

iXMF:Interactive

XMF


• First open, non-proprietary standard for a file format for interactive audio content

• Uses MMA’s XMF container technology• Allows game developers to more easily

create cross-platform titles• Makes it easier for audio professionals to

develop interactive content• Developed by the iXMF-WG inside IASIG• V1.0 spec about to be published• Prominent early adopter

iXMF: Overview


iXMF: Motivation (1)There Is

NoStandard

BuildProprietary

Tool

LimitedTime & $for Tool

LimitedTool

Quality

Harderto MakeContent

LimitedContentQuality

:-(

Do ItRight Next

Time

What’s theIndustry

Standard?

Break the‘Depressing Circle’!


BuildProprietary

Tool

LimitedTime & $

LimitedTool

Quality

Harderto MakeContent

LimitedContentQuality

:-(

Do ItRight Next

Time

What’s theIndustry

Standard?

iXMF: Motivation (1)There Is

NoStandard


• Many companies continually re-creating the wheel – in-house / proprietary• Much of what they do is the same

– only the format is different • This is redundant, parallel work –> no

synergies• User base (+budget) is very small,

so tool quality improves slowly (at best)• Tool quality tends to limit IA content

quality

• IA work done on one platform doesn’t port to any other platform--> Re-do for every platform--> IA tends to be seen as cost-prohibitive--> Less IA gets used

iXMF: Motivation (2)


• A Project BBQ idea• An IASIG Working Group

• Linda Law is Chair• Chris Grigg is Technical Lead

• Long Requirements phase• Draft Specification• Public Review period ended in May 2008• Final edits to v1.0 spec under way now• Release of v1.0 spec expected soon

(October?)

iXMF: Process (Standardization)


The iXMF File• Self-Contained IA Content Module• Includes both Media and Adaptive Behaviors

iXMF: Technical Design (1)

IXMF File

Media Files

Extension Data

Metadata

Playback Behavior Rules

IXMF File

Media Files

Extension Data

Metadata

Playback Behavior Rules


Runtime Architecture• iXMF File feeds IA Engine


Host Application

Host Audio System

Players

Cues

MIDI DLS

IXMF File

Players Players

Audio

iXMF Engine

Host Application

Host Audio System

Players

Cues

MIDI DLS

IXMF File

Players Players

Audio

iXMF Engine

Pla

tfo

rm P

rovi

de

dM

idd

lew

are

Ga

me


Cues – soundtrack element objects• iXMF file contains zero or more Cues• Cue = one soundtrack element –

a piece of music, a sfx, or a change to music/sfx• Game requests sound/music element

by starting a Cue:• by name: startCue( “mainTitlesMusic” );• by number: startCue( kMainTitlesMusic );

• In iXMF file, Cue contains (or references):• zero or more media Chunks• some Scripts



Scripts – methods / events of Cues• Stored in iXMF file in portable binary format• Variables – iXMF Global / Cue class / Cue instance • Expressions – all the C operators

• User Subroutines – call another user Script

• Built-In Statements• Generic programming stuff• Specialized IA primitives,

to minimize the need for programming• Extension Statements• No user-defined Statements! – Keep it simple

• Tools can hide Scripts behind GUI



Automatic Behaviors (“Engine Model”)

• Script hooks:• IxmfFileLoad, IxmfFileUnload• CueStart, CueStop, CueCancel• CommitPoint, ChangeOver• Markers can call Scripts

• NextChunk chaining• Mix Transitions – crossfade, sync points/rules• SyncGroups• MixGroups• Media LRU tracking• Scripts can invoke game Callbacks• Game can read/write iXMF variables



Custom Extensibility:• Add Script commands

• “Opcode Spaces”• Add custom media formats:

• Platform-specific audio formats• Generative music• Even video, graphics

• Extension Area in every iXMF data structure

Community Extensibility:• Future Profiles of the v1.0 standard • Future versions of the iXMF standard



Portability• Cues & Scripts will work properly

on any iXMF player IF:• The media formats used are supported AND• ( No extensions are used by content

OR Any extensions used are supported OR Any extensions used aren’t critical )

• Profiles can manage portability expectations• Content is often mastered for specific

platforms – portability not an issue there• A high-res iXMF “source” file can give

authoring-time portability• Export to platform-specific runtime formats



• iXMF is non-proprietary• iXMF is royalty-free to use or implement• So: Expect multiple runtime implementations,

platforms, authoring tools --> Ecosystem• Commercial --> Competition --> Improvement• Expect interoperability for base features / profiles• Don’t expect custom extensions to be portable• Look for integration with

game dev / media mgmt databases• Hope that the IA community can start

speaking a common language• Expect vendors & IASIG to be responsive• Expect IA to become the norm, not exotic

iXMF: The Ecosystem


V1.0 Spec releasing soon. Then:• New Versions• New Features• New Media Types• Profiles for specialization as needed – e.g.:

• Advanced Games• Simple/Casual Games• Mobile Phones/Games• Musical Instruments• DJ Applications• Location-Based Entertainment• Little-Endian

Join the iXMF Working Group to participate!

iXMF: The Future


3D MIDI Controllers

2

3D MIDIControllers


3D MIDI: Overview

Provides set of controllers for independent positioning on each MIDI channel

Uses spherical coordinate system: azimuth angle, elevation angle, and distance

Seamless integration with legacy pan control

Egocentric approach describes head-related 3D sources

Well-defined standardized behavior Extensible through MMA standardization

process V1.0 to be published soon.


3D MIDI: Position Controllers Registered Parameter Number (RPN)

Bank 0x3D Azimuth and Elevation Angular

Resolution of 0.02 degrees from -180 to +179.98 degrees

Distance represented as ratio of distance to a maximum distance: e.g. 0.000061 is 6.1 centimeters if max. distance is 1km.


3D MIDI: Pan Spread and Roll Angle Pan Spread defines the 3D angular

distance of the MIDI stereo field represented by the legacy pan controller

Roll Angle defines the angle relative to horizontal of the MIDI stereo field represented by the legacy pan controller


3D MIDI: Position Model


3D MIDI: Gain Parameters Gain parameters provide content author

control Gain

Independent of CC#7/11 Gain at max distance

Gain when distance ratio is 1.0 Reference distance

No additional gain below this distance ratio


3D MIDI: Distance Model

attenuation

distance

0 dB

max_attenuation

reference_distance max_distance


3D MIDI: Ex. Distance of a Fly The buzz of a fly would typically not

be heard beyond 10 meters away maximum_distance = 10

Sounds significantly louder a few centimeters away from your ear reference_distance_ratio = 0.01 --

between 0 and 10 cm, the fly sound is not attenuated


3D MIDI: Future Enhancements Reverberation Controls Source / Reverb mix dependency

on distance. Environmental effects (e.g. occlusion) Could be made compatible

with IA-SIG I3DL2 guideline Subject to MMA Standardization

Process


High Definition Protocol

for MIDI Devices3

HD Protocolfor

MIDI Devices


• Whole new generation of the basic MIDI message protocol

• Address shortcomings of the MIDI 1.0 protocol (1983)

• Grow the worldwide MIDI markets by opening up new functionality,new musical expressiveness

• HD work started in 2006 by MMA Tech Board

• Standardization work still under way• Overview Only!

All Technical Details Subject to Change!

HD-PMD: Introduction


MIDI 1.0 Technology Needs an Update:• MIDI 1.0 first released in 1983• HW transport slow / Timing not tight• Complicated msg structure / Parsing:

• Many multi-message functions• Message size varies• Byte aligned

• Hardware orientation --> Hard to write SW for • Advanced features difficult to access• Expansion room exhausted long agoDesired New Functionality Includes: • Better support for non-keyboard instrument

models (strings, voices, horns, winds etc.)• Better support for non-western music• Better support for instruments with continuous

pitch / very fine pitch nuance

HD-PMD: Motivation


• MMA Working Group in US/Europe• AMEI Working Group in Japan• Joint & Parallel technical work• Weekly WG teleconferences• WG email reflector• MMA/AMEI coordination via email• MMA/AMEI Joint Face-to-Face meetings

in LA / SF / Tokyo 2-3x per year

HD-PMD: Standards Process


• Transports:An ongoing discussion –• Traditional wired transports• Software-only transports:

• Network/comm protocols• Plug-In APIs• OS MIDI services APIs

• Messages• Significantly reconfigured • Simplicity• Future extensibility• New musical functionality

HD-PMD: Technical Design


• Uniform Packet 64/128 bits (8/16 bytes)

Simplifies transports - Memory efficient(MIDI 1.0 has 1, 2, 3, or N-byte messages)

• Message = 1 or More Packets• Single-Packet Messages (SPM)• Multi-Packet Messages (MPM)

• Fixed # of Packets• Variable # of Packets (VLM)

HD-PMD: Messages


• MIDI 1.0• Totally full.• Every “Status Nybble”

either in use, or else officially deprecated

• Extensions very cumbersome

• HD is shown at right• Lots of expansion room• Especially with sub-commands

HD-PMD:Opcode SpacesCompared

QuickTime™ and a decompressor

are needed to see this picture.


Notes:• More Velocity resolution (16-bit – 512x MIDI 1.0 depth)• Precise Pitch option (1/256 semitone resolution)• Trigger Index option (like MIDI 1.0 Note Number)

Controllers:• More Value resolution (16/32-bit – 2-4x MIDI 1.0 depth)• More Controller Numbers available (32x MIDI 1.0 space)• No more MSB / LSB / Data Entry!• CC is a unified message, including Bank number• RPN now just another controller number• NRPN now a unified message – all MIDI 1.0 numbers available• Per-Channel and Per-Note Controllers, RPNs, and NRPNs

Addressing:• More Channels (7 bits - 8x MIDI 1.0 space)• Groups concept - Add’l dimension of addressing

(5 bits - effectively 256x MIDI 1.0 channel space)

HD-PMD: New Functionality

(Proposed – Under Discussion)


• More joint MMA/AMEI Meetings• Draft v1.0 Specification (English)

is in progress• Experimental implementations of

candidate transports ongoing• Decide on v1.0 Spec(s) release date(s)• Decide on possible additional

Recommended Practices for HD

After v1.0 Spec Release:• Continuing process to standardize new

features… forever… and ever… and ever…

HD-PMD: What’s Next?


Questions?

Questions?


• YES you can join the iXMF Working Group…

• YES you can get a copy of the v1.0 iXMF Spec soon…

…at www.iasig.org

• YES you can get a copy of the 3D MIDI Spec soon…

• YES if you are commercially involved in designing or

producing MIDI HW or SW you can join the MMA and the MIDI High Def Working Group…

…at www.midi.org

Answers!


Chris GriggBeatnik Inc.Chairman, MMA Tech [email protected]

Tom SavellCreative Labs – Advanced Technology CenterMMA Tech [email protected]

(You just sat through)

New MIDI-Based Game Audio Techniques

chris grigg beatnik inc. chairman, mma tech board chrisg @ beatnik tom savell

Documents