dynamic sound for android
DESCRIPTION
Slides for my AnDevCon talk from fall 2013, covering Pure Data from an Android app or game developer's perspectiveTRANSCRIPT
Dynamic Audio For Apps and Games
slides: is.gd/dynamic_sound_android
code: github.com/thillerson/dynamic-sound-android
Presentation tackmobile.com
is.gd/dynamic_sound_android !
github.com/thillerson/dynamic-sound-android !
Presentation tackmobile.com
About Me
• @thillerson, +thillerson
• Developer at Tack Mobile(tackmobile.com), @tackmobile
• Android, iOS, and Mobile Web
• Rails, Node, maybe Elixir (one day)
Presentation tackmobile.com
Rawk Star
About Me
Presentation tackmobile.com
About Me
• Bass
• Electronic Music
• Some day I hope to play this thing --------->
Presentation tackmobile.com
Sound Design Work: Start
• Alarm App
• Alarm Sounds
• Play Store: “tackmobile”
Presentation tackmobile.com
Sound Design Work: Adrift
• Puzzle Game - “Adrift” in App Store
• Background Music soundcloud.com/tackmobile
• Sound Effects
Presentation tackmobile.com
Programming Sound
• http://pragprog.com/book/thsound
• What we’re talking about today...
• ... and much more
Presentation tackmobile.com
When We Are Done, You Will
• Understand Sound Design
• Understand Procedural Audio
• Know what Pd is, and how libpd makes it awesome
• Be excited about Dynamic Audio in your apps and games
Presentation tackmobile.com
What is Sound Design?
Presentation tackmobile.com
A Sound Designer
Ben Burtt
Presentation tackmobile.com
A Sound Designer...
• Conceptualizes
• Captures or Designs
• Records
• Engineers
• Produces
• ... the sound for an experience
Presentation tackmobile.com
Sound Designer Skills
• Understand Sound
• Intuitive vs. Theoretical
• Music vs. Sound
• Audio Engineering
Presentation tackmobile.com
Sound Design in Mobile Apps
• Developers/Designers wear lots of hats
• Control
• Efficiency
• Asset size
Presentation tackmobile.com
Sound Design Principles
Presentation tackmobile.com
A Sound Designer’s Goals
• Audible Feedback
• Fulfill Expectations
• Immersion
• Emotional Response
Presentation tackmobile.com
Sound Design and Good UX
• Your app doesn’t need sound
• Expect to be turned off
• Don’t make me listen to you
• Listen on lots of devices/environments
• Engineer for small speakers
• Make your sound mean something!
Presentation tackmobile.com
Always
• Test with phone calls
• Test while playing music
• Have fallbacks - e.g. No music, just effects
• Provide control to the user
Presentation tackmobile.com
A Good Example: Clear
• Realmac Software
• Josh Mobley
• Meaningful Sound
• demo
Presentation tackmobile.com
Sound
Presentation tackmobile.com
Longitudinal Waves
Presentation tackmobile.com
Of Pressure
Presentation tackmobile.com
Through A Medium
Presentation tackmobile.com
Transverse Wave
Longitudinal Wave
Presentation tackmobile.com
• Longitudinal Waves
• Of Pressure
• Through a Medium
Sound
Presentation tackmobile.com
Properties of Waves
• A - Wavelength (Distance), Period (Time)
• B - Amplitude
• C - Frequency - 1/Period (measured in Hertz)
A
B
B
C 1 2 3
Presentation tackmobile.com
Fourier Series
• Based on his work ------->
• Periodic signals composedof simple oscillatingfunctions
• Sine/Cosine waves arefundamental
Presentation tackmobile.com
Oscillators
• Something that moves according to a periodic function
• A physical body
• An electronic device
Presentation tackmobile.com
Basic Electronic Sound
oscillator
voltage
magnet
pressurewaves
speaker
Presentation tackmobile.com
Digital Audio
• Stream of Numbers
• Signal
• Frequency and Amplitude
• DAC - Digital to Analog Converter
Presentation tackmobile.com
Dynamic Audio
Presentation tackmobile.com
Procedural Audio
• Functions
• Objects
• Controls and Components
• Produce a Signal
• “Building Sound from First Principles”
Presentation tackmobile.com
Pure Data
• Procedural Audio
• Visual Programming Environment
• Free and Open Source
• pure-data.info
Presentation tackmobile.com
Pure Data 101
Presentation tackmobile.com
Signals and Values
• Signals are streams of numbers, usually audio data
• Values are numbers or symbols
Presentation tackmobile.com
• Pd’s class
• or, kind of like a command line tool
Objects
osc~ 440
signal inlet inlet
signal outlet
class name arguments
Presentation tackmobile.com
Hot vs Cold Inlets
• Hot inlets make objects immediately act
• Cold inlets usually store a value
• Execution order is right to left
Presentation tackmobile.com
• Signal connections are thick
• Others are thin
Connections
osc~ 440
*~
dac~
1
Presentation tackmobile.com
Messages
• Used for sending values without a connection
• [send] and [receive] objects
• Message boxes
Presentation tackmobile.com
libpd
• http://libpd.cc
• Peter Brinkmann - Google Engineer
• Enables you to embed an instance of Pd in your app
• iOS and Android
Presentation tackmobile.com
libpd for Android
• Patches should be compressed raw resources
• Utilities for dealing with patches
• Provides interfaces for audio management
• PdBase bridges to Pd
• Pass messages, floats, and lists
• Provides PdService for long running audio
Presentation tackmobile.com
Examples
Presentation tackmobile.com
Source Code
• github.com/thillerson/dynamic-sound-android
• Examples of capabilities, not necessarily the best architecture
• Goal: understand capabilities, have a place to start
Presentation tackmobile.com
libpd and gradle
• At this time there’s no official support yet
• github.com/libpd/pd-for-android/pull/7 adds support.
• I have not tested this yet
Presentation tackmobile.com
Example 1: A Simple Keyboard
• Sine wave oscillator
• Single voice
• Note on/off
Presentation tackmobile.com
Example 1: A Simple Keyboard Patch
osc~
dac~
receive midinote
mtof receive gate
line~
*~
;
midinote 60;
gate 1;
;
gate 0$1 1
Presentation tackmobile.com
Example 2: Space Rocks
• Side scroller space shooter
• Very simple implementation - not production ready game code
• With dynamic sound!
Presentation tackmobile.com
Example 2: A Game Patchload a square waveat startup
Thruster
noise~
*~
samphold~
dac~
hip~ 200
phasor~ 800
line~
$1 200
*~ 0.1
receive thruster
BulletShip_HitAsteroid_Hit
samphold~
tabosc4~ square
dac~
line~
phasor~ 8000
tabosc4~ square
line~
unpack f f
0 $1
tabosc4~ square
line~
unpack f f
0 $1
hip~ 1000
unpack f f
0 $1
*~ 0.5
receive asteroid receive ship receive bullet
8-bit-ifythese sounds
cut low frequencies
commentcomment
Test_Messages;bullet 25 200;
;thruster 0
;thruster 1
pd wavesetup
;ship 1000 50;
;asteroid 200 100;
Presentation tackmobile.com
Programming Sound• http://pragprog.com/
book/thsound
Presentation tackmobile.com
Further Reading
• puredata.info && libpd.cc
• youtube.com/user/cheetomoskeeto
• http://designingsound.org
• mitpress.mit.edu/books/designing-sound
Presentation tackmobile.com
Questions
Thank you!Dynamic Audio • Tony Hillerson
• We’re Hiring!
• Excellent Team
• Awesome Projects
• Great Office