synthesis by example creation of sound synthesis algorithms using evolutionary methods by: ricardo...
TRANSCRIPT
SYNTHESIS BY EXAMPLE
Creation of sound synthesis algorithms using evolutionary methodsBy: Ricardo A. García [email protected]
MIT - Media LabMachine Listening Group© 2001
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
INTRODUCTION
How is that sound coming out of your soundcard being generated?
Is that a real violin? What kind of controls do we have over it?
• pitch, length, other performance parameters • sound attributes: brightness, harmonic content, etc
The questions: How are those sound synthesis algorithms
conceived? Can this design process be automated?
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
In a nutshell...
User supplies a set of examples (inputs/target output) The system ”reverse engineers" the sound and
returns a sound synthesis technique that is capable of rendering a sound "similar" to the target sound!
?Inputs
Example
Output
TARGET
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Talk outline
Introduction Sound Synthesizers Design of Sound Synthesis Techniques (SST) Proposed approach for design Genetic Programming Automatic Generation of Sound Synthesizers
(AGeSS) system Examples
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
What is a sound synthesizer?
Any method or device used to produce sound Sound Synthesis Technique (SST): Algorithm that
produces sound samples, usually implemented in a digital computer
Inputs: – internal parameters (init time)– control signals (time varying)
Output: – sound samples (Waveform)
Structured audio, symbolic representationModel analysis
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Inside a SST
Functional form of the algorithm Hard to do!• Prefabricated formulas and algorithms
Y=A*sin(x+B)+C
Internal parameters “Easy to do”• Parameter estimation• Analysis tools
A=0.4, B=0.1, C=-0.3
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Traditional design
Pick a Functional Form Do parameter estimation to fit your target
…
LPC
FM
Additive
ParameterEstimation
TARGETInputs
USE IT
Human selects Functional Form
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Proposed approach “Suggest” functional form (suggest “new” SST) Do parameter estimation Compare with target Suggest new SST
ParameterEstimation
TARGETInputs
SuggestSST
functional form + Initial Parameters
Functional ElementsCompare
with Target
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Representation of SSTs
Pseudocode
Formula
Topology graph
assign 3 to xsum x and 0.4return x
)(2sin3.0 tkfy
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
SST space
SST space: all the possible valid combinations of a set of functional elements and their connections! HUGE!
Hypothesis: ”given a set of inputs, target and fitness metrics, it is possible
to find the functional-form and internal parameters of a SST capable of synthesizing a sound “close” to the target”.
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Design of SST as a search
Specs: Inputsoutput (TARGET)Fitness metric
Search: find a point (SST, Functional form + internal parameters) that fulfills the specs
EXAMPLE
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
The Genetic Programming Loop
O p tim ize in te rn a l P a ram e te rs
R u n to p o lo g y& ren d e r so u n d
C o m p a re ren d e re d so u n dan d ta rg e t so u n d
(F itn ess F u n c tio n )
S e le c t R an d o m In d iv id u a lb ased o n fitn e ss
C rea te N ew In d iv id u a lsu s in g G e n e tic O p era tio n s
(co p y, c ro sso v e r, m u ta tio n )
R ep lace o ld p o p u la tio nw ith n ew p o p u la tio n
S TA RTC rea te n ew
ran d o m p o p u la tio n
N ew P o p u la tio n
Write fitn ess va lue fo r each ind iv id u a l
If f itn e ss v a lu e < d esired f itn essth e n F IN IS H (g o a l a c co m p lish ed )
R en d e red
Targ e t
C o n tro l in p u ts
8 7 .2 3
3 3 2 .8
2 3 6 .6
9 11 .0 1
2 8 7 7 .2 2
HUMANVs
MACHINE
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Comparing sounds --fitness function--
Analytical distance metrics:MSE of complex spectrogram(magnitude and phase)
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Perceptual Fitness Function
A perceptual distance metricsPerceptually relevant components: full errorAbove masking threshold: error up to M.T.Below masking threshold: no error
O (i,j)T (i,j)
TM (i,j)
fu ll e rro r
O (i,j)T (i,j)
TM (i,j)e rro r u p to TM (i,j)
O (i,j)T (i,j)
TM (i,j)
n o e rro r
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Developmentmapping from expression tree to topology graph
S te p 0
S te p 1
S te p 2
S te p 3
S te p 4
S O U R C E 0
S O U R C E 1
1R E N D E R 0
S O U R C E 0
S O U R C E 1
1R E N D E R 0M U LT
S O U R C E 0
S O U R C E 1 R E N D E R 03 2
1M U LT
S O U R C E 0
S O U R C E 1 R E N D E R 03 2
1M U LT
A D D
S O U R C E 0
S O U R C E 1 R E N D E R 03 2
1M U LT
A D DK O S C IL
E M B RY O
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
AGeSS system
Automatic Generation of Sound Synthesizers Matlab scripts and C++ programs
U S E RA G eS S
S Y S T E M
In p u ts- R u n p a ram e te rs- E x am p le s
O u tp u tS S T (to p o lo g y g rap h )
In p u ts Ta rg e t
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
Examples
A synthetic tone (des35comp2.avi)
A piano note (des44comp2.avi)
FM synthesizer (chow727.avi)
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
FM synthesis example
Target formula
Output (generation 220+)
FS
MIM
FS
CtAts f
ff 2sin22sin)()(
2010021 )()()()()( ktfkosciltfosciltftfkosciltAkout
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
FM synthesis example (scale)
Training example (A880, B988)
Output (best of generation 850)
Scale
Scale with best of generation 850
MIT - Media LabMachine Listening Group
Synthesis by ExampleRicardo A. Garcia © 2001
For more information…
Ricardo A. GarciaEmail: [email protected]: http://www.media.mit.edu/~rago