![Page 1: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/1.jpg)
Learning to Play Jazz with Deep Belief Networks
Greg Bickerman, Harvey Mudd College Sam Bosley, Stanford University Peter Swire, Brandeis University
Robert Keller, Harvey Mudd College
![Page 2: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/2.jpg)
MoHvaHon
• People are able to improvise jazz on the spot • Jazz ImprovisaHon
– PaKerned and structured – CreaHve and novel
• Could a machine learn to improvise as well as a human?
![Page 3: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/3.jpg)
MoHvaHon
• ArHficial jazz improvisers already exist – GenJam
• Supervised geneHc learning – Impro‐Visor
• Extensive musical knowledge built in
• Interested in unsupervised learning • Minimal representaHonal assumpHons
![Page 4: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/4.jpg)
Restricted Boltzmann Machines
• 2‐layer network – Visible layer – Hidden layer
• Nodes – Interconnected – Can be set ON or OFF
• Weights – Assigned to each connecHon – Symmetric
B
B
Visible Nodes
Hidden Nodes
![Page 5: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/5.jpg)
AcHvaHon
• Nodes acHvated probabilis)cally based on acHvaHon states of nodes in opposite layer – Compute weighted sum of acHve connecHons – AcHvaHon funcHon determines probability of firing
![Page 6: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/6.jpg)
AcHvaHon
B
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B ?
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B ?
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B 1
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B 1 ?
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B 1 ?
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
B 1 0
B 0 1 0
Visible Nodes
Hidden Nodes
0 1
![Page 7: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/7.jpg)
AcHvaHon
0 1 0
Visible Nodes
Hidden Nodes
0 1
![Page 8: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/8.jpg)
Input / Output
• Input – Binary data sequences – Mapped onto visible neurons
• Output – IdenHcally sized data sequences – Read off of visible neurons
![Page 9: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/9.jpg)
Input / Output
B
B
Visible Nodes
Hidden Nodes
1 1 0 0
Input
1 0 0 1
Output
![Page 10: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/10.jpg)
Input/Output
Visible Nodes
Hidden Nodes
1 1 0 0
Input
1 0 0 1
Output
![Page 11: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/11.jpg)
Training
• ContrasHve divergence method – AcHvate network normally
– AcHvate network with inputs “clamped” – Adjust weights to make normal acHvaHon behave more like clamped acHvaHon
![Page 12: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/12.jpg)
Deep Belief Networks
• Use individual RBMs as layers in a larger network
• Hidden layer of one RBM forms input layer of another
• If single RBMs learn features about data, DBMs learn features about features
Layer 1
Layer 2
Layer 3
![Page 13: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/13.jpg)
Encoding Scheme
• Requirements – Binary encoding – Music must be encoded in a string of standard length
– Each note must be the same “distance” from every other note
![Page 14: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/14.jpg)
Encoding Scheme
• Break melody into beat subdivisions
• Each subdivision contains 18 bits – 1 Sustain Bit – 1 Rest Bit – 12 Pitch Bits – 4 Octave Bits
S R C C# D D# E F F# G G# A A# B 1 2 3 4
Pitch Bits Octave Bits Rest Bit Sustain Bit
![Page 15: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/15.jpg)
Encoding Scheme
0 0 1 1 1 0 1
0 1 1 1 0 1
0 1 1 0 1 1 0 1 1
1 0 1
0
0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
0 0
0 0 0
0 0
0 0 0 0
0 0
0
0
0
0
S R C C# D D# E F F# G G# A A# B 1 2 3 4
Pitch Bits Octave Bits Rest Bit Sustain Bit
![Page 16: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/16.jpg)
Chord Encoding Scheme
1
1
0 0
0
0 0 1 1
0
0 0 0 0
0
0 0 0 0
0
0 0 0 0
0
0 0 1 1
0
1 1 0 0
0
0 0 0 0
0
0 0 1 1
0
0 0 0 0
1
1 1 0
0 0 1
0 0 0
1
C C# D D# E F F# G G# A A# B
Chord Bits
1 0 0 0 0 0 0 0 0 0 1 1
1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1
CM
FM
GM
CM
![Page 17: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/17.jpg)
IniHal Dataset
• Children’s songs – 2 measures
– 8th note resoluHon – Simple chords – 14 melodies
• Generated similar songs
![Page 18: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/18.jpg)
Main Dataset
• Jazz licks – 4 measures
– 12 beat subdivisions • (32nd triplet note resoluHon)
– ii‐V7‐I‐VI7 chord progression – 100+ licks
• Transcribed • HandwriKen
![Page 19: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/19.jpg)
Windowing
• Rather than training on enHre piece at once, break music into “windows”
• Start with the first measure of a lick and gradually move window forward
• Allows learning/generaHng arbitrary length music sequences with a fixed size network
![Page 20: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/20.jpg)
Windowing
![Page 21: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/21.jpg)
Windowing
![Page 22: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/22.jpg)
Windowing
![Page 23: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/23.jpg)
Windowing
![Page 24: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/24.jpg)
Windowing
![Page 25: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/25.jpg)
GeneraHng New Melodies
• Need to specify a chord sequence over which to generate a new melody.
• Chord bits are “clamped” during generaHon so that they can influence the melody being generated without changing themselves
![Page 26: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/26.jpg)
GeneraHng New Melodies
• Use a windowing strategy analogous to our windowed training method
• As each successive beat is generated the whole melody and chord sequence shijs forward to make room for the next beat
![Page 27: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/27.jpg)
GeneraHng New Melodies
Chord Seed
Random Data
Chord Seed
Generated Melody
DBN
![Page 28: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/28.jpg)
Results
• Rhythmically stable
• Respects chord tones
• Occasional color tones
• Very few foreign tones
![Page 29: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/29.jpg)
Results
• Trained on transposiHons as well – Generated music following key of given chord progression
– Succeeded with up to four transposiHons
![Page 30: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/30.jpg)
Example Training Licks
![Page 31: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/31.jpg)
Example Generated Licks
![Page 32: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/32.jpg)
More Generated Examples
![Page 33: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/33.jpg)
Random Music
![Page 34: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/34.jpg)
Future Work – Repeated Notes
• Our machines produced disproporHonate numbers of repeated notes
• Can sound staHc or too immobile for jazz
![Page 35: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/35.jpg)
Future Work – Repeated Notes
• Possible soluHon: post processing – Merge repeated notes together
– Results in a smoother output, but starts to cross line of unsupervised learning
• Ideally, machine should avoid repeated notes in the first place
![Page 36: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/36.jpg)
Future Work – Training Algorithm
• Slow – OpHmizaHon
– ParallelizaHon – AdapHve terminaHon
• SensiHve to training presentaHon order – Randomize training inputs
![Page 37: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/37.jpg)
Future Work – Chord Inference
• We believe our work naturally lends itself to the open problem of inferring unknown chords for a melody – Currently we provide a chord seed to generate a melody.
– If we instead provide a melody as input, we could determine which chords fit that melody
![Page 38: Learning to Play Jazz with Deep Belief Networks - Conference Version](https://reader034.vdocuments.us/reader034/viewer/2022051521/5869c3651a28ab357d8ba2f7/html5/thumbnails/38.jpg)
Conclusion
• Unsupervised learning algorithm
• Based on probabilisHc neural network theory
• Able to create novel jazz licks based on an exisHng corpus
• Minimal assumpHons about musical knowledge