Upload File

let’s talk about bach

59
Let’s talk about Bach

Upload: others

Post on 26-Oct-2021

5 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Let’s talk about Bach

Let’s talk about Bach

Page 2: Let’s talk about Bach

Automatic composition https://soundcloud.com/bachbot

https://soundcloud.com/bachbot
Page 3: Let’s talk about Bach

Let’s talk about Bach BachBot

Page 4: Let’s talk about Bach

BachBot Autmatic Stylistic Composition with LSTM

Feynman Liang GOTO Amsterdam, 13 June 2017

Page 5: Let’s talk about Bach

About Me

• Software Engineer at

• MPhil Machine Learning @ University of Cambridge

• Joint work w/ Microsoft Research Cambridge

Page 6: Let’s talk about Bach

Executive summary• Deep recurrent neural network model for music capable of:

• Polyphony

• Automatic composition

• Harmonization

• Learns music theory without prior knowledge

• Only 7% out of 1779 participants n a musical Turing test performed better than random chance

Page 7: Let’s talk about Bach

Goals of the work

• Where is the frontier of computational creativity?

• How much has deep learning advanced automatic composition?

• How do we evaluate generative models?

vs

Page 8: Let’s talk about Bach

Overview• Music primer

• Chorales dataset preparation

• RNN primer

• The BachBot model

• Results

• Musical Turing test

Page 9: Let’s talk about Bach

If you’re a hands-on typefeynmanliang.github.io/bachbot-slides

docker pull fliang/bachbot:aibtb docker run --name bachbot -it fliang/bachbot:cornell bachbot datasets prepare bachbot datasets concatenate_corpus scratch/BWV-*.utf bachbot make_h5 bachbot train bachbot sample ~/bachbot/scratch/checkpoints/*/checkpoint_<ITER>.t7 -t bachbot decode sampled_stream ~/bachbot/scratch/sampled_$TMP.utf docker cp bachbot:/root/bachbot/scratch/out .

http://feynmanliang.github.io/bachbot-slides
Page 10: Let’s talk about Bach

Music primer

Page 11: Let’s talk about Bach

Modern music notation

Page 12: Let’s talk about Bach

Pitch: how “high” or “low” a note is

Page 13: Let’s talk about Bach

Duration: how “long” a note is

Page 14: Let’s talk about Bach

Polyphony: multiple simultaneous voices

Page 15: Let’s talk about Bach

Piano roll: convenient computational representation

Page 16: Let’s talk about Bach

Fermatas and phrases

Page 17: Let’s talk about Bach

Chorales dataset preparation

Page 18: Let’s talk about Bach

http://web.mit.edu/music21/

http://web.mit.edu/music21/
Page 19: Let’s talk about Bach

Transpose to Cmaj/Amin (convenience), quantize to 16th notes (computational)

Page 20: Let’s talk about Bach

Transposition preserves relative pitches

Page 21: Let’s talk about Bach

Quantization to 16th notes preserves meter and affects less than 0.2% of dataset

Page 22: Let’s talk about Bach

Question: How many chords can be constructed from 4 voices, each with 128 pitches?

Answer: O(1284)! Data sparsity issue…

Handling polyphony

Page 23: Let’s talk about Bach

Serialize in SATB order

START (59, True) (56, True) (52, True) (47, True)

||| (59, True) (56, True) (52, True) (47, True)

|||

(.) (57, False) (52, False) (48, False) (45, False)

||| (.)

(57, True) (52, True) (48, True) (45, True)

||| END

O(1284) => O(128) vocab. size!

Page 24: Let’s talk about Bach

Size of pre-processed dataset

Page 25: Let’s talk about Bach

Recurrent neural networks (RNN) primer

Page 26: Let’s talk about Bach

Neuron Input x, output y, parameters w, activations z

Page 27: Let’s talk about Bach

Feedforward neural network

Page 28: Let’s talk about Bach

Memory cell

Page 29: Let’s talk about Bach

Long short-term memory (LSTM) cell

Page 30: Let’s talk about Bach

Stacking memory cells to form a deep RNN Unrolling for training

Page 31: Let’s talk about Bach

The BachBot model

Page 32: Let’s talk about Bach

Sequential prediction training criteria

https://karpathy.github.io/2015/05/21/rnn-effectiveness/

https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 33: Let’s talk about Bach

Model formulationRNN dynamics Initial state (all 0s) Prob. distr. over sequences

+

Need to choose the RNN parameters…

…in order to maximize the probability of the real

Bach chorales.

=

Page 34: Let’s talk about Bach

Back-propagation

Page 35: Let’s talk about Bach

Optimizing model architecture

Page 36: Let’s talk about Bach

GPUs deliver a 8x performance speedup

Page 37: Let’s talk about Bach

Depth matters (to a certain point)

Page 38: Let’s talk about Bach

Hidden state size matters (to a certain point)

Page 39: Let’s talk about Bach

LSTM memory cells matter

Page 40: Let’s talk about Bach

The final model• Architecture

• 32-dimensional vector space embedding

• 3 layered, 256 hidden unit LSTMs

• “Tricks” during training

• 30% dropout

• Batch normalization

• 128 timestep truncated back-propagation through time (BPTT)

Page 41: Let’s talk about Bach

Dropout improves generalization

Page 42: Let’s talk about Bach

Automatic composition

• Sample next symbol from

• Condition model on sampled symbol

• Repeat

https://karpathy.github.io/2015/05/21/rnn-effectiveness/

https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 43: Let’s talk about Bach

Harmonization

• Let denote the fixed parts

• Need to choose highest probability symbols for the free parts

• Greedy 1-best selection

Page 44: Let’s talk about Bach

Results

Page 45: Let’s talk about Bach

Activations are difficult to interpret! Input and memory cell (layer 1 and 2) activations

Page 46: Let’s talk about Bach

Layers closer to the output resemble piano roll (consequence of sequential training criteria) Memory cell (layer 3) activations, output activations, and predictions

Page 47: Let’s talk about Bach

Model learns music theory!

• L1N64 and L1N138: Perfect cadences with root position chords in tonic key

• L1N151: A minor cadences ending phrases 2 and 4

• L1N87 and L2N37: I6 chords

Page 48: Let’s talk about Bach

Harmonization: given a melody (here C major scale)

Page 49: Let’s talk about Bach

Harmonization: produce the accompanying parts https://soundcloud.com/bachbot

https://soundcloud.com/bachbot
Page 50: Let’s talk about Bach

Harmonization: can also be used on non-Baroque melodies! https://soundcloud.com/bachbot

https://soundcloud.com/bachbot
Page 51: Let’s talk about Bach

Musical Turing test http://bachbot.com

http://bachbot.com
Page 52: Let’s talk about Bach
Page 53: Let’s talk about Bach
Page 54: Let’s talk about Bach

Participants by country

Page 55: Let’s talk about Bach

Participants by age group and music experience

Page 56: Let’s talk about Bach

Correct discrimination rates for composition (SATB) and harmonization (others) questions

Page 57: Let’s talk about Bach

More experienced respondents tend to do better

Page 58: Let’s talk about Bach

Conclusion• Deep LSTM generative model for composing, completing, and generating

polyphonic music

• Open source (github.com/feynmanliang/bachbot)

• Integrated into Magenta’s “Polyphonic RNN” model (magenta.tensorflow.org)

• Appears to learn music theoretic without prior knowledge

• Largest music Turing test to date with over 1779 participants

• Average performance on music Turing test only 7% better than random guessing

http://github.com/feynmanliang/bachbot
http://magenta.tensorflow.org
Page 59: Let’s talk about Bach

Thank You!

• Questions?

• Need a world-class development team for your next project?

• Email me at: [email protected]

mailto:[email protected]?subject=

Let’s Talk Policy

Let’s Talk Trash

Let’s talk about…

Let’s talk Scrum

Let’s Talk Tech

Let’s Talk About Countries

Let’s talk a bout SOAP, baby. Let’s talk a bout UPnP

Let’s Talk History

Let’s talk about food

Poverty& International Trade. Agenda Oxfam Oxfam Let’s talk about globalization Let’s talk about globalization Let’s talk about poverty Let’s talk about

Let’s talk Social Business!

"Let’s Talk!" Training Guide

Let’s talk about SOAP, baby. Let’s talk about UPnP

Let’s talk atoms!

Let’s talk about grain solutions Let’s talk One Source. … · Let’s talk SOLUTIONS GRAINabout MANUFACTURERS OF GRAIN STORAGE, HANDLING & CONDITIONING EQUIPMENT There is more

Let’s Talk Science!

Let’s talk about

Let’S Talk About…. Tanning

Let’s Talk About Sex…

Let’s Talk Plastic

Let’s talk Trade - Government of Jersey · Let’s talk Trade campaign on lines similar to the Let’s talk Brexit campaign of 2017. 6 Foreword Let’s talk Trade Foreword Over

Let’s Talk Health

Let’s Talk … Report

Let’S Talk About Her

Let’s Talk About Waterletstalkaboutwater.nl/LetsTalkAboutWaterDelft2018.pdf · 22 Landforms & Landscapes Let’s Talk About Water Welcome to the sixth edition of Let’s taLk about

Let’s Talk T echnology

Let’s Talk About Logic

Let’s Talk Gluten

Let’s Talk About Menopause

Let’s talk about love

Let’s talk about audits

Let’s Talk Trash !

Let’s talk product

Let’s talk about you

Let’s talk about SOAP, baby. Let’s talk about UPnP. CON 23/DEF CON 23 presentations/DEF… · Let’s talk about SOAP, baby. Let’s talk about UPnP. Ricky “HeadlessZeke”