![Page 1: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/1.jpg)
Learning to Align Polyphonic Music. Slide 1
Learning to Align Polyphonic Music
Shai Shalev-Shwartz
Hebrew University, Jerusalem
Joint work with
Yoram Singer, Google Inc.
Joseph Keshet, Hebrew University
![Page 2: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/2.jpg)
Learning to Align Polyphonic Music. Slide 2
Motivation
Symbolic representation:
Acoustic representation:
Two ways for representing music
![Page 3: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/3.jpg)
Learning to Align Polyphonic Music. Slide 3
Symbolic Representation
time
pitch
- pitch
symbolic representation:
- start-time
![Page 4: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/4.jpg)
Learning to Align Polyphonic Music. Slide 4
Acoustic Representation
Feature Extraction
(e.g. Spectral Analysis)
acoustic representation:
acoustic signal:
Time
Fre
qu
en
cy
0 0.5 1 1.5 20
500
1000
1500
2000
2500
3000
3500
4000
![Page 5: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/5.jpg)
Learning to Align Polyphonic Music. Slide 5
Time
Fre
qu
en
cy
0 0.5 1 1.5 20
500
1000
1500
2000
2500
3000
3500
4000
The Alignment Problem Setting
time
pitch
actual start-time:
![Page 6: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/6.jpg)
Learning to Align Polyphonic Music. Slide 6
The Alignment Problem Setting
Goal: learn an alignment function
alignment function
actual start-times
acoustic representation
- pitch
symbolic representation
- start-times
![Page 7: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/7.jpg)
Learning to Align Polyphonic Music. Slide 7
Previous Work
• Dynamic Programming (rule based)• Dannenberg 1984• Soulez et al. 2003• Orio & Schwarz 2001
• Generative Approaches• Raphael 1999• Durey & Clements 2001• Shalev-Shwartz et al. 2002
![Page 8: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/8.jpg)
Learning to Align Polyphonic Music. Slide 8
Our Solution
Discriminative Learning Algorithm
Training Set
Alignment function
Discriminative Learning from examples
![Page 9: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/9.jpg)
Learning to Align Polyphonic Music. Slide 9
Why Discriminative Learning?
“When Solving a given problem, try to avoid a
more general problem as an intermediate step” (Vladimir Vapnik’s principle for solving problems using a
restricted amount of information)
Or, if you would like to visit Barcelona, buy a ticket !
Don’t waste so much time on writing a paper for ISMIR 2004 …
![Page 10: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/10.jpg)
Learning to Align Polyphonic Music. Slide 10
Outline of Solution
1. Define a quantitative assessment of alignments
2. Define a hypotheses class - what is the form of our alignment functions :
a. Map all possible alignments into vectors in an abstract vector-space
b. Find a projection in the vector-space which ranks alignments according to their quality
3. Suggest a learning algorithm
![Page 11: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/11.jpg)
Learning to Align Polyphonic Music. Slide 11
Assessing alignments
e.g.
![Page 12: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/12.jpg)
Learning to Align Polyphonic Music. Slide 12
Feature Functions for Alignment
feature functionfor alignment
Assessing the quality of a suggested alignment
acoustic and symbolic representation
suggested alignment
(actual start-times)
e.g.
e.g.
![Page 13: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/13.jpg)
Learning to Align Polyphonic Music. Slide 13
Feature Functions for Alignment
correct alignment
slightly incorrect alignment
grossly incorrect alignment
Mapping all possible alignments into a vector space
![Page 14: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/14.jpg)
Learning to Align Polyphonic Music. Slide 14
Main Solution Principle
grossly incorrect alignment
correct alignment
slightly incorrect alignment
Find a linear projection that ranks alignments according to their quality
![Page 15: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/15.jpg)
Learning to Align Polyphonic Music. Slide 15
slightly incorrect alignment
Main Solution Principle (cont.)An example of projection with low confidence
correct alignment
grossly incorrect alignment
![Page 16: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/16.jpg)
Learning to Align Polyphonic Music. Slide 16
slightly incorrect alignment
Main Solution Principle (cont.)An example of incorrect projection
correct alignment
grossly incorrect alignment
![Page 17: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/17.jpg)
Learning to Align Polyphonic Music. Slide 17
Hypotheses class
The form of our alignment functions:
predict the alignment which attains the highest projection
defines the direction of projection
![Page 18: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/18.jpg)
Learning to Align Polyphonic Music. Slide 18
Learning algorithm
Optimization Problem:
• Given a training set:
• Find:
• a projection and
• a maximal confidence scalar
such that the data is ranked correctly:
![Page 19: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/19.jpg)
Learning to Align Polyphonic Music. Slide 19
Algorithmic aspects• Iterative algorithm:
• Works on one alignment example at a time• The algorithm works in polynomial time although the
number of constraints is exponentially large• Simple to implement
• Convergence:• Converges to a high confidence solution• #iterations depends on the best attainable confidence
• Generalization:• The gap between test and train error decreases with the
#examples. The gap is bounded above by
![Page 20: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/20.jpg)
Learning to Align Polyphonic Music. Slide 20
Experimental Results• Task: alignment of polyphonic piano music• Dataset: 12 musical pieces where sound and MIDI
were both recorded + other performances of the same pieces in MIDI format
• Features: see in the paper• Algorithms:
• Discriminative method
• Generative method: Generalized Hidden Markov Model (GHMM)
• Using the same features as in the discriminative method
• Using different number of Gaussians (1,3,5,7)
![Page 21: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/21.jpg)
Learning to Align Polyphonic Music. Slide 21
Experimental Results (Cont.)
Our discriminative method outperforms GHMM
GHMM-1
GHMM-3
GHMM-5
GHMM-7
Discrim
inativ
e
Loss (ms)
70
80
60
50
40
30
20
10
![Page 22: Learning to Align Polyphonic Music. Slide 1 Learning to Align Polyphonic Music Shai Shalev-Shwartz Hebrew University, Jerusalem Joint work with Yoram](https://reader036.vdocuments.us/reader036/viewer/2022062300/56649d4c5503460f94a2a28e/html5/thumbnails/22.jpg)
Learning to Align Polyphonic Music. Slide 22
The End