using the shape of music to compute the similarity between symbolic musical pieces
DESCRIPTION
We describe a new approach for computing the similarity between symbolic musical pieces, based on the differences in shape between the interpolating curves defining the pieces. We outline several requirements for a symbolic musical similarity system, and then show how our model addresses them. Results obtained by an early prototype, tested with the MIREX 2005 Symbolic Music Similarity collections, show that this approach performs well. This new model opens a new and promising line for further research on the melodic similarity problem.TRANSCRIPT
![Page 1: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/1.jpg)
Using the Shape of Music to
Compute the Similarity between
Symbolic Musical Pieces
Julián Urbano, Juan Lloréns, Jorge Morato and Sonia Sánchez-Cuadrado http://julian-urbano.info
Twitter: @julian_urbano
CMMR 2010 · Málaga, Spain · June 24th
![Page 2: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/2.jpg)
Outline
• Introduction
• Melodic Similarity Requirements
• General Solutions to the Requirements
• A Model Based on Interpolation
• Implementation and Experimental Results
• Conclusions and Future Work
2
![Page 3: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/3.jpg)
Symbolic Melodic Similarity
• Given a musical piece (i.e. query), retrieve others deemed melodically similar to it (i.e. results)
• Traditional approaches? [Typke et al., 2005a]
▫ Geometry [Ukkonen et al., 2003][Typke et al., 2004]
▫ n-grams [Uitdenbogerd et al., 1999][Doraisamy et al., 2003]
▫ Alignment [Hanna et al., 2007]
• What do we do?
▫ Use local an alignment algorithm
▫ whose symbols are n-grams
▫ according to a geometric substitution function
3
![Page 4: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/4.jpg)
General Requirements
• Any Music Information Retrieval system should meet several requirements [Selfridge-Field, 1998][Byrd et al., 2002][Mongeau et al., 1990]
• Particularly focused on non-experts
• We just put together and thoroughly describe traditional and well-known requirements mostly related with transposition invariance
▫ Vertical requirements (i.e. pitch)
▫ Horizontal requirements (i.e. time)
4
![Page 5: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/5.jpg)
Vertical Requirements
• Query [simplified riff from Layla by Dereck and the Dominos]
• Octave Equivalence
5
![Page 6: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/6.jpg)
Vertical Requirements (II)
• Query
• Degree Equality
6
![Page 7: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/7.jpg)
Vertical Requirements (III)
• Query
• Note Equality
7
![Page 8: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/8.jpg)
Vertical Requirements (IV)
• Query
• Pitch Variation
8
![Page 9: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/9.jpg)
Vertical Requirements (V)
• Query
• Harmonic Similarity
9
![Page 10: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/10.jpg)
Vertical Requirements (and VI)
• Voice Separation
10
![Page 11: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/11.jpg)
Horizontal Requirements
• Query [simplified beginning from op.81 no.10 by S. Heller]
• Time Signature Equivalence
11
![Page 12: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/12.jpg)
Horizontal Requirements (II)
• Query
• Tempo Equivalence
12
![Page 13: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/13.jpg)
Horizontal Requirements (III)
• Query
• Duration Equality
13
![Page 14: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/14.jpg)
Horizontal Requirements (and IV)
• Query
• Duration Variation
14
![Page 15: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/15.jpg)
General Vertical Solutions
• Octave Equivalence ▫ Disregard octave number but consider relative
changes (G5 to C6 is not the same as G5 to C5).
• Degree Equality ▫ Use the degrees within the tonality
• Note Equality ▫ Use actual pitch values
• Some approaches use both, but key signature is not always available in SMF [Hanna et al., 2007]
• The accepted solution is to consider relative pitch differences between successive notes
15
![Page 16: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/16.jpg)
General Horizontal Solutions
• Time Signature Equivalence ▫ Just ignore it
• Tempo Equivalence ▫ Use actual note durations
• Duration Equality ▫ Use score durations
• Again, this information is not mandatory in SMF, and users with different expertise would prefer different approaches
• It is usual to just ignore time altogether, or use the duration ratio between successive notes
16
![Page 17: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/17.jpg)
A Model based on Interpolation
• Consider the time-pitch plane
• Arrange the notes as points in the plane, according to their pitch and duration
• With different voices, get new pitch-dimensions sharing the same time dimension
• Define the curve Ci(t) as the one interpolating the notes of the i-th voice (pitch-dimension)
17
![Page 18: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/18.jpg)
A Model based on Interpolation (II)
18
![Page 19: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/19.jpg)
A Model based on Interpolation (and III)
• The similarity of two pieces is thought of as their similarity in shape
• Most requirements are directly met
▫ Neither pitch nor time invariants change the shape of the curve
▫ Pitch and Duration Variations can be measured analytically
19
![Page 20: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/20.jpg)
Measure of Similarity
• Consider the curves as polynomials
▫ C(t)=antn+an-1tn-1+…+a1t+a0
• The first derivative measures how much the shape is changing at any time
• The shape dissimilarity between two curves (songs) can be measured as the area between their first derivatives
20
![Page 21: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/21.jpg)
It is Metric
• Non-negativity
▫ diff(C, D) ≥ 0
• Identity of indiscernibles
▫ diff(C, D) = 0 C = D
• Symmetry
▫ diff(C, D) = diff(D, C)
• Triangle inequality
▫ diff(C, E) ≤ diff(C, D) + diff(D, E)
• So we could use vantage objects [Bozkaya et al., 1999]
21
![Page 22: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/22.jpg)
Interpolation with Splines
• Easier to handle than Lagrange’s polynomials
• They avoid the Runge’s phenomenon [de Boor, 2001]
22
![Page 23: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/23.jpg)
Interpolation with Splines (II)
• Defined as piece-wise functions
• Very handy to measure the Pitch and Duration Variations
▫ Span durations can be normalized from 0 to 1
23
![Page 24: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/24.jpg)
Interpolation with Splines (and III)
• Defined as parametric functions
▫ One function per dimension
• Pitch and Time can be compared separately
• Voices can be isolated easily
▫ Using partial derivatives
• More weight can be given to pitch than to time
24
![Page 25: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/25.jpg)
First Implementation
• Dynamic programming has been widely used with textual representations of music
▫ Levenshtein distance
▫ Needleman-Wunsch global alignment
▫ Smith-Waterman local alignment [Smith et al., 1981]
Shown to be the most effective [Hanna et al., 2007, 2008]
• The symbols in the sequences are defined as n-grams of successive notes, according to the spans defined by the curve
• The substitution score between two n-grams is the area between their curves’ derivatives
25
![Page 26: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/26.jpg)
First Implementation (and II)
• We used degree 3 Uniform B-Splines [de Boor, 2003]
▫ Results in spans of 4 notes (n-gram length)
Noted be effective [Doraisamy et al., 2003]
• Pitch relative to the first note’s
▫ 74, 81, 72, 76
▫ 7, -2, 2 (actually 0, 7, -2, 2)
• Duration relative to the first note’s
▫ 240, 480, 240, 720
▫ 2, 1, 3 (actually 1, 2, 1, 3 or 1/7, 2/7, 1/7, 3/7)
26
![Page 27: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/27.jpg)
Results
• Tested with MIREX 2005 test collections
▫ Training and evaluation collections
▫ 11 queries per collection
▫ About 550 songs per collection
▫ Partially ordered lists with relevants [Typke et al., 2005b]
▫ Effectiveness measured with ADR [Typke et al., 2006]
27
![Page 28: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/28.jpg)
Results (II)
• Two alternatives tested
▫ Kpitch=1 and Ktime=0
▫ Kpitch=0.75 and Ktime=0.25
Chosen by others [Doraisamy et al., 2003][Hanna et al., 2007]
• We found the improvement of considering time completely incidental
28
Tuning Collection Avg. Min. Max.
Kpitch=1 , Ktime=0 Training 0.639 0.271 0.864
Kpitch=0.75 , Ktime=0.25 Training 0.643 0.312 0.864
Kpitch=1 , Ktime=0 Evaluation 0.709 0.314 0.911
Kpitch=0.75 , Ktime=0.25 Evaluation 0.710 0.314 0.911
![Page 29: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/29.jpg)
Results (and III)
• Compared with the official MIREX 2005 results
▫ We would have ranked first
▫ Best ADR scores for 5 of the 11 queries
bold for best per query, italics for best per system
* for significant difference at the 0.10 level, ** at the 0.05 level and *** at the 0.01 level
29
Query Splines GAM O US TWV L(P3) L(DP) FM
190.011.224-1.1.1 0.803 0.820 0.717 0.824 0.538 0.455 0.547 0.443
400.065.784-1.1.1 0.879 0.846 0.619 0.624 0.861 0.614 0.839 0.679
450.024.802-1.1.1 0.722 0.450 0.554 0.340 0.554 0.340 0.340 0.340
600.053.475-1.1.1 0.911 0.883 0.911 0.911 0.725 0.661 0.650 0.567
600.053.481-1.1.1 0.630 0.293 0.629 0.486 0.293 0.357 0.293 0.519
600.054.278-1.1.1 0.810 0.674 0.785 0.864 0.731 0.660 0.527 0.418
600.192.742-1.1.1 0.703 0.808 0.808 0.703 0.808 0.642 0.642 0.808
700.010.059-1.1.2 0.521 0.521 0.521 0.521 0.521 0.667 0.521 0.521
700.010.591-1.4.2 0.314 0.665 0.314 0.314 0.314 0.474 0.314 0.375
702.001.406-1.1.1 0.689 0.566 0.874 0.675 0.387 0.722 0.606 0.469
703.001.021-1.1.1 0.826 0.730 0.412 0.799 0.548 0.549 0.692 0.561
Average 0.710 0.660 0.650 0.642 0.571* 0.558*** 0.543** 0.518***
![Page 30: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/30.jpg)
Conclusions
• We presented a new geometric model to compute the similarity of symbolic pieces
▫ Opens a very promising line for further research
• It has a very intuitive interpretation, but not so intuitive implementation
• A very early prototype has shown to perform quite well with the MIREX 2005 test collections
▫ Would have ranked first
▫ Though not significantly better than the top 3
• The modeling of time is once again shown not to improve the overall effectiveness
30
![Page 31: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/31.jpg)
So Now What?
• We presented a very early work • We are currently improving it
▫ As of today we reach avg. ADR scores of over 0.82
• Other considerations ▫ Local alignment? Domain-dependant tuning? ▫ Uniform B-Splines? Cardinal? Hermite? ▫ n-grams of length 4? Split at inflection points? ▫ Area between derivatives? Between the curves? ▫ Shape as a nominal variable (concave, convex)? ▫ Harmony: all possible paths? Polyphony?
• We will see... ▫ Submitting 3 or 4 versions to MIREX 2010
31
![Page 32: Using the Shape of Music to Compute the similarity between Symbolic Musical Pieces](https://reader034.vdocuments.us/reader034/viewer/2022052602/559d39f21a28ab1a078b459f/html5/thumbnails/32.jpg)
And That’s It!
Picture by 姒儿喵喵
32