The Effectiveness Study of Music Information Retrieval

Arbee L.P. ChenNational Tsing Hua University

2002 ACM International CIKM Conference

Outline

• Motivation• The Ultima Project

– The 1D-List approach

– The APS approach

• The Effectiveness Study– Estimating all relevant references

• Conclusion

Motivation

• Various approaches were proposed to provide efficient and effective content-based retrieval of music objects– Music representations

• pitch, rhythm, contour, chord

– Index structures• tree-based index, list-based index, n-gram index

– Query processing methods• exact match, partial match, approximate match

Motivation (Cont.)

• A platform is needed for the evaluation of various music information retrieval methods– Efficiency

• response time

– Effectiveness• recall-precision curve

• The Ultima project builds such a platform– Same data set and query set– Also serves as a testbed whenever new approaches

are proposed

The Ultima Project (Cont.)

Architecture• Mediator

• Query processing module

• Report module

• Summarization module

• Query generation module

• Data store

Data Store (MS Access)

SMF

Med

iato

r

Query Processing Module

Table

to the InternetSummarization Module

1D-List APS APM

Report Module

Query Generation Module

The Ultima Project (Cont.)

• Two approaches have been compared – 1D-List

– APS

Approach Representation Index structure

1D-List melody string list-based

APS sequence of music segments suffix tree-based

The 1D-List Approach

• The 1D-List approach– Music objects are coded as melody strings

• “so-mi-mi-fa-re-re-do-re-mi-fa-so-so-so”

– Melody strings are organized as linked lists– Both exact and approximate matching can be handl

ed

• Exact link, insertion link, dropout link, transposition link

The 1D-List Approach (Cont.)

1:7 1:11:41:21:5

2:1 1:31:6

2:9 2:71:91:8

2:82:22:5

2:32:6

2:4

2:10

2:11 2:12

1:10 1:11

1:12

1:13

do re mi fa so la si

1:7 1:21:5

2:1 1:31:6

2:9 1:91:8

2:22:5

2:62:10

2:11 2:12

do re mi

start end 1:7 1:21:5

2:1 1:31:6

2:9 1:91:8

2:22:5

2:62:10

2:11 2:12

do re mi

start end

(a) (b) (c)

The APS Approach

• The APS approach– Music objects are coded as sequences of

music segments• four segment types to model the music contour• pitch and duration are considered

– Index structures• one-dimensional and two-dimensional augmented suffix

tree

– Both exact and approximate matching can be handled

The APS Approach (Cont.)

• Representation

note number

beat

60

62

65

64

67

(B, 3, -3)

(A, 1, +1)

(D, 3, -3)

(B, 1, -2)

(C, 1, +2)

(C, 1, +2)

(C, 1, +1)

type A

type B

type C

type D

The APS Approach (Cont.)

1 4

AB C

B C

C $

$

2 5

3

(a)

root

A

C

A

(b)

root

A<1,1>

C<7,8> C<1,3>

A<7,8> A<3,4>

N1

N2

The suffix tree of the string S=“ABCAB”

(a) An example of suffix tree(b) A 1-D augmented suffix tree

1

A

$

A

B

$B2

A

B

$3

The APS Approach (Cont.)

• Similarity measure– Given a query sequence Q = (i1, j1, k1) (i2, j2, k2) ... (in, jn, kn),

and a candidate sequence from the database C = (i1, x1, y1) (i2, x2, y2) ... (in, xn, yn).

iip

n

i

ppitch

n

iiid

dduration

beatdurationpitchduration

pitchpitchdurationduration

ykMaxPitchifMaxPitch

if

nsimdis

xjnMaxDuratioifnMaxDuratio

if

nsimdis

wwww

simdiswsimdiswCQSIMDIS

,MIN and ,1

_)3(

,MIN and , 1

_)2(

1,0 and 1)1(where

__,_

1

2

1

The Effectiveness Study

• Traditional measures of effectiveness are precision and recall

• However, the number of relevant references are usually unknown– it is unrealistic for the user to make relevant

judgments to all music objects in the database

retrievedarethatreferencesnumber of

relevantarethatreferencesretrievednumber ofprecision

referencesrelevantnumber of

relevantarethatreferencesretrievednumber ofrecall

The Effectiveness Study (Cont.)

• How to estimate the number of relevant references NR?– ASx is the set of relevant objects from the top x ranked results

– RSx is the set of the top x ranked results retrieved by an approach

– Assumption 1: , ,• the number of the retrieved results is a function of the number of retrieve

d relevant objects

• Assumption 2: , where B is a positive integer

• Based on the two assumptions, NR can be derives as follows:

xx ASfRS DBx 1for 1 and 1 AS

1 BBRS xASx

)1)(BRS(log

))1B(DB(log AS

xB

Bx

NR

The Effectiveness Study (Cont.)

rank relevance recall precision

1 Y 0.1 1

2 Y 0.2 1

3 Y 0.3 1

4 0.3 0.75

5 Y 0.4 0.80

6 0.4 0.67

7 Y 0.5 0.71

8 0.5 0.63

9 Y 0.6 0.67

10 Y 0.7 0.70

11 0.7 0.64

12 0.7 0.58

13 Y 0.8 0.62

14 0.8 0.57

15 0.8 0.53

16 Y 0.9 0.56

17 0.9 0.53

18 0.9 0.50

19 Y 1 0.53

20 1 0.50

Rank relevance recall precision

1 Y 0.1 1

2 Y 0.2 1

3 Y 0.3 1

4 0.3 0.75

5 Y 0.4 0.80

6 0.4 0.67

7 Y 0.5 0.71

8 0.5 0.63

|RSx| = 8, |ASx| = 5, |DB| = 20

The Effectiveness Study (Cont.)

MethodFactor

APS

1D-List1-D AST (duration

)

1-D AST (pitch

)2-D AST

Number of music objects for generating queries

10 10

Is the query sample a refrain or an incipit?

refrain/incipit refrain/incipit

Length of query sample, denoted L

6/10 (segment) 8/12 (note)

Number of query samples per music object

4 4

Threshold setting of approximation for a query

sample

th_d = 0, 0.5, 1.0th_p = 0, 0.5, 1.0

K=0, 4, 7 (for L=8)K=0, 6, 11 (for L=12)

Total number of posing queries 120 120

Experiment setup

The Effectiveness Study (Cont.)

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Recall

Pre

cisi

on

1D-AST_D0

1D-AST_P0

2D-AST_PD0

1D-List_K0

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Recall

Pre

cisi

on1D-AST_D0.51D-AST_P0.5

2D-AST_PD0.51D-List_K4,6

• Experiment results

The Effectiveness Study (Cont.)

• Experiment results

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Recall

Pre

cisi

on

1D-AST_D_R

1D-AST_P_R

2D-AST_R

1D-List_R

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

Recall

Pre

cisi

on

1D-AST_D_In

1D-AST_P_In

2D-AST_In

1D-List_In

The Effectiveness Study (Cont.)

• Experiment results– 1D-list achieves a high precision in the limited

range of recall, while a moderate precision for the APS family can be obtained

– Comparing the APS family, the precision in a descending order is: 1D-AST (pitch), 2D-AST, and 1D-AST (duration)

– In average, the effectiveness of “incipit” queries is better than “refrain” queries

Conclusion

• The Ultima project builds a platform for evaluating the performance of various approaches of music information retrieval

• A new measure for estimating the number of relevant references is proposed

• Future work– Design and implement the summarization module

as well as the query generation module– Extend the project for evaluating polyphonic musi

c retrieval methods