the music information retrieval evaluation …textual lyrics, libretti bibliographic title,...
TRANSCRIPT
![Page 1: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/1.jpg)
The Music Information Retrieval
Evaluation eXchange (MIREX):
An Introductory Overviewhttp://music-ir.org/mirexwiki
J. Stephen Downie
Graduate School of Library and Information Science
University of Illinois at Urbana-Champaign
![Page 2: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/2.jpg)
My Life: A Bunch of Acronyms
IMIRSEL: International Music Information Retrieval System Evaluation Laboratory
HUMIRS: Human Use of Music Information Retrieval Systems
M2K: Music-to-Knowledge
MIREX: Music Information Retrieval Evaluation eXchange
NEMA: Networked Environment for Music Analysis
![Page 3: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/3.jpg)
What is MIR?
Born ca. 1960‟s in IR research
Major recent growth precipitated by advent
of networked digital music collections
Informed by multiple disciplines and
literatures
ISMIR started in 2000
![Page 4: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/4.jpg)
Defining Music Information
Retrieval? Music Information Retrieval (MIR) is the process of
searching for, and finding, music objects, or parts of music objects, via a query framed musically and/or in musical terms
Music Objects: Scores, Parts, Recordings (WAV, MP3, etc.), etc.
Musically framed query: Singing, Humming, Keyboard, Notation-based, MIDI file, Sound file, etc.
Musical terms: Genre, Style, Tempo, etc.
![Page 5: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/5.jpg)
What makes MIR so tricky?
Multifaceted
Multimodal
Multirepresentational
Multiexperiential
Multicultural
Given the inherent
complexities of music
information, only a
multidisciplinary research
approach could possibly
lead to the development
of a robust MIR system.
Music information is:
![Page 6: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/6.jpg)
Multifaceted (Pt. 1)
Pitch Pitch is “the perceived quality of a sound that is chiefly a function
of its fundamental frequency in --the number of oscillations per second ” (Randel 1986)
Also, the distance between pitches: intervals
Temporal Meter, duration, rhythm, tempo, etc.
Harmonic When two or more pitches occur at the same time, a
simultaneity, or harmony, occurs. Also known as polyphony, while absence of polyphony is called monophony.
![Page 7: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/7.jpg)
Multifaceted (Pt. 2)
Timbre Tone-colour
Flute v. Kazoo v. Violin v. Bass Drum
Editorial Fingerings, Ornamentation, Dynamic instructions (e.g., ppp, p,
...f, fff), Slurs, Articulations, Stacatti, Bowings, etc.
Textual Lyrics, Libretti
Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc.
![Page 8: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/8.jpg)
Multirepresentational (Pt. 1)
Solfege do, re, mi, fa, so, etc.
Pitch names A, B, C, D, E, F#, Ab, etc.
Chord Names Cmaj, Dmin, Am7, etc.
Scale Degree I, II, III, IV, V, VI, VII
Interval
+1, 0, -3, -8, +6, etc.
![Page 9: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/9.jpg)
Multirepresentational (Pt. 2)
MIDI Events:
• Graphic Score:
![Page 10: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/10.jpg)
Music representation is VERY
heterogeneous!
![Page 11: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/11.jpg)
Multimodal
Music as thought Tune running through head
Music as auditory events Sound waves hitting eardrums
Sound in electromechanical formats
WAV, MP3, AU, CD, LPs and Tapes
Music as graphic language Symbolic representations
Scores
MIDI files and other discrete encodings
etc.
![Page 12: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/12.jpg)
Multiexperiential (Pt. 1)
Music as object of study Perform, Analyze
Music as foreground Concert going, Deliberate audition
Music as backgroundMovie scores, Shopping malls, Housecleaning
Music social signifier Protest, Peace, Group songs, “Brow-ness”: High,
Middle, Low, etc.
![Page 13: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/13.jpg)
Multiexperiential (Pt. 2)
Music as aide memoire Soundtrack recordings, Camp songs, War songs, Ballades, etc.
Music as tradition Hymns, Folksongs, Nursery songs, etc.
Music as drug Stimulation
Stay awake, Frenzied dancing, etc.
Relaxation
Stress relieve, Forgetfulness, Sleep, etc.
Seduction
![Page 14: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/14.jpg)
Multicultural
Different notation/representational schemes E.g., Modern art music
Lack of notation/representational schemes E.g., Jazz (improvised), Aural and oral traditions
Different scales and modes E.g., Quartertone music, Gamelan music, Eastern music
Different grammars of musical affect and gesture E.g., Inuit throat music, Indian ragas
Different accessibility to recordings and recording technologies
![Page 15: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/15.jpg)
The “Brass Ring” MIR System
Multimodal, Multirepresentational, Multicultural
Has a meaningful abstracting/thumbnail feature for determinations and browsing
Employs an intelligent, user-definable, experientially-grounded, relevance-feedback/classification mechanism: User inputs “song” into system and can tell system which
aspect(s) of the music (e.g., throbbing bass, sweet violins, tempo, rap-like vocals, etc.) is/are the key factor(s) that should be the basis for gathering similar items
Would overcome user input errors
![Page 16: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/16.jpg)
MIREX Model
Based upon the TREC approach:
Standardized queries/tasks
Standardized collections
Standardized evaluations of results
Not like TREC with regard to distributing data collections to participants
Music copyright issues, ground-truth issues, overfitting issues
![Page 17: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/17.jpg)
Audio Description Contest
Barcelona 2004
Music Technology Group (Dr. Serra‟s Lab)
Contest Categories
Genre Classification/Artist Identification
Melody Extraction
Tempo Induction
Rhythm Classification
MIREX built upon the lessons learned by ADC
![Page 18: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/18.jpg)
IMIRSEL: First Principles
1. Security for the music materials
2. Accessibility for international, domestic
and internal researchers
3. Sufficient computing and storage
infrastructure for the computationally-
and data-intensive MIR/MDL techniques
examined
![Page 19: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/19.jpg)
Virtual Research Labs Model
Supercomputer
Virtual Research
Lab-2 (VRL-2)
Supercomputer
Virtual Research
Lab-1 (VRL-1)
Supercomputer
Virtual Research
Lab-n (VRL-n)
Supercomputer
Virtual Research
Lab-4 (VRL-4)
Supercomputer
Virtual Research
Lab-3 (VRL-3)
TerascaleDatastore-4
TerascaleDatastore-3
TerascaleDatastore-2
TerascaleDatastore-1
TerascaleDatastore-n
Rights-Restricted Terascale MIR Multi-Modal Test Collections (Audio, Symbolic, Graphic, Metadata, etc.)
NCSA Music Data Secure ZoneSuper-Bandwith I/O Channel
Real-World Research
Lab-1 (London?)
Real-World Research
Lab-2 (Barcelona?)
Real-World Research
Lab-n (?)
Real-World Research
Lab-3 (Tokyo?)
Real-World Research
Lab-4 (Urbana?)
Command/Control/Derived Data
traffic via Internet
Legend:
Other
Rights-Restricted Music
Collections / Services
Other
Open-Source Music
Collections / Services
Connection to International
MIR Grid
Proposed International
Music Information Retrieval Grid
![Page 20: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/20.jpg)
Music-to-Knowledge (M2K)
Goal: Have both a toolset and the
evaluation environment available to
researchers
Visual data flow programming built upon
NCSA‟s Data-to-Knowledge (D2K)
machine learning environment
Java-based thus easily portable
Supports distributed computing
![Page 21: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/21.jpg)
M2K: Main Goals
Promote collaboration and sharing through
a common, modular toolset
A „black box‟ approach to provide
commonly needed algorithms for fast
prototyping
Alleviate the „reinventing the wheel‟
problem
![Page 22: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/22.jpg)
How M2K/D2K Works
Signal processing and machine learning code is written into modules
Modules are „wired‟ together to produce more complicated programs called itineraries
Itineraries can then be run or used themselves as modules allowing nesting of programs
Individual modules and nested itineraries can be assigned to be parallelized across all machines in a network, or to individual machines in a network
![Page 23: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/23.jpg)
A Picture is Worth 1000 Words:
Music Classifier Example
![Page 24: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/24.jpg)
Music Classifier Example:
Feature Extraction Nested Itinerary
![Page 25: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/25.jpg)
Editing Parameters and
Component Documentation
![Page 26: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/26.jpg)
MIREX Overview
Began in 2005
Tasks defined by community debate
Data sets collected and/or donated
Participants submit code to IMIRSEL
Code rarely works first try
Huge labour consumption getting programmes
to work
Meet at ISMIR to discuss results
![Page 27: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/27.jpg)
Audio Description Contest
Barcelona 2004
Music Technology Group (Dr. Serra‟s Lab)
Contest Categories
Genre Classification/Artist Identification
Melody Extraction
Tempo Induction
Rhythm Classification
MIREX built upon the lessons learned by ADC
![Page 28: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/28.jpg)
2005 2006 2007 2008
Number of Tasks
(includes Sub-tasks)10 13 12 18
Number of Runs 86 92 122 168
MIREX Summary Data
![Page 29: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/29.jpg)
TASK 05 06 07 08Audio Artist Identification 7 7 11
Audio Beat Tracking 5
Audio Chord Detection 15*
Audio Classical Composer ID 7 11
Audio Cover Song Identification 8 8 8
Audio Drum Detection 8
Audio Genre Classification 15 7 26*
Audio Key Finding 7
Audio Melody Extraction 10 10 * 21*
Audio Mood Classification 9 13
Audio Music Similarity 6 12
Audio Onset Detection 9 13 17
Audio Tag Classification 11
Audio Tempo Extraction 13 7
Multiple F0 Estimation 16 15
Multiple F0 Note Detection 11 13
Query-by-Singing/Humming 23 * 20 * 16*
Query-by-Tapping 5
Score Following 2 3
Symbolic Genre Classification 5
Symbolic Key Finding 5
Symbolic Melodic Similarity 7 18 ** 8
![Page 30: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/30.jpg)
Runtime Extremes!
Audio Melody Extraction
Fastest : 56 Seconds
Slowest: 5 Days
![Page 31: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/31.jpg)
Some Innovation Highlights
Some New Tasks
Audio Cover Song
Audio and Symbolic Similarity
Mood Classification
New Evaluations
Multiple parameters in Onset Detection
Evalutron 6000: Human similarity judgments
Friedman and Tukey‟s HSD tests
![Page 32: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/32.jpg)
Some 2008 Highlights
Some New Tasks
Audio Chord Detection
Audio Tag Classification
GenreLatin Sub-task
Query-by-Tapping
New Melody Extraction 2008 Set
New Evaluations
Expanded Friedman and Tukey‟s HSD tests
![Page 33: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/33.jpg)
Onset Detection
![Page 35: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/35.jpg)
Evalutron 6000
Audio Similarity Symbolic Similarity
# Graders 24 21
# Graders per Q/C pair 3 3
# Queries per grader 7-8 15
Size of Candidate lists Max 30 15
# Of Q/C pairs evaluated
per graderMax 240 225
# Of queries 60 17
![Page 36: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/36.jpg)
Evalutron 6000 Data
SMS AMS
No. of events logged
No. of submitted algorithms
Total no. of queries
Total no. of query-candidate pairs
No. of graders
No. of queries per grader
Avg. size of candidate lists
Avg. no. of evaluations per grader
23,491
8
17
905
21
15
15
225
46,254
6
60
1,629
24
7-8
27
205
![Page 37: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/37.jpg)
Scoring Distributions
![Page 38: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/38.jpg)
Differences in Evaluator EffortA
vg.
# o
f a
ud
itio
ns p
er
ca
nd
idate
![Page 39: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/39.jpg)
Friedman Tests
Friedman‟s ANOVA Table
Source SS df MS Chi-Sq Prob>Chi-Sq
Columns 84.733 5 16.947 24.291 0.000
Error 961.767 295 3.260
Total 1046.50 359
Audio Music Similarity and Retrieval
![Page 40: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/40.jpg)
Friedman‟s Test:
Tukey‟s HSD Multiple Comparisons
TeamID TeamID Lowerbound Mean Upperbound Significance
EP TP -0.963 0.008 0.980 FALSE
EP VS -0.755 0.217 1.188 FALSE
EP LR -0.630 0.342 1.313 FALSE
EP KWT -0.030 0.942 1.913 FALSE
EP KWL 0.320 1.292 2.263 TRUE
TP VS -0.763 0.208 1.180 FALSE
TP LR -0.638 0.333 1.305 FALSE
TP KWT -0.038 0.933 1.905 FALSE
TP KWL 0.312 1.283 2.255 TRUE
VS LR -0.847 0.125 1.097 FALSE
VS KWT -0.247 0.725 1.697 FALSE
VS KWL 0.103 1.075 2.047 TRUE
LR KWT -0.372 0.600 1.572 FALSE
LR KWL -0.022 0.950 1.922 FALSE
KWT KWL -0.622 0.350 1.322 FALSE
![Page 41: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/41.jpg)
HSD Comparisons of Top
SubmissionsComparison Task
Rank Rank ACS06 AMS06 QBSH06 ACS07 AMS07 QBSH07 SM07
1 2 TRUE FALSE FALSE FALSE FALSE FALSE FALSE
1 3 TRUE FALSE FALSE TRUE FALSE FALSE FALSE
1 4 TRUE FALSE FALSE TRUE FALSE TRUE FALSE
2 3 FALSE FALSE FALSE FALSE FALSE FALSE FALSE
2 4 FALSE FALSE FALSE FALSE FALSE TRUE FALSE
3 4 FALSE FALSE FALSE FALSE FALSE FALSE FALSE
![Page 42: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/42.jpg)
Enter NEMA
Originally entitled: “Pan-Galactic-Distributed-Music-Analysis-Tools-Project-With-No-Clever-Name”
Tie-ins with Software Environment for the Advancement of Scholarly Research (SEASR)
UIUC, McGill (CA), Goldsmiths (UK), Queen Mary (UK), Southampton (UK), Waikato (NZ)
1 January 2008 to 31 December 2010 Funded 11 December 2007 (Yippee!)
![Page 43: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/43.jpg)
Example Consideration
Music classification (artist, genre, etc) is often broken down into a feature extractionfollowed by a machine learning stage
Some researchers focus only on one stage or the other
Difficult to evaluate the success of approaches in this case
Ideally, would evaluate all feature extractors against all classifiers
![Page 44: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/44.jpg)
Integrating Other Tools
Must also provide a means of support for all the
other toolsets people use
MATLAB, Marsyas, Weka, Clam, ACE, and on and
on
External integration modules allow for non-M2K
or JAVA-based programs to be used
E.g. C/C++ compiled binaries, MATLAB, etc
External processes called through the Java
runtime environment
![Page 45: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/45.jpg)
An External Classification
Algorithm
![Page 46: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/46.jpg)
NEMA Vision
In the new NEMA reality, for example, it should become common place for researchers at Lab A to easily build a virtual collection from Library B and Lab C, acquire the necessary ground-truth from Lab D, incorporate a feature extractor from Lab E, amalgamate the extracted features with those provided by Lab F, build a set of models based on pair of classifiers from Labs G and Hand then validate the results against another virtual collection taken from Lab I and Library J. Once completed, the results and newly created features sets would be, in turn, made available for others to build upon.
![Page 47: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/47.jpg)
Researcher 1 Researcher 2 Researcher 3
MIREX DIY
OMRAS2 TWM
jMIR
myExperiment
Grid-based feature extraction tools
(M2K, jAudio, OMRAS@Home, etc.)
Music database 1 Music database 2 Music database 3
Grid-based feature extraction tools
(M2K, jAudio, OMRAS@Home, etc.)
Grid-based feature extraction tools
(M2K, jAudio, OMRAS@Home, etc.)
Web service calls (Only features returned)
(Data passed internally)
Greenstone
Maestro
Music Ontology
NEMA‟S SEASR Framework
NEMA Portal
OMEN
Results Aggregators
Classification Modules
(ACE, M2K, OMRAS2)
Data Cleaning Tools
(MusicMetadataManager)
Data Exchange Tools(ACE XML, M2K, RDF)
High Level Services Low Level Services Discovery &
Sharing Services
Web
Front EndsWikis Code
Repositories
Archives
![Page 48: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/48.jpg)
The Amazing……
![Page 49: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/49.jpg)
![Page 50: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/50.jpg)
![Page 51: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/51.jpg)
![Page 52: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/52.jpg)
![Page 53: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/53.jpg)
![Page 54: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/54.jpg)
![Page 55: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/55.jpg)
![Page 56: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/56.jpg)
![Page 57: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/57.jpg)
![Page 58: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/58.jpg)
![Page 59: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/59.jpg)
Something To Read!
Downie, J. Stephen (2008). The Music
Information Retrieval Evaluation Exchange
(2005-2007): A window into music
information retrieval research. Acoustical
Science and Technology 29 (4): 247-255.
Available at:
http://dx.doi.org/10.1250/ast.29.247
![Page 60: The Music Information Retrieval Evaluation …Textual Lyrics, Libretti Bibliographic Title, Catalogue Num., Composer, Publisher, Lyricist, etc ... A Picture is Worth 1000 Words: Music](https://reader034.vdocuments.us/reader034/viewer/2022042416/5f30daf4fd22363f65690d92/html5/thumbnails/60.jpg)
Acknowledgements
Special Thanks to:
The Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign
The Andrew W. Mellon Foundation
The National Science Foundation (Grant No. NSF IIS-0327371)
Prof. Frank Tompa and all our University of Waterloo Colleagues!