Upload File

more like this: machine learning approaches to music similarity

  • Home
  • Technology
  • More Like This: Machine Learning Approaches to Music similarity
72
More Like This: Machine Learning Approaches to Music Similarity Brian McFee Computer Science & Engineering University of California, San Diego

Upload: brian-mcfee

Post on 05-Dec-2014

915 views

Category:

Technology


1 download

Report

DESCRIPTION

The slides from my dissertation talk. Thesis available at http://cseweb.ucsd.edu/~bmcfee/papers/bmcfee_dissertation.pdf

TRANSCRIPT

Page 1: More Like This: Machine Learning Approaches to Music similarity

More Like This:Machine Learning Approaches

to Music Similarity

Brian McFee

Computer Science & EngineeringUniversity of California, San Diego

Page 2: More Like This: Machine Learning Approaches to Music similarity

Music discovery in days of yore...

Page 3: More Like This: Machine Learning Approaches to Music similarity

Music discovery 2.0: the present

f

• ~20 million songs available

• Discovery is still largely human-powered

Page 4: More Like This: Machine Learning Approaches to Music similarity

A Google for music?

Page 5: More Like This: Machine Learning Approaches to Music similarity

A Google for music?

• Standard text search can work with meta-data• Can we predict meta-data from audio? ⁃ [Turnbull, 2008], [Barrington, 2011]

Page 6: More Like This: Machine Learning Approaches to Music similarity

Query by example

• Natural, user-friendly alternative to text search

Page 7: More Like This: Machine Learning Approaches to Music similarity

Query by example

• Natural, user-friendly alternative to text search

Page 8: More Like This: Machine Learning Approaches to Music similarity

Query by example

• Natural, user-friendly alternative to text search

Page 9: More Like This: Machine Learning Approaches to Music similarity

This talk

• Learning algorithms for QBE, geared toward music discovery

• We'll look at two consumption models:

• Evaluation derived from user behavior

Passive listening(playlist generation)

Active browsing(search & ranking)

Page 10: More Like This: Machine Learning Approaches to Music similarity

Learning similarity

Page 11: More Like This: Machine Learning Approaches to Music similarity

Defining similarity: semantics?

Song similarity=

tag similarity?

Page 12: More Like This: Machine Learning Approaches to Music similarity

Defining similarity: semantics?

• Drawbacks: - Choosing, weighting vocabulary is surprisingly difficult - Hard to maintain quality at scale

Page 13: More Like This: Machine Learning Approaches to Music similarity

Defining similarity: human judgements?

• Which is more similar?[M. & Lanckriet, 2009, 2011]

Page 14: More Like This: Machine Learning Approaches to Music similarity

Defining similarity: human judgements?

• Which is more similar?

• Drawbacks: ambiguity, subjectivity, scale

[M. & Lanckriet, 2009, 2011]

Page 15: More Like This: Machine Learning Approaches to Music similarity

Collaborative filter similarity

• Collect listening histories for (lots of!) users

• Song similarity = portion of users in common

Page 16: More Like This: Machine Learning Approaches to Music similarity

Collaborative filter similarity

• Collaborative filters perform well... - ... for tagging [Kim, Tomasik, & Turnbull, 2009] - ... and playlisting [Barrington, Oda, & Lanckriet, 2009] - ... and recommendation (Yahoo, Last.fm, iTunes...)

• Implicit feedback requires no additional effort from users

• ... but fails on unpopular items: the cold start problem!

Page 17: More Like This: Machine Learning Approaches to Music similarity

Learning from a collaborative filter[M., Barrington, & Lanckriet, 2010, 2012]

1.

2.

3.

Page 18: More Like This: Machine Learning Approaches to Music similarity

Learning from a collaborative filter[M., Barrington, & Lanckriet, 2010, 2012]

1.

2.

3.

Page 19: More Like This: Machine Learning Approaches to Music similarity

Learning from a collaborative filter[M., Barrington, & Lanckriet, 2010, 2012]

1.

2.

3.

Page 20: More Like This: Machine Learning Approaches to Music similarity

Rankings in audio space

Rankings in CF space

=

Metric learning to rank

• The goal:

Page 21: More Like This: Machine Learning Approaches to Music similarity

Ranking by (learned) distance

Targetrankings

=

Metric learning to rank

• The goal:[M. & Lanckriet, 2010]

Page 22: More Like This: Machine Learning Approaches to Music similarity

Ranking by (learned) distance

Targetrankings

=

Metric learning to rank

• The goal:

• Optimize a linear transformation for ranking

[M. & Lanckriet, 2010]

Page 23: More Like This: Machine Learning Approaches to Music similarity

Structure prediction: nearest neighbors

• Setup: database , rankings

• PSD matrix transforms features

• Order by distance from :

Page 24: More Like This: Machine Learning Approaches to Music similarity

Structure prediction: nearest neighbors

• Setup: database , rankings

• PSD matrix transforms features

• Order by distance from :

• encodes each (query, ranking) pair

Page 25: More Like This: Machine Learning Approaches to Music similarity

Metric learning to rank (MLR)

Score fortarget ranking

Score for anyother ranking

Predictionerror

+>

• Supported losses Δ: AUC, KNN, MAP, MRR, NDCG, Prec@k

Page 26: More Like This: Machine Learning Approaches to Music similarity

MLR solver• Cutting-plane algorithm based on 1-slack Structural SVM [Joachims, et al. 2009]

• Repeat until convergence:

Constraintgeneration

(DP)

Semi-definiteprogramming

Page 27: More Like This: Machine Learning Approaches to Music similarity

MLR solver• Cutting-plane algorithm based on 1-slack Structural SVM [Joachims, et al. 2009]

• Repeat until convergence:

Constraintgeneration

(DP)

Semi-definiteprogramming

Sequence of QPs

Page 28: More Like This: Machine Learning Approaches to Music similarity

MLR solver• Cutting-plane algorithm based on 1-slack Structural SVM [Joachims, et al. 2009]

• Repeat until convergence:

• Multiple kernel extensions: [Galleguillos, M., Belongie, & Lanckriet 2011]

Constraintgeneration

(DP)

Semi-definiteprogramming

Sequence of QPs

Page 29: More Like This: Machine Learning Approaches to Music similarity

Audio pipeline

Audio signal

Page 30: More Like This: Machine Learning Approaches to Music similarity

Audio pipeline

Audio signal 1. Feature extraction

Bag of ΔMFCCs

Page 31: More Like This: Machine Learning Approaches to Music similarity

Audio pipeline

Audio signal 1. Feature extraction

Bag of ΔMFCCs

Codeword hist.

2. Vector quantization

Page 32: More Like This: Machine Learning Approaches to Music similarity

Audio pipeline

Audio signal

PPK

1. Feature extraction

Bag of ΔMFCCs

Codeword hist.

2. Vector quantization

3. Probability product kernel

Page 33: More Like This: Machine Learning Approaches to Music similarity

Audio pipeline

Audio signal

PPK

CF similarity

MLR

Supervision

Features

Page 34: More Like This: Machine Learning Approaches to Music similarity

Evaluation: CAL10K

• Last.fm collaborative filter - 360K users, 186K artists

• CAL10K songs - 5.4K songs, 2K artists (after CF matching)

[Celma, 2008]

[Tingle, Turnbull, & Kim, 2010]

Page 35: More Like This: Machine Learning Approaches to Music similarity

Evaluation: CAL10K

• Last.fm collaborative filter - 360K users, 186K artists

• CAL10K songs - 5.4K songs, 2K artists (after CF matching)

• Evaluation: - Split artists into train/val/test - Target rankings: top-10 most similar train artists

[Celma, 2008]

[Tingle, Turnbull, & Kim, 2010]

Page 36: More Like This: Machine Learning Approaches to Music similarity

Evaluation: comparison

• Gaussian mixture models + KL divergence - 8 component, diagonal covariance GMM per song

• Auto-tags: predict 149 semantic tags from audio [Turnbull, 2008]

• [Our method] VQ+MLR: 1024 codewords

• Expert tags: 1053 tags from Pandora [Tingle, et al., 2009]

Page 37: More Like This: Machine Learning Approaches to Music similarity

Similarity learning: results

GMM (KL)

Auto-tags

Auto-tags + MLR

Audio VQ

Audio VQ + MLR

Expert tags (cos)

Expert tags + MLR0.65 0.70 0.75 0.80 0.85 0.90 0.95

AUC

Page 38: More Like This: Machine Learning Approaches to Music similarity

Example playlists

The Ramones - Go Mental

Def Leppard - Promises The Buzzcocks - Harmony In My Head Los Lonely Boys - Roses Wolfmother - Colossal Judas Priest - Diamonds and Rust (live)

Page 39: More Like This: Machine Learning Approaches to Music similarity

Example playlists

The Ramones - Go Mental

Def Leppard - Promises The Buzzcocks - Harmony In My Head Los Lonely Boys - Roses Wolfmother - Colossal Judas Priest - Diamonds and Rust (live)

The Buzzcocks - Harmony In My Head Mötley Crüe - Same Ol' Situation The Offspring - Gotta Get Away The Misfits - Skulls AC/DC - Who Made Who (live)

MLR

Page 40: More Like This: Machine Learning Approaches to Music similarity

Example playlists

Fats Waller - Winter Weather

Dizzy Gillespie - She's Funny That WayEnrique Morente - SoleaChet Atkins - In the MoodRachmaninov - Piano Concerto #4Eluvium - Radio Ballet

Page 41: More Like This: Machine Learning Approaches to Music similarity

Example playlists

Fats Waller - Winter Weather

Dizzy Gillespie - She's Funny That WayEnrique Morente - SoleaChet Atkins - In the MoodRachmaninov - Piano Concerto #4Eluvium - Radio Ballet

Chet Atkins - In the MoodCharlie Parker - What Is This Thing Called Love?Bud Powell - OblivionBob Wills & His Texas Playboys - Lyla LouBob Wills & His Texas Playboys - Sittin' On Top of the World

Page 42: More Like This: Machine Learning Approaches to Music similarity

Scaling up: fast retrieval

• Audio similarity search for a million songs?

• Idea: Index data with spatial trees

• 100-NN search over 900K songs: - Brute force: 2.4s - 50% recall: 0.14s 17x speedup - 20% recall: 0.02s 120x speedup

[M. & Lanckriet, 2011]

Page 43: More Like This: Machine Learning Approaches to Music similarity

Similarity learning: summary

• Collaborative filters provide user-centric music similarity

• CF similarity can be approximated by audio features

• Audio search can be done quickly at large-scale

Page 44: More Like This: Machine Learning Approaches to Music similarity

Playlist generation

Page 45: More Like This: Machine Learning Approaches to Music similarity

Playlist generation

• Goal: generate a "good" song sequence - Music auto-pilot (given context)

• Many existing algorithms, but no standard evaluation

• What makes one algorithm better than another?

Page 46: More Like This: Machine Learning Approaches to Music similarity

Playlist evaluation 1: Human survey

• Idea: generate playlists, ask for opinions

• Impractical at large-scale: - Huge search space - User taste, expertise can be problematic - Slow, expensive

• Does not facilitate rapid evaluation and optimization

Page 47: More Like This: Machine Learning Approaches to Music similarity

Playlist evaluation 2: Information retrieval

• Idea: - Define "good" and "bad" playlists - Predict the next song, measure accuracy

• But what makes a bad playlist?

• Do users agree on good/bad?

Page 48: More Like This: Machine Learning Approaches to Music similarity

A generative approach

• Playlist algorithm = distribution over playlists

• Don't evaluate synthetic playlists

• Do evaluate the likelihood of generating real playlists

[M. & Lanckriet, 2011b]

Page 49: More Like This: Machine Learning Approaches to Music similarity

The playlist collection: AOTM-2011

• Art of the Mix - 13 years of playlists - ~210K playlist segments - ~100K songs from MSD

• Top 25 playlist categories: - Genre: Punk, Hip-hop, Reggae... - Context: Road trip, Break-up, Sleep... - Other: Mixed genre, Alternating DJ...

Page 50: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

1. Start with a set of songs

Page 51: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

2. Select a subset (e.g., jazz songs)

Page 52: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

3. Select a song

Page 53: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

4. Select a new subset

Page 54: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

4. Select a new subset

Page 55: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

5. Select a new song

Page 56: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

6. Repeat...

Page 57: More Like This: Machine Learning Approaches to Music similarity

A simple playlist model

6. Repeat...

Page 58: More Like This: Machine Learning Approaches to Music similarity

Connecting the dots...

• Random walk on a hypergraph - Vertices = songs - Edges = subsets

• Edges derived from: - Audio clusters, tags, lyrics, era, popularity, CF - or combinations/intersections

• Goal: optimize edge weights from example playlists

Page 59: More Like This: Machine Learning Approaches to Music similarity

Playlist model

exp. prior

playlists

transitions

edge weights

Page 60: More Like This: Machine Learning Approaches to Music similarity

Playlist generation: evaluation

• Setup: - Split playlist collection into train/test - Learn edge weights on training playlists - Evaluate average likelihood of test playlists

• Train per category, or all together

• Compare against uniform shuffle baseline

Page 61: More Like This: Machine Learning Approaches to Music similarity

Random walk results

ALLMixed

ThemeRock-pop

Alternating DJIndie

Single artistRomanticRoad trip

PunkDepression

Break upNarrativeHip-hop

SleepElectronic

Dance-houseR&B

CountryCover songs

HardcoreRockJazzFolk

ReggaeBlues

0% 5% 10% 1 5% 20% 25%

Log-likelihood gain over random shuffle

Global modelCategory-specific

Page 62: More Like This: Machine Learning Approaches to Music similarity

Stationary model results

ALLMixed

ThemeRock-pop

Alternating DJIndie

Single artistRomanticRoad trip

PunkDepression

Break upNarrativeHip-hop

SleepElectronic

Dance-houseR&B

CountryCover songs

HardcoreRockJazzFolk

ReggaeBlues

Log-likelihood gain over random shuffle

-15% -10% -5% 0% 5% 10% 15% 20%

Global modelCategory-specific

Page 63: More Like This: Machine Learning Approaches to Music similarity

Example playlists

70s & soulAudio #14 & funkDECADE 1965 & soul

Lyn Collins - ThinkIsaac Hayes - No Name BarMichael Jackson - My Girl

Audio #11 & downtempoDECADE 1990 & trip-hopAudio #11 & electronica

Everything But The Girl - BlameMassive Attack - Spying GlassBjörk - Hunter

Rhythm & Blues

Electronic music

Page 64: More Like This: Machine Learning Approaches to Music similarity

Playlist generation summary

• Generative approach simplifies evaluation

• AOTM-2011 collection facilitates learning and evaluation

• Robust, efficient and transparent feature integration

Page 65: More Like This: Machine Learning Approaches to Music similarity

The future

Page 66: More Like This: Machine Learning Approaches to Music similarity

Directions for future work

• Audio features: coding, dynamics and rhythm

• Playlist models: mixtures, long-range interactions

• UI models: interactive, context-aware, diversity

Page 67: More Like This: Machine Learning Approaches to Music similarity

Personalized recommendation

• The Million Song Dataset Challenge

• Listening histories for 1.1M users, 380K songs

• Task: personalized song recommendation

[M., Bertin-Mahieux, Ellis, & Lanckriet, 2012]

Page 68: More Like This: Machine Learning Approaches to Music similarity

Conclusion

• MLR can optimize distance metrics for ranking, QBE retrieval

• Audio similarity can approximate a collaborative filter

• Generative playlist model integrates data, models dynamics

• User-centric evaluation makes it all possible

Page 69: More Like This: Machine Learning Approaches to Music similarity

Thanks!

Page 70: More Like This: Machine Learning Approaches to Music similarity
Page 71: More Like This: Machine Learning Approaches to Music similarity

Metric partial order feature

• Score is large when distances match ranking

Page 72: More Like This: Machine Learning Approaches to Music similarity

Playlist weights: 6390 edges

Audio CF Era Familiarity Lyrics Tags Uniform

ALLMixed

ThemeRock-pop

Alternating DJIndie

Single ArtistRomanticRoadTrip

PunkDepression

Break UpNarrativeHip-hop

SleepElectronic music

Dance-houseRhythm and Blues

CountryCover

HardcoreRockJazzFolk

ReggaeBlues

• Audio & CF: k-means (16/64/256)• Era: year, decade, decade+5• Familiarity: high/med/low

• Lyrics: LDA (k=32, top-1/3/5)• Tags: Last.fm top-10• Conjunctions


COSINE SIMILARITY APPROACHES TO RELIABILITY OF LIKERT

Backward Machine Transliteration by Learning Phonetic Similarity

Gene Prediction: Similarity-Based Approaches

A Similarity-Based Machine Learning Approach for Detecting Adversarial Android … · 2015-01-22 · A Similarity-Based Machine Learning Approach for Detecting Adversarial Android

Inductive Reasoning on Ontologies: Similarity-Based Approaches

22 Machine Vision: Approaches and Limitations

Machine Learning Approaches to Brewing Beer

Machine-Learning-Based Classification Approaches toward

Similarity-based Approaches to Short-term Load …Similarity-based Approaches to Short-term Load Forecasting Grzegorz Dudek Electrical Department, Czestochowa University of Technology,

Adaptive Machine Learning Approaches to Seasonal

Synergy of Database Techniques and Machine Learning ...CIKM 2019: Synergy of Database Techniques and Machine Learning Models for String Similarity Search and Join 27 Token-based similarity:

Design and Evaluation of Spatial Similarity …petrakis/publications/petr01b.pdfDesign and Evaluation of Spatial Similarity Approaches for Image Retrieval Euripides G.M. Petrakis Department

Gene Prediction: Similarity-Based Approaches. An Introduction to Bioinformatics Algorithms Outline The idea of similarity-based

Dynamic Programming II Gene Prediction: Similarity-Based Approaches The idea of similarity-based approach to gene prediction Exon Chaining Problem Spliced

Article Molecular Fingerprint-Derived Similarity Measures ...clok.uclan.ac.uk/24855/1/24855 Similarity-Based... · 2 17 ABSTRACT 18 Computational approaches are increasingly used

Feature-based approaches to semantic similarity assessment of concepts using Wikipedia

Comparing Machine Learning-Centered Approaches for

Quantitative approaches to linguistic similaritycysouw.de/home/presentations_files/cysouw... · Quantitative approaches to linguistic similarity Michael Cysouw Work-in-Progress, 9

Improving measurements of similarity judgments with machine-learning … · SIMILARITYJUDGMENTSANDMACHINELEARNING 1 Improving measurements of similarity judgments with machine-learning

Evaluating Machine Learning Approaches to Classify

Incorporating Machine-Learning into Music Similarity ...sjc/KW_ACM_06.pdf · Incorporating Machine-Learning into Music Similarity Estimation ... of these techniques consider only

Neural Machine Translation Inspired Binary Code Similarity ...Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs Fei Zuo y, Xiaopeng Li , Patrick

Bethany Percha - Machine Learning Approaches Poster

MACHINE LEARNING APPROACHES USING MATLAB

Machine learning approaches for surrogate modeling

Machine Learning Approaches to Macroeconomic Forecasting

Neural Machine Translation Inspired Binary Code Similarity ......Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs Fei Zuo y, Xiaopeng Li

MACHINE LEARNING APPROACHES FOR BIOMATERIAL MODELLING

Machine Translation: Challenges and Approaches

Robust Semantic Text Similarity Using LSA, Machine Learning, … · Robust Semantic Text Similarity Using LSA, Machine Learning, and Linguistic Resources Abhay Kashyap · Lushan Han

Theoretical Approaches to Machine Learning

Similarity-Based Approaches to Natural Language …arXiv:cmp-lg/9708011 19 Aug 1997 Harvard University Technical Report TR-11-97 Similarity-Based Approaches to Natural Language Processing

Topological approaches in machine learning

Gene Prediction: Similarity-Based Approaches - …bix.ucsd.edu/bioalgorithms/presentations/Ch06_GenePred...An Introduction to Bioinformatics Algorithms Similarity-Based Approach to

Learning Models of Similarity: Metric and Kernel Learningmilos/courses/cs3750... · Learned Model of Similarity Machine Learning Model Desired Output for Task . How to Learn a Similarity