music and machine learning

0.5setgray0

0.5setgray1

Music and Machine LearningUsing Machine Learning for the Classification of

Indian Music: Experiments and Prospects

Paritosh K. Pandya

School of Technology and Computer Science

Tata Institute of Fundamental Research

email: pandya@tifr.res.in

http://www.tcs.tifr.res.in/∼pandya

SNDT 2006 – p.

Outline

Motivation (Example)

Introduction to Machine Learning

Intelligent Music Processing Examples

Indian Music: Some questions

Automatic Raag Recognition System

SNDT 2006 – p.

Computers and Information Processing

Evolution of computers

Scientific calculations: e.g. planetary orbits

Data processing: e.g. inventory

Multimedia Rich Text, Graphics, Pictures, Animations,Video, Sound and Music.Computers can store, edit, process and display all ofthese!!

Internet and World-wide Web:

SNDT 2006 – p.

The Classical Music Archives

SNDT 2006 – p.

Computers and Arts

Computers and networks are increasingly used forstoring, processing (editing, cataloguing, searching),and disseminating artistic content.

Web Portals with artistic, educational andresearch-oriented content are becoming available e.g.complete works of Shakespeare

Computers can be used to analyse artistic content innew and sophisticated manners.

Computer as a tool for research in humanities and arts:Example Discovery channel news (2003): Computers toreveal Shakespeare

SNDT 2006 – p.

Learning Machines

Traditionally, computers are calculating devices. How tocalculate must be fully pre-programmed.

People observe patterns in nature, they discover rulesand they learn.

Can Computers learn? A question addressed byartificial intelligence.

Learning System A system capable of the autonomousacquisition and integration of knowledge.

SNDT 2006 – p.

How do systems learn?

Supervised Learning: Learn from examples.

Training Example Set (annotated)

Feature Selection (Input)

Target function representation

Statistical learning and classification

X

y

y = Number ofoccurrences of Ma after<Ga,MaTiv,Pa>

x = Number of oc-currences of Re after<Ga,MaTive,Pa>

SNDT 2006 – p.

Neural Nets

Learning functions

Given monthly icecream sales and average temperature forlast 10 years, predict icecream sales this summer.

SNDT 2006 – p.

Why use learning system?

The relationship between data elements is notformalized. Only examples are available.

Relationship between data items is buried within largeamount of data.

Data mining: using historical data to discover relationshipsand using this to improve future performance.

SNDT 2006 – p.

Applications of Machine Learning

Speech recognition

Image recognition (Face recognition)

Identifying Genes

Predicting Drug Activity

Cataloguing Faint Objects in Astronomical Data

Detecting Credit Card Frauds

Predicting Medical Outcomes from Historic Data

Detecting Hacking and Intrusion from Network Load

Computational Linguistics

SNDT 2006 – p. 10

Music Performance Visualisation

Performance worm [Widmer, Vienna]

Different players have different ways of building tensionor expression in the music

Measure subtle changes in beat level tempo versusloudness for each note played.

Represent this visually in tempo-loudness space as atrajectory called "performance worm".

SNDT 2006 – p. 11

Our Experiments

Real-time melody tracker (Click)

Time (s)0 15

Pitc

h (H

z)

120

300

sa

sa

re

ga

ma

pa

dha

ni

SNDT 2006 – p. 12

Recognition of Concert Pianists

Characterisation of Personal Expression FeaturesClassification

Classification between 22 piano players

Classification based on performance worm like data

Achieved accuracy comparable with human listeners.

[Saunders et al (2005)]

SNDT 2006 – p. 13

Islands of Music

Intelligent structuring and exploration of digital musiccollections [Pampalk et al (2004)]

Grouping of Music bySimilarity

Genre and Style

Performer

Timbral and rythmic con-tent

Automatic classification of music by Genre: Classical,Country, Disco, HipHop, Jazz and Rock [Pye 2000]About 90% success on 176 songs

SNDT 2006 – p. 14

Music Structure Analysis

Structure in Music Composition

repetition, transposition, call and response, rythmicpatterns and harmoic sequences

shape of a song e.g. AABA

Automatic structure analysis attempts to discover suchstructure [Danenberg,CMU]

Beat tracking and Tempo Detection

Identifying time signatures and tempo

Marking beat positions within music [Simon Dixon]

SNDT 2006 – p. 15

Music representation: Audio

Audio Waveform Santur playing(Click)

Rich set of features: Pitch, Amplitude, Spectra

SNDT 2006 – p. 16

Structured Music Representation

Example: Musical Score Notation

SNDT 2006 – p. 17

Computers and Music Notation

MIDI files: computer representation of musical score.

Can be recorded from keyboards etc.

Synthesizers: MIDI → Sound

Issue Expressive Music Representation(a hot topic for research!)

Music Notation for Indian Music

Bhatkhande or Paluskar systems

Not used by professional musicians

Lacks structures

SNDT 2006 – p. 18

Swarupa: Structured Music

define kaida2 as pat(

8.pat(dha,te,te,dha | te,te,dha,dha |

te,te,dha,ge | ti,na,ke,na),

8.pat(ta,te,te,ta | te,te,ta,ta |

te,te,dha,ge | dhin,na,ge,na) );

define palta1 as pat(

8.pat(dha,te,te,dha,te,te,dha,dha |

dha,te,te,dha,te,te,dha,dha),

8.pat(dha,te,te,dha,te,te,dha,dha |

te,te,dha,ge,dhin,na,ge,na),

8.pat(ta,te,te,ta,te,te,ta,ta |

ta,te,te,ta,te,te,ta,ta),

8.pat(dha,te,te,dha,te,te,dha,dha |

te,te,dha,ge,dhin,na,ge,na) );

SNDT 2006 – p. 19

Swarupa: Structured Music

define kaida2n as[A::[ 2:dhatita::[dha,te,te], dhadha::[dha,dha]

tite::[te, te], dha,ge | tin,na,ke,na ];B::[ A.1{khali} | C::[A.2 | A.3{bhari}] ]];

define palta1 as[ 3:[A.1 |] C; 3:[B.1 |] C; ];

define palta2 as[ A.1 | D::4:5%[te,te] ; B ;

B.1 | D ; B ];

Synthesis Swarupa → Audio (See MuM Webpage)Music transcription: Audio → Score

SNDT 2006 – p. 20

Indian Music Research using AI

Some topics

Classification of MusicRaag RecognitionClassification of Music in Thaats and JatisClassification of Raags into Time Cycle, SeasonalCycles,Classification of music by gharanasPerformer recognitionIdentification of Raag LakshansAssociation of Bhaav with musical performance

[B.Chaitanya Deva, 1981]

Beat tracking and taal recognition

Identification of musical structureSNDT 2006 – p. 21

Associated Applications

Music Visualisation

Musical query processing from large annotated musicaldatabases.

Automatic music composition

Automatic accompaniment

Pursuit: Distance Education of Indian Music

SNDT 2006 – p. 22

Machine Recognition of Raags

Raga performance as sequence of notes.Stop Sa Re Ga Pa Ga Re Sa Stop Dha Sa Re Ga

Sequential pattern classification problem

Data is not unordered set of samples.

Data elements occur in an order: spatial or temporal.

The probability of next data element crucially dependson the order of occurrence of preceeding elements.

Hidden Markov Models (HMM) are widely used.

SNDT 2006 – p. 23

Finite state automaton for Raag

Bhoopali

2Ga

3Re

4Pa

5Sa

6Dha

7Stop System can be in one of

finite number of states

Current state depends onthe past and current inputseen

Current state and next in-put determines the possiblenext states

Experiments: Manual con-struction of raag automatabased on Bhatkhande Books[Sahasrabuddhe]

SNDT 2006 – p. 24

Hidden Markov Model of a Raag

Bhoopali

2Ga 1.00

0.09

3Re 1.00

0.45

4Pa 1.00

0.235

Sa 0.99

0.06

0.51

0.41

0.45

0.076

Dha 1.00

0.41

0.38

0.07

0.32

0.62

0.30

7Stop 1.00

0.46

0.05

0.18

0.25

0.06

Probability ofseeing a note inthe given state.

Probability ofmoving from onestate to another

An HMM model canbe learnt from train-ing data

Analysis Given anHMM and a note se-quence, compute itsprobability of occur-rence.

SNDT 2006 – p. 25

Raag Recognition using HMM

Hidden Markov Model for a raag

Finite state automata

Probability of “seeing a note” in each state.

Probability of transition between states.

HMM model can be learnt from a set of training data

Given a note seqeuence we can compute its probabilitywithin given Raag HMM model.

SNDT 2006 – p. 26

Kansen: A raga recognition system

An Experiment at TIFR using a Toolkit HTK:

Learns HMM for each raag from training data(Baum-Welch Algorithm)

Training data: (Bhatkhande, IITK) collection of midi filesof raags played on keyboard. We use 29 raag database.

Test data: sequence of notes

Output: probability of the sequence being in each raag.

Preliminary Results

About 86 percent success on 29 raag recognition

Confusion between close raags

Insufficiency of dat a significant reason

(Joint work with Bhaumik Choksi and K. Samudravijaya)SNDT 2006 – p. 27

Bhatkhande

MIDI File Database of Indian Raags (IIT, Kanpur)

Adana, AheerBhairav, AlhiyaBilawal, Bageshri,Bahar, Basant, BasantMukhari, Behag, Bhoopali,BhoopaliTodi, ChandraKauns, ChhayaNut, Des,Durga, Gaud, Hamir, JataShwari, JaunaPuri Jogiya,Lalit, Malkauns, MiyankiMalhar, Multani, Pahadi,Peelu Sohini, TilakKamode, Tilang, Todi

Each midi file created by playing from Keyboard (e.g.Des.mid)

Basic database of 29 Raags (above)

Full database of 300+ Raags

SNDT 2006 – p. 28

Demonstration

Input Stop Sa Sa DhaKo Sa GaKo Re Stop Sa Ga Ga MaGaKo Re GaKo Re Sa Ni DhaKo Ni Sa Re Sa Ni SaOutput Log of probability of being in a raagI=1 t=0.02 W=ChandraKauns v=1

I=2 t=0.02 W=ChhayaNut v=1

I=3 t=0.02 W=Hamir v=1

I=4 t=0.02 W=Pahadi v=1

I=5 t=0.02 W=MiyankiMalhar v=1

I=6 t=0.02 W=Adana v=1

I=8 t=0.02 W=Peelu v=1

J=0 S=0 E=1 a=-164.11 l=0.000

J=1 S=0 E=2 a=-163.60 l=0.000

J=2 S=0 E=3 a=-160.92 l=0.000

J=3 S=0 E=4 a=-160.72 l=0.000

J=4 S=0 E=5 a=-158.36 l=0.000

J=5 S=0 E=6 a=-117.88 l=0.000

J=13 S=0 E=8 a=-88.55 l=0.000

Summary Peelu (-88) Adana (-117) MiyankiMalhar (-158)SNDT 2006 – p. 29

Demonstration (cont)

Stop NiKo Dha Ni Sa NiKo Pa Ma Pa GaKo Ma Re SaBahar (-20.5) MiyankiMalhar (-23) Adana (-53)

Stop Ma Pa NiKo Dha Ni Ni Sa Stop Ni Sa Re GaKoGaKo Ma Re SaMiyankiMalhar (-52) Bahar (-66) Adana (-75)

Stop Ma Pa Ni Ni Sa Ni Sa Sa Stop Pa Ni Sa Re NiKoDha Pa Stop Pa Dha Ma Ga Re Stop Ga Re Ni SaDes (-127) Gaud (-139) MiyankiMalhar (-144)

Stop Ga Ma DhaKo DhaKo Pa Stop Ma Pa GaKo MaReKo Sa Stop Ga Ma Pa DhaKo Ni Sa DhaKo PaBasant Mukhari (-105) Peelu (-106) Jogiya (-130)

SNDT 2006 – p. 30

Demonstration (cont)

Stop Ni Sa GaKo ReKo Sa Stop Ni Sa GaKo MaTiv PaStop GaKo MaTiv Pa Ni Sa DhaKo Pa MaTiv GaKoReKo SaMultani (-76) Todi (-105) ChandraKauns (-169)

Stop DhaKo Ni Sa ReKo GaKo Stop ReKo GaKo ReKoSa Stop Sa ReKo GaKo MaTiv ReKo GaKo ReKo SaTodi (-58) Bhoopali Todi (-83) Multani (-124)

SNDT 2006 – p. 31

Conclusions

Computer analysis and machine learning providesinteresting new method of analysing music. It allowsmany intuitive and qualitative observations to be madeobjective, precise and quantitative.

Research with computational techniques lead to directapplications in music technology.

Intelligent music analysis is almost untried for IndianMusic.

Work requires collaboration between musicologists,computer scientists and electrical engineers.

Music researchers must help by building corpuses andannotated datasets for future machine analysis.

SNDT 2006 – p. 32