exploitingaudio data formusicological research€¦ · 2006: abitur, max-reger-gymnasium amberg...

Exploiting Audio Data for MusicologicalResearch

Christof WeißInternational Audio Laboratories Erlangen

[email protected]

18.7.2019

Heidelberg Computational Humanities Summer School

2

2006: Abitur, Max-Reger-Gymnasium Amberg

2006-2012: Physics diploma, Universität Würzburg

2006-2011: Music diploma Composition, HfM Würzburg(Prof. Heinz Winbeck)

2011-2012: Concert diploma Composition (Tobias Schneid)

2012-2015: Ph. D. Fraunhofer Institute for Digital Media Technology (IDMT), Ilmenau, Thüringen, funded by Stiftung der Dt. Wirtschaft (sdw)

Computational Methods for Tonality-Based Style Analysis ofClassical Music Audio Recordings

Since 09/2015: AudioLabs Erlangen / freelancing composer

2018: KlarText award for science communication

Christof Weiß

3

International Audio Laboratories Erlangen

4

Prof. Dr. Jürgen HerreAudio Coding

Prof. Dr. Bernd EdlerAudio Signal Analysis

Prof. Dr. Meinard MüllerSemantic Audio Processing

Prof. Dr. Emanuël HabetsSpatial Audio Signal Processing

Prof. Dr. Frank WefersVirtual Reality

Dr. Stefan TurowskiCoordinator AudioLabs-FAU

AudioLabs - FAU

5

Audio

Audio Coding

Music ProcessingPsychoacoustics

3D Audio

International Audio Laboratories Erlangen

6

Hendrik Schreiber

Frank Zalkow

Sebastian Rosenzweig

Christof Weiß

Group Prof. Meinard Müller

7

Music Synchronization Structure Analysis

Tempo Estimation and Beat Tracking

Automatic Music Transcription

Harmony Analysis

Music Processing / Music Information Retrieval (MIR)

8

1. Signal Processing: Chroma Features

2. Harmony Analysis

3. Cross-Version Analysis: Wagner’s Ring

4. Corpus Analysis: Composer Styles

Outline

9

Signal Processing: Chroma Features

Time (seconds)

Freq

uenc

y(H

z)

Example: C major scale (piano)

Score

Audio – Waveform

Audio - Spectrogram

10


Example: C major scale (piano)

Audio – Chromagram

Audio – Chromagram (normalized)

Time (seconds)

Pitc

h cl

ass

Pitc

h cl

ass

Time (seconds)

11


Salie

nce

/ Lik

elih

ood

L. van Beethoven,Fidelio, Overture,Slovak Philharmonic

Time (seconds)

Time (seconds)

12



Time (seconds)

Time (seconds)

Spectrogram: Time – Frequency

Salie

nce

/ Lik

elih

ood

13



Time (seconds)

Time (seconds)

Salie

nce

/ Lik

elih

ood

MID

I pitc

hnu

mbe

r

Log-Frequency-Spectrogram: Time – Pitch

14



Time (seconds)

Time (seconds)

Salie

nce

/ Lik

elih

ood

Pitc

h cl

ass

Chromagram: Time – Pitch class

15


Orchestra

Piano


Fidelio, Overture,arr. Alexander ZemlinskyM. Namekawa, D.R. Davies, piano four hands

16

Piano


Fidelio, Overture,arr. Alexander ZemlinskyM. Namekawa, D.R. Davies, piano four hands

17

Salie

nce

Salie

nce

Salie

nce

Time (seconds)

Time (seconds)

Time (seconds)

Pitc

h cl

ass

Cho

rdty

pe

Inte

rval

cate

ogry

Pitc

h cl

ass

AugmentedDimished

MinorMajor

TritoneFourth / fifth

Major third / minor sixthMinor third / major sixth

Major second / minor seventhMinor second / major seventh


Chromagram

Chord types

Interval categories

→ transposition-invariant features!

18


Time (minutes)

Pitc

h cl

ass

Pitch class

Chromagram: Full piece

Chroma statistics:

19

2. Harmony Analysis

20

Harmony analysis of music: Different concepts Concepts relate to different temporal granularity

ChordsCM GM7 Am

Global key

Local keyC major G major C major

Global key detection

Chord recognition

Music transcriptionNote level

Segment level

Chord level

Movement level C major

MelodyMiddle voices

Bass line

Local key detection

Motivation

21


Global key Global key detection



MelodyMiddle voices

Bass line

Local keyC major G major C majorSegment level Local key detection

Motivation

ChordsCM GM7 Am Chord recognitionChord level

22

Source: www.ultimate-guitar.com

The Beatles, Let it be – Chords

Harmony Analysis: Chords

23


ChordsCM GM7 Am

Global key

Local keyC major G major C major

Global key detection

Chord recognition


Segment level

Chord level


MelodyMiddle voices

Bass line

Local key detection

Motivation

24


ChordsCM GM7 Am

Global key Global key detection

Chord recognition


Chord level


MelodyMiddle voices

Bass line

Local keyC major G major C majorSegment level Local key detection

Motivation

25

E maj B maj E maj B maj E maj

Modulation

Stollen Stollen Abgesang

„Bar form“

Johann Sebastian Bach, Choral “Durch Dein Gefängnis” (St. John’s Passion) – Local keys

Musical form:

Harmony Analysis: Local Keys

26

E maj B maj

E maj



27

B majE maj G

D

E

BD♭

E♭

F

A

G ♭F#

A ♭

B♭

E ♭mD#m

B ♭mFm

Cm

Gm

Dm

G#m

C#m

F#m

Bm

EmAm

C

♭ ♯

Circle of fifths

Series of fifths

Johann Sebastian Bach, Choral “Durch Dein Gefängnis”(St. John’s Passion) – Local keys


28

B majE maj G

D

E

BD♭

E♭

F

A

G ♭F#

A ♭

B♭

E ♭mD#m

B ♭mFm

Cm

Gm

Dm

G#m

C#m

F#m

Bm

EmAm

C

♭ ♯

Circle of fifths

Series of fifths



0 diatonic

29

B majE maj G

D

E

BD♭

E♭

F

A

G ♭F#

A ♭

B♭

E ♭mD#m

B ♭mFm

Cm

Gm

Dm

G#m

C#m

F#m

Bm

EmAm

C

♭ ♯

Circle of fifths

Series of fifths



1# diatonic

30

B majE maj G

D

E

BD♭

E♭

F

A

G ♭F#

A ♭

B♭

E ♭mD#m

B ♭mFm

Cm

Gm

Dm

G#m

C#m

F#m

Bm

EmAm

C

♭ ♯

Circle of fifths

Series of fifths



2♭ diatonic

31

5#4# GD

E

BD♭

E♭

F

A

G ♭F#

A ♭

B♭

E ♭mD#m

B ♭mFm

Cm

Gm

Dm

G#m

C#m

F#m

Bm

EmAm

C

♭ ♯

Circle of fifths

Series of fifths



32

Example: J.S. Bach, Choral “Durch Dein Gefängnis” Score – Piano reduction

Visualization of Diatonic Scales

33

Example: J.S. Bach, Choral “Durch Dein Gefängnis” Audio – Waveform (Scholars Baroque Ensemble, Naxos 1994)

Time (seconds)


34

Example: J.S. Bach, Choral “Durch Dein Gefängnis” Audio – Spectrogram (Scholars Baroque Ensemble, Naxos 1994)


35

Example: J.S. Bach, Choral “Durch Dein Gefängnis” Audio – Chroma features (Scholars Baroque Ensemble, Naxos 1994)


36

Example: J.S. Bach, Choral “Durch Dein Gefängnis” Chroma features – smoothing


37



38



39



40


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Re-ordering to perfect fifth series

41


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Re-ordering to perfect fifth series

42

4#


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales (7 fifths)

43

5#


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales (7 fifths)

44


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales – multiplication

45



46



47

4# 5# 4# 5# 4#



48

4 #(E major)


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales – shift to global key

49

4 #(E major)


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales – shift to global key

50

C. Weiß, J. Habryka, “Chroma-Based Scale Matching for Audio Tonality Analysis” Proc. Conference on Interdisciplinary Musicology, 2014.


Example: J.S. Bach, Choral “Durch Dein Gefängnis” Diatonic Scales – relative

Modulation

[1]

51

L. v. Beethoven – Sonata No. 10 op. 14 Nr. 2, 1. Allegro — 0 ≙ 1(Barenboim, EMI 1998)


Exposition Exposition Durchführung Reprise

sonataform

52

R. Wagner, Die Meistersinger von Nürnberg, Vorspiel — 0 ≙ 0(Polish National Radio Symphony Orchestra, J. Wildner, Naxos 1993)


53

3. Cross-Version Tonality Analysis of Wagner’sRing des Nibelungen

54

Cooperation: Musicology

DFG-funded project: “Computer-Assisted Analysis of Harmonic Structures” Harmony analysis and visualization 1st phase: 2014–2018, 2nd phase: 2019–2023

Partners: Rainer Kleinertz, Musicology Univ. Saarland Stephanie Klauk, Musicology Univ. Saarland Meinard Müller, AudioLabs FAU Christof Weiß, AudioLabs FAU

Central work: Richard Wagner, Der Ring des Nibelungen

How is harmony organized at the large scale?

55

1 2 3 4

Cross-Version Analysis

No. Conductor Recording hh:mm:ss1 Barenboim 1991–92 14:54:552 Boulez 1980–81 13:44:383 Böhm 1967–71 13:39:284 Furtwängler 1953 15:04:225 Haitink 1988–91 14:27:106 Janowski 1980–83 14:08:347 Karajan 1967–70 14:58:088 Keilberth/Furtwängler 1952–54 14:19:569 Krauss 1953 14:12:2710 Levine 1987–89 15:21:5211 Neuhold 1993–95 14:04:3512 Sawallisch 1989 14:06:5013 Solti 1958–65 14:36:5814 Swarowsky 1968 14:56:3415 Thielemann 2011 14:31:1316 Weigle 2010–12 14:48:46

16 different performances (versions) Manual measure annotations for 3 versions

56

Cross-Version Analysis

16 different performances (versions) Manual measure annotations for 3 versions

Visualize cross-version consistency with gray scale

Ampl

itude

Time (measures)

Time (seconds)

57

Act 1

Act 2

Act 3

Die Walküre WWV 86 B

58


Act 1, measures 955–1012 Sieglinde’s narration

Weiss / Kleinertz / Müller, Möglichkeiten der computergestützten Erkennung und Visualisierung harmonischer Strukturen — eine Fallstudie zu Richard Wagners "Die Walküre",Proc. Jahrestagung der Gesellschaft für Musikforschung 2015

[2]

59


Act 1, measures 955–1012 Sieglinde’s narration

Weiss / Kleinertz / Müller, Möglichkeiten der computergestützten Erkennung und Visualisierung harmonischer Strukturen — eine Fallstudie zu Richard Wagners "Die Walküre",Proc. Jahrestagung der Gesellschaft für Musikforschung 2015

[2]

60

Act 1

Act 2

Act 3


61


Act 3, measures 724–789 Wotan’s punishment

Weiss / Zalkow / Müller / Klauk / Kleinertz, Computergestützte Visualisierung harmonischer Verläufe: Eine Fallstudie zu Wagners Ring,Proc. Jahrestagung der Gesellschaft für Informatik 2017

[3]

62


Act 3, measures 724–789 Wotan’s punishment

Weiss / Zalkow / Müller / Klauk / Kleinertz, Computergestützte Visualisierung harmonischer Verläufe: Eine Fallstudie zu Wagners Ring,Proc. Jahrestagung der Gesellschaft für Informatik 2017

[3]

63

Die Walküre WWV 86 BA. Lorenz, 1924

64

Exploring Tonal-Dramatic Relationships

Histograms of Analysis over timeD

iato

nic

Scal

es

Das RheingoldWWV 86 A

3897 measures

Die WalküreWWV 86 B

5322 measures

SiegfriedWWV 86 C

6682 measures

GötterdämmerungWWV 86 D

6040 measures

Time (measures)

65


Die Walküre – Sword motif

Diatonic Scales

Zalkow / Weiss / Müller, Exploring Tonal-Dramatic Relationships in Richard Wagner's Ring Cycle,Proc. Int. Conference on Music Information Retrieval 2017

[4]

66


Siegfried – Sword motif

Diatonic Scales


[4]

67


Das Rheingold – Valhalla motif


[4]

Chords

68


Die Walküre – Valhalla motif

ChordsZalkow / Weiss / Müller, Exploring Tonal-Dramatic Relationships in Richard Wagner's Ring Cycle,Proc. Int. Conference on Music Information Retrieval 2017

[4]

69

4. Corpus Analysis: Composer Styles

70

Global chroma statistics (audio) 1783 – W. A. Mozart, „Linz“ symphony KV 425, 1. Adagio / Allegro

G#Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class

Circle of fifths →

Tonal Complexity

71

Global chroma statistics (audio) 1883 – J. Brahms, Symphony No. 3, 1. Allegro con brio (F major)

Ab Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class

Tonal Complexity


72

Global chroma statistics (audio) 1940 – A. Webern, Variations for Orchestra op. 30

Ab Eb Bb F C G D A E B F# C#

1

0.8

0.6

0.4

0.2

0

Salie

nce

Pitch class

Tonal Complexity


73

Realization of complexity measure Γ Distribution over Circle of Fifths

Relating to different time scales!

𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 0

𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 1 𝐿𝑒𝑛𝑔𝑡ℎ

𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 1 𝐶𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦 0.6Tonal Complexity

74

Weiss / Müller, Quantifying and Visualizing Tonal Complexity,Proc. Conference on Interdisciplinary Musicology 2014

[5]

Com

plex

ityΓ

Tonal Complexity – Chords

75

Tonal Complexity – Beethoven’s Sonatas

76

17001650 1750 1800 1850 1900 1950 2000

2000 pieces

Piano and Orchestra music

Analyzing Composer Styles

77

17001650 1750 1800 1850 1900 1950 2000

Haydn, Joseph1732 – 1809100 works in dataset


78

17001650 1750 1800 1850 1900 1950 2000

1700 1750 1800 1850 1900 1950

1

0.9

0.8

0.7

Com

plex

ityΓ

Complexity Global

Complexity Mid-scale

Complexity Local


79

Clustering Composition Years

Weiss / Mauch / Dixon / Müller, Investigating Style Evolution of Western Classical Music: A Computational Approach, Musicae Scientiae 2018

[6]

80

Clustering Individual Pieces

81

17001650 1750 1800 1850 1900 1950 2000

C. P. E. Bach, Sonata Wq. 49No. 3 in E minor, II. AdagioC. Colombo, Piano

R. Schumann, Sonata No. 2op. 22, II. AndantinoB. Glemser, Piano

Clustering Composers

82

Clustering Composers

Hierarchical clustering From Bioinformatics

83

Tonal Complexity: Jazz Solos

Year

Com

plex

ity

Year

Com

plex

ity

Symbolic transcription

Audio recording (harmonic part)

84

Tonal Complexity: Jazz Solos

Year

Com

plex

ity

Year

Com

plex

ity

Symbolic transcription

Audio recording (harmonic part)

Weiss / Balke / Abesser / Müller, Computational Corpus Analysis: A Case Study on Jazz Solos, Proc. Int. Conference on Music Information Retrieval 2018

[7]

85

Signal Processing & Machine Learning

Music Analysis & Music Theory

Chroma-basedfeatures

Tonal complexity

Local keys

Modulations

Chordtransitions

EngineeringComputer Science

MusicologyHumanities

Diatonic scalerelations

Conclusions

Spectrogram

Chromagram

Interval types

Style analysis

Evolution oftonal features

Composer clustering

Visualization

Musical form

Harmony analysis

86

Signal Processing & Machine Learning

Music Analysis & Music Theory

Chroma-basedfeatures Modulations

EngineeringComputer Science

MusicologyHumanities

Conclusions

Spectrogram

Chromagram

Musical form

Harmony analysis

Tonal complexity

Local keys

Chordtransitions

Diatonic scalerelations

Interval types

Style analysis

Evolution oftonal features

Composer clustering

Visualization

„Computational“ phenomena!

exploitingaudio data formusicological research€¦ · 2006: abitur, max-reger-gymnasium amberg...

Documents