1 roberto bresin – humaine workshop s2s 2 – 2004.09.19_21 real-time visualization of musical...
Post on 31-Mar-2015
212 Views
Preview:
TRANSCRIPT
1Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Real-time Visualizationof
Musical Expression
Roberto BresinSpeech, Music and Hearing Dep. - KTH
http://www.speech.kth.se/~robertohttp://www.speech.kth.se/music
2Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Outlook
• Expressive music performance
• Acoustical cues of importance for communicating expressivity
• How acoustical cues can influence the performance style/intended emotion
• Real-time Visualization of Musical Expression
• Examples and demonstration
3Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The MusicianThe musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
4Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The Score and the Performance
How important is the performance?
• Dead-pan by computer and sampler• Schumann’s Träumerei by Alfred Brendel
10
20
30
40
50
60
70
80 ²DR [%] Horowitz 65
-10
-20
-30
1 2 3 4 5 6 7 8 9
10
20
30
40
50
60
70
80
-10
-20
-30
10
10
10
20
30 ²DR [%] Schnabel
-10
-20
1 2 3 4 5 6 7 8 9
30
-10
-20
10
20
30
40
50 ²DR [%] Brendel
-10
-20
-30
1 2 3 4 5 6 7 8 9
20
30
40
50
-10
-20
-30
IOI (%) BrendelIOI (%) Brendel
Tim
e d
evia
tio
n f
rom
sco
re
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
5Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The Score and the Performance
10
20
30
40
50
60
70
80 ²DR [%] Horowitz 65
-10
-20
-30
1 2 3 4 5 6 7 8 9
10
20
30
40
50
60
70
80
-10
-20
-30
10
10
10
20
30 ²DR [%] Schnabel
-10
-20
1 2 3 4 5 6 7 8 9
30
-10
-20
10
20
30
40
50 ²DR [%] Brendel
-10
-20
-30
1 2 3 4 5 6 7 8 9
20
30
40
50
-10
-20
-30
IOI (%) BrendelIOI (%) BrendelTim
e d
evia
tio
n f
rom
sco
re
IOI (%) IOI (%) SchnabelSchnabel
IOI (%) IOI (%) Horowitz 65Horowitz 65
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
6Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Design of Performance Rules
Performance rules obtained mainly with 2 methods:
• analysis-by-synthesis• analysis-by-measurement
Generative grammar for automatic music performance
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
7Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Dead-pan, K=0
Exaggerated, K = 4.4
Moderate, K = 2.2
Inverted, K = -2.2
Duration contrast ruleIn
ter o
nset
I nte
rval d
evia
t ion
s (
%)
8Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Phrase Arch Rule
Dead-pan
Exaggerated
ΔIO
I (%
)
9Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
KTH Performance Rules
• Descriptions of different performance principles used by
musicians
• General applicability
• K values change the overall quantity of each rule
• Context dependency
• ~ 30 rules
• 30 years of research at KTH
ScoreScore RulesRules PerformancePerformance
K valuesK values
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
10Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
KTH Performance Rules
Phrasing Phrase archFinal
ritardandoPunctuationHigh loud
Harmonic/melodic tension Harmonic/melodic charge
Repetitive patterns and grooves Swing
Articulation PunctuationStaccato/legato
Accents Accent rule
Ensemble timing Ensemble swingMelodic sync
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
11Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Director MusicesA program for modelling music performancehttp://www.speech.kth.se/music/performance
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
12Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Mapping from Emotional Expression to Rule Parameters
For each emotion:
Select a palette of rule parameters according to previous findings
Mapping
Emotional expression
Rule parameters
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
13Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Cues for the simulation of
emotions in music
performance
(by A.Gabrielsson and P.Juslin,
Psychology of Music, 1996, vol. 24)
• No expression
• Tenderness
• Solemnity
• Happiness
• Sadness
• Anger
• Fear
Synthesis of Emotional ExpressionThe musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
14Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
• TENDERNESSslow mean tempo (Ga96)slow tone attacks (Ga96)low sound level (Ga96)small sound level variability (Ga96)legato articulation (Ga96)soft timbre (Ga96)large timing variations (Ga96)accents on stable notes (Li99)soft duration contrasts (Ga96)final ritardando (Ga96)
• HAPPINESSfast mean tempo (Ga95)small tempo variability (Ju99)staccato articulation (Ju99)large articulation variability (Ju99)high sound level (Ju00)little sound level variability (Ju99)bright timbre (Ga96)fast tone attacks (Ko76)small timing variations (Ju/La00)sharp duration contrasts (Ga96)rising micro-intonation (Ra96)
• ANGERhigh sound level (Ju00)sharp timbre (Ju00)spectral noise (Ga96)fast mean tempo (Ju97a)small tempo variability (Ju99)staccato articulation (Ju99)abrupt tone attacks (Ko76)sharp duration contrasts (Ga96)accents on unstable notes (Li99)large vibrato extent (Oh96b)no ritardando (Ga96)
• SADNESSslow mean tempo (Ga95)legato articulation (Ju97a)small articulation variability (Ju99)low sound level (Ju00)dull timbre (Ju00)large timing variations (Ga96)soft duration contrasts (Ga96)slow tone attacks (Ko76)flat micro-intonation (Ba97)slow vibrato (Ko00)final ritardando (Ga96)
• FEARstaccato articulation (Ju97a)very low sound level (Ju00)large sound level variability (Ju99)fast mean tempo (Ju99)large tempo variability (Ju99)large timing variations (Ga96)soft spectrum (Ju00)sharp micro-intonation (Oh96b)fast, shallow, irregular vibrato (Ko00)
Positive Valence
Negative Valence
High ActivityLow Activity
From Juslin (2001)
15Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Example: SADNESS
Expressive Cue Analysis Synthesis (Director Musices)
Tempo Slow Tone IOI is lengthened by 30%
Sound level Moderate or low Sound level is decreased by 6 dB
Articulation Legato Tone duration = Tone IOI
Time deviations Moderate Duration Contrast Rule (k = -2)
Phrase Arch Rule applied on phrase level (k = 1.5)
Phrase Arch Rule applied on sub-phrase level (k = 1.5)
Final ritardando Yes Obtained from the Phrase Arch Rule
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
16Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
IOI deviations
dB deviations
articulation
Example: SADNESSModel Score
17Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
ConclusionsConclusions
Emotional expressioncan be derived directlyfrom the music score,
simplyby enhancing music structure
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
18Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Better Monophonic Ringtones!
Today (nom.)
Natural
:-) Happy
>:-( Angry
:-( Sad
=|:-| Solemn
Mozart G minor
0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8 0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
0 2 4 6 8 10 12 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Dead-pan
Happy
Angry
Natural
Sad
Solemn
www.notesenses.com
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
19Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Usher, Burn
MechanicalMusicalHappy
Jennifer Ellison, Bye Bye Boy
MechanicalMusicalRomantic
Better Polyphonic Ringtones!
www.notesenses.com
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
20Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
pDM –performance rules in real-timeAnders Friberg + MEGA project (IST EU)
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
21Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Real-time Visualization of
Musical Expression
22Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Background
Feel-Me project
Design a computer program for teaching students to play expressively
The system includes a tool for automatic extraction of acoustic cues (CUEX):
pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
23Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Tempo
Loudness
Timbre
Encoding Decoding
Cue Utilization Cue Utilization
The Listener
Articula.
The Performance The Performer intention expressive cues judgment
Accuracy
others
Anger Anger
rPerformer rListener
Matching
.87
.26 .47 .63
-.26
.22 .55 .61
-.39
.92
Lens model: quantifies the expressive communication between performer and listener
24Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Aim
Design a tool for real-time visual feedback to expressive performance
Mapping of acoustic cues:
– Non-verbal– Intuitive– Informative (including emotional expression)
Previous studies: cross-modality speeds stimuli discrimination
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The Listener Recognition of emotionVisualisation of musical expression
25Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Cue analysis
Expression mapper
Audio Emotion
TempoSound level
Articulation...
Implementations
CUEX
Simplified real-timeversion
Mult. Regression
Fuzzy inspired
Recognition of EmotionThe musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
26Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
From audio to emotion recognition
Standardisation
Audio input
Cue extraction
Fuzzy mapping
- +0
∑ / 3
Happiness 0-1Happiness 0-1
temposoundlevel articulation
∑ / 3
Sadness 0-1
∑ / 3
Anger 0-1
Fuzzy mapping
- +0
Fuzzy mapping
- +0
The musicianIntroductionModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionBody ExpressionGhost in the Cave
27Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment
2 melodies, Brahms (minor) & Haydn (Major)3 instruments (piano, guitar, saxophone)12 performances per instrument (12 emotional intentions)
24 colour nuances8 levels of hue 2 levels of brightness 2 levels of saturation
2 groups of 11 subjects each(1 group per melody)
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
28Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment: main resultsHUE
Happiness YellowFear BlueSadness Violet & BlueAnger RedLove Blue & Violet
BRIGHTNESS
Observed tendency:Minor tonality Low brightness (Dark colours)Major tonality High brightness (Light colours)
Interaction involving sadness:Even for major tonality low brightness is preferred for all instruments
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
29Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment: main resultsBox & Whis ker Plo t: happ ines s
Brigh tnes s
ha
pp
ine
ss
Mean ±SE ±1 ,96*SE
Inst
rum
en
t: P
ian
o
-0 , 5
0 ,0
0 ,5
1 ,0
1 ,5
2 ,0
2 ,5
3 ,0
Inst
rum
en
t: G
uita
r
-0 , 5
0 ,0
0 ,5
1 ,0
1 ,5
2 ,0
2 ,5
3 ,0
P iec e : H A Y D N
Inst
rum
en
t: S
axo
ph
on
e
1 . 0 0 . 5-0 ,5
0 ,0
0 ,5
1 ,0
1 ,5
2 ,0
2 ,5
3 ,0
P iec e : B R A H MS
1 .0 0 . 5
30Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment: main resultsBox & Whis ker Plo t: s adnes s
Brigh tnes s
sad
ne
ss
Mean ±SE ±1 ,96*SE
Inst
rum
en
t: P
ian
o
0 , 20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6
Inst
rum
en
t: G
uita
r
0 , 20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6
P iec e : H A Y D N
Inst
rum
en
t: S
axo
ph
on
e
1 . 0 0 . 50 ,20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6
P iec e : B R A H MS
1 .0 0 .5
31Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment: main resultsSimilar colour palettes within instruments
DISGUST (SAX)
0
0,5
1
1,5
2
2,5
3
3,5
4
red
red
_d
ark
red
_lig
ht
ora
ng
e
ora
ng
e_
da
rk
ora
ng
e_
ligh
t
yello
w
yello
w_
da
rk
yello
w_
ligh
t
gre
en
gre
en
_d
ark
gre
en
_lig
ht
cya
n
cya
n_
da
rk
cya
n_
ligh
t
blu
e
blu
e_
da
rk
blu
e_
ligh
t
vio
let
vio
let_
da
rk
vio
let_
ligh
t
ma
ge
nta
ma
ge
nta
_d
ark
ma
ge
nta
_lig
ht
Haydn
Brahms
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
32Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Experiment: main resultsSHAME (SAX)
0
1
2
3
4
5re
d
red
_d
ark
red
_lig
ht
ora
ng
e
ora
ng
e_
da
rk
ora
ng
e_
ligh
t
yello
w
yello
w_
da
rk
yello
w_
ligh
t
gre
en
gre
en
_d
ark
gre
en
_lig
ht
cya
n
cya
n_
da
rk
cya
n_
ligh
t
blu
e
blu
e_
da
rk
blu
e_
ligh
t
vio
let
vio
let_
da
rk
vio
let_
ligh
t
ma
ge
nta
ma
ge
nta
_d
ark
ma
ge
nta
_lig
ht
Haydn
Brahms
SHAME (GUITAR)
0
1
2
3
4
5
red
red
_d
ark
red
_lig
ht
ora
ng
e
ora
ng
e_
da
rk
ora
ng
e_
ligh
t
yello
w
yello
w_
da
rk
yello
w_
ligh
t
gre
en
gre
en
_d
ark
gre
en
_lig
ht
cya
n
cya
n_
da
rk
cya
n_
ligh
t
blu
e
blu
e_
da
rk
blu
e_
ligh
t
vio
let
vio
let_
da
rk
vio
let_
ligh
t
ma
ge
nta
ma
ge
nta
_d
ark
ma
ge
nta
_lig
ht
Haydn
Brahms
SHAME (PIANO)
0
1
2
3
4
5
red
red
_d
ark
red
_lig
ht
ora
ng
e
ora
ng
e_
da
rk
ora
ng
e_
ligh
t
yello
w
yello
w_
da
rk
yello
w_
ligh
t
gre
en
gre
en
_d
ark
gre
en
_lig
ht
cya
n
cya
n_
da
rk
cya
n_
ligh
t
blu
e
blu
e_
da
rk
blu
e_
ligh
t
vio
let
vio
let_
da
rk
vio
let_
ligh
t
ma
ge
nta
ma
ge
nta
_d
ark
ma
ge
nta
_lig
ht
Haydn
Brahms
33Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The ExpressiBall Expressive performance space as a mapping of acoustical cues and emotions
X Tempo Color EmotionY Sound level Shape Articulation Z Attack velocity & Spectrum energy
Lo
ud
So
ft
Slow Fast
StaccatoAngryFast attackHigh energy
Lo
ud
So
ftSlow Fast
LegatoSadSlow attackLow energy
34Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The ExpressiBall
DEMO!
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
35Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
ExpressiBall:Current & Future Work…
• Sonification of the ExpressiBall
• Usability test with students
• Set-up depending on instrument/spectrum• Use complex colour palettes (Pictures?
Abstract patterns/shapes?)
• Other possible applications:• ”Colour Monitor” in discotheques• Computer screen saver• …
The musicianMusical communicationModelling music performanceEmotional colouringApplications
The ListenerRecognition of emotionVisualisation of musical expression
36Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
Outlook
• Expressive music performance
• Acoustical cues of importance for communicating expressivity
• How the use of acoustical cues can influence the performance style/intended emotion
• Real-time Visualization of Musical Expression
• Examples and demonstration
http://www.speech.kth.se/music/performance
37Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
• FET-Open Coordination Action (contract no. IST-2004-03773)
• 11 partners (3 Humaine partners)
• Started: June 1, 2004
• Duration: 3 years
• Overall objective:
”To bring together the state-of-the-art research in the sound domain and in the proper combination of human sciences, technological research and neuropsychological sciences that does relate to sound and sense”.
Sound to Sense, Sense to Soundhttp://www.s2s2.org
38Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21
The End
http://www.speech.kth.se/music/performance
top related