jaewoo chung scotty vercoe mas630/affective computing spring '05 the affective listener
TRANSCRIPT
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
“How remarkable would it be if one could experience and express the spectrum of emotions embodied in music originating from oneself, without the crutch of a composer’s intercession…Can the touch that lies behind music be tapped?”–Manfred Clynes
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Goal:• Direct music using affective cues
Issues:• Devise mapping scheme of music parameters• Correlate affective signals with music parameters• Disambiguate data collected during music listening• Develop algorithm to navigate music map by affect
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Presentation Outline
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Existing ResearchBerlyne (1974):• liking is greatest for stimuli of moderate arousal• arousal and liking should be dependent; familiarity and complexity affect arousal levelRussell et al., (1981): Circumplex model of emotion• puts emotions in a circle around grid of pleasant-unpleasant and arousing-sleepyNorth and Hargreaves (1997):• correlated circumplex model to valence/arousal dimensions• replaced pleasant-unpleasant axis with like-dislike axis in circumplex• coordinates of liking-arousal was found to be a reliable predictor of emotional reactionSchubert (1996):• people often enjoy music that is unpleasantRitossa and Rickard (2004):• tested dimensions of pleasantness and liking in circumplex• one of eight emotions reliably predicted using arousal, familiarity & pleasantness• pleasantness was more useful predictor of emotions than liking
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Existing ResearchMarrin and Picard (1998): Conductor’s Jacket• records a variety of affective signals during conductingHealey, Picard and Dabek (1998): Affective DJ• uses a mixture of physiological and subjective data• compares SCR from first 30 seconds of current song to last 30 seconds of previous song• algorithm selects songs to change from current affective state to desired stateKim and André (2004): Composing Affective Music• uses ECG, EMG, SCR and RESP data• composes algorithmically• SCR is useful indicator for unsettling-relaxing• EMG useful indicator for positive-negative
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Music AffectMapping
Music ParameterMapping
user study
music box
Project Overview1) Small-scale listening
experiment:Change music parameters, and
observe physiological and self-report data.
2) Music generator:Develop real-time algorithm that
modifies music parameters based on affect.
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Music AffectMapping
Music ParameterMapping
user study
music box
layering
complexity
instrument layering:frequency-domain densitycomplexity: time-domain density
Music Parameter Mapping
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Music AffectMapping
Music ParameterMapping
user study
music box
arousal
valence
arousal: reaction level to musicvalence: subject’s like/dislike of music
Music Affect Mapping
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
higharousal
like
Music Affect Mapping
lowarous
al
dislike
engagingannoying
soothingboring
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
higharousal
like
The Challenge
lowarous
al
dislike
engagingannoying
soothingboring
Current State Goal State
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
The Challenge
engagingannoyingCurrent State Goal State
engagingboring
engagingsoothing
soothingannoying
soothingboring
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Changing Music Affect
engagingannoyingengagingboring
engagingsoothing
soothingannoying
soothingboring
MusicParameters Probability
?f-1, t-1
f-1, t+1
f+1, t-1
f+1, t+1
Start State End State
• Given set changes in parameters, what are the trends for motion in
affective space?• How do those trends change depending on the initial affective state?
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Listening Experiment
Preparation of Music Clips:
• Five pieces with constant tempo were composed, ranging from jazz, rock to electronic music.
• For each piece, looping audio segments were produced to reflect the layer/complexity map.
• Segments were arranged into a 4x4 matrix for each piece.• For each piece, four clips were assembled by traversing the matrix
as follows:1. Increasing complexity2. Increasing instrument layering3. Decrease complexity4. Decrease instrument layering
((•Listen•))
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Listening Experiment
Preparation for Data Collection:
Physiological Data• Galvactivator set up to measure GSR• microphone connected to measure presence of foot-tapping
Self-report Data• dual 7-point scales were set up for subject’s affective response self-report• two sets were prepared for subjects to report their initial and final
reactions
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Listening Experiment
Conducting the Experiment:
• 8 participants: 6 male, 2 female• 20 audio clips (25-45 seconds each) were played for each subject• experiment lasted approximately 25 minutes
Physiological Data• GSR was sampled every second throughout entire listening, • BPM and velocity of foot-tapping was sampled every second, and presence
of foot-tapping was sampled every 2 seconds
Self-report Data• subject self-reported affective response twice during each clip (initial &
final)
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
90
92
94
96
98
100
102
104
1 24 47 70 93 116 139 162 185 208 231 254 277 300 323 346 369 392 415 438 461 484 507 530 553 576 599 622 645 668 691 714 737 760 783 806 829 852 875 898
46
48
50
52
54
56
58
60
1 39 77 115 153 191 229 267 305 343 381 419 457 495 533 571 609 647 685 723 761 799 837
Total Physiological Response - Subject 3
Total Physiological Response - Subject 4
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
Subject #2: physiological data
MUSIC CLIP #9: Increase Complexity
layering
complexity
0
2
4
6
8
10
12
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Subject #2: self-report data
like
annoying engaging
dislike
boring soothing
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
Subject #6: physiological data
MUSIC CLIP #9: Increase Complexity
layering
complexity
Subject #6: self-report data
like
annoying engaging
dislike
boring soothing
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
UNUSUAL EXAMPLE:
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
-6
-4
-2
0
2
4
6
-6 -4 -2 0 2 4 6
-6
-4
-2
0
2
4
6
-6 -4 -2 0 2 4 6
-6
-4
-2
0
2
4
6
-6 -4 -2 0 2 4 6
-6
-4
-2
0
2
4
6
-6 -4 -2 0 2 4 6
engaging
soothingboring
annoying
presence of foot-tapping
GSR
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
GSR fallingGSR rising
S1: engaging S2: soothing S3: boring S4: annoying
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Data Analysis
s1 s2 Sub totalPositive valence GSR UP 20 11 31
GSR DOWN 46 17 630.65 0.35 10.73 0.27 1
s3 s4 Sub totalNegative valence GSR UP 9 10 19
GSR DOWN 8 18 260.47 0.53 10.31 0.69 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1 2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1 2
GSR: up down
GSR: up down
S1: engaging S2: soothing S3: boring S4: annoying
S1
S2
S1
S2
S3
S4
S3
S4
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Algorithm Overview
A pair of state transition models was constructed based on physiological and self-report data:
The first model tries to detect the listener’s current affective state.
The second model chooses the direction most likely to induce the goal state.
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Probability Table for Detecting Affective State
s1 s2 Sub totalPositive valence GSR UP 20 11 31
GSR DOWN 46 17 630.65 0.35 10.73 0.27 1
s3 s4 Sub totalNegative valence GSR UP 9 10 19
GSR DOWN 8 18 260.47 0.53 10.31 0.69 1
S1: engaging S2: soothing S3: boring S4: annoying
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Affective State Transition Model
ACTION: Increase Complexity
layering
complexity
S1: engagingS2: soothingS3: boringS4: annoying
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
ACTION: Increase Layering
layering
complexity
S1: engagingS2: soothingS3: boringS4: annoying
Affective State Transition Model
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
ACTION: Decrease Complexity
layering
complexity
S1: engagingS2: soothingS3: boringS4: annoying
Affective State Transition Model
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
ACTION: Decrease Layering
layering
complexity
S1: engagingS2: soothingS3: boringS4: annoying
Affective State Transition Model
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Summary of State Transitions
0
0.1
0.2
0.3
0.4
0.5
0.6
1 2 3 4
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 3 4
Series1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 3 4
Series1
S1-engagingS4-annoying
S3-boring
S1 S2 S3 S40
0.1
0.2
0.3
0.4
0.5
1 2 3 4
Series1
S2-soothingS1 S2 S3 S4 S1 S2 S3 S4
S1 S2 S3 S4
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
Summary of State Transitions
S1: engagingS2: soothingS3: boringS4: annoying
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
System Diagram
Physiological data input
Control
Affective statedetection algorithm
Direction choosing algorithm
Change music parameter
Goal stateInitial state
Action
Music
Affect induced
Perception
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective ListenerConclusions
• changes in music parameters can be correlated to affective response to music• Markov chains are a useful tool for constructing a predictive listening system• specific observations about mapping layering/complexity to arousal/valence: • engaged listeners tend to stay engaged • annoyed listeners tend to stay annoyed • soothed listeners tend to stay soothed, but also easily bored or engaged • bored listeners tend to become interested by any change in parameters • annoyed listeners are more likely to be engaged if first induced to boredom
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective Listener
1. Existing Work2. Project Overview:
• Music Parameters
• Affective Signals3. Listening
Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective ListenerFuture Work
• improve accuracy of predictions by incorporating more user data• improve affective state predictions using additional affective signals• apply affect-parameter mapping to algorithmic composition• use machine learning to customize predictions to individual subject
Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05
The Affective ListenerReferences
Berlyne, D.E. (1974) Studies in New Experimental: Steps Towards an Objective Psychology of Aesthetic Appreciation. Halstead Press.
Clynes, M. (1977) Sentics. Anchor Press.Healey, J., Picard, R. and Dabek, F. (1998) ‘A new affect-perceiving interface and its
application to personalized music selection,’ Proc. of the Perceptual User Interfaces Workshop, 4-6.
Kim, S. and André, E. (2004) ‘Composing affective music with a generate and sense approach,’ Proc. of Flairs 2004. American Association for Artificial Intelligence.
Marrin, T. (2000) Inside the Conductor’s Jacket: Analysis, Interpretation and Musical Synthesis of Expressive Gesture. PhD thesis, MIT Media Lab, Cambridge, MA.
Meyer, L. (1956) Emotion and Meaning in Music. University of Chicago Press.North, A.C. and Hargreaves, D.J. (1997) ‘Liking, arousal potential and the emotions
expressed by music’, Pscyhomusicology 14:77-93.Ritossa, D. and Rickard, N. (2004) The relative utility of ‘pleasantness’ and ‘liking’
dimensions in predicting the emotions expressed by music. Psychology of Music. v.32(1):5-22
Russell, J.A. (1980) ‘A Circumplex Model of Affect’, Journal of Personality and Social Psychology 39: 1161-78.
Rutherford, J. and Wiggins, G.A. (2002) ‘An experiment in the automatic creation of music which has specific emotional content,’ Proc. for the 7th International Conference on music Perception and Cognition, Sydney, Australia.
Schubert, E. (1996) ‘Enjoyment of negative emotions in music: an associative network explanation’, Pscyhology of Music 24: 18-28.
Sloboda, J.A. and Juslin P.N. (2001) Music and Emotion: Theory and Research. Oxford University Press.