story reaction structures to emotion detection
DESCRIPTION
Talk presented at ACM MULTIMEDIA, SRMC WORKSHOP.New York, USA, 2004TRANSCRIPT
Nelson Zagalo, Vasco Branco, Anthony Barker
University of Aveiro, Portugal
Story Reaction Structures to Story Reaction Structures to Emotion DetectionEmotion Detection
ACM MULTIMEDIA WORKSHOP SRMC, 2004
IntroductionIntroduction
We’ll present story structures that can be elicited by emotion recognition in storytelling, in order to maximize user emotions and therefore the signification experience.
We feel emotional when we understand story events as important to us.
Emotions are responsible for our cognition capacity.
1/20
Emotion in StorytellingEmotion in StorytellingOral and Cinematic process
Affective Storytelling Process 2/20
Style and RepresentationStyle and Representation
Story
StyleRepresentation
Nuclei
Catalysers
ORAL
Tone of VoiceFacial Expression
PostureGesture
Eye Direction
CINEMATIC
FramingEditingMusic Sound
Lighting
ORAL / CINEMATIC
3/20
Cinematic and OralCinematic and Oral
Representation
Style
Cinematic
Oral
Static
Dynamic
Storyteller reinforces or subtracts according to the story-listeners
expression of emotions, using new elements from Total Story
Representation
Style
Static and Dynamic
Viewer Emotions
4/20
Viewer EmotionsViewer Emotions
Two axes dimensions, one for intensity (arousal) as the “tension-sleep” axis and the other for quality (valence) as the “pleasant - unpleasant” axis.
The points of convergence between Intensity and Quality axes, give us pre-defined labels that identify the emotional-state in a categorical point-of-view.
Four dramatic clusters. General emotional areas that influence the story viewer - anxiety, happiness, sadness and relax
Intensity
Quality
Coding Emotions
Story Viewer Emotions
5/20
Dynamic CinemaDynamic Cinema
Applying the model of the story viewer emotions to cinematic elements in search for a dynamic development.
Emotional Film Structures
6/20
Emotion FlowEmotion Flow
Director’s Version
over-stimulation
under-stimulation
(A) Overall Story Emotion(B) Intermediate Tension sequences
Viewer’s Will
traditional dramatic arcs
Director and Viewer Control
In the Viewer’s Will, the unbalancing of the tension sequences, results in an (A) overall equal or below to any (B) intermediate sequence.
7/20
We’ll analyse here the small unit intermediate tension arc of the Director’s version emotional flow.
Intermediate Tension Arc
Intermediate tension arc unitEmotion FlowEmotion Flow
8/20
Reaction StructuresReaction Structures
The application of the four dramatic emotion clusters to the intermediate tension arc, give us the arc of emotions, the reaction structures guider.
viewer arc of emotions
Arc of emotions
Story Viewer Emotions Intermediate tension arc
=+
9/20
Reaction StructuresReaction Structures
Framing Editing Sound Lighting Music Catalysers
Intensity(variation)
Strong Close-ups
Reduce frame
duration
Fast tempos, crescendo
Darker Explicit
Quality(adding)
Hardly distinguishing.
Shade near night light.
Dissonant chords. Very
high/very low pitches
Anxiety cluster: anger, fear and
disgust representations
Having detected an emotional state in the viewer with low levels of arousal, in a sequence of rising tension, the story would react to get the intended emotional effect from the user
ANXIETY
10/20
Framing Editing Sound Lighting Music Catalysers
Intensity(variation)
Medium frames,
some close ups
Expand frame
duration
Reduce tempo,
decrescendoGreyer Implicit
Quality(adding)
Sufficiently distinguishing.
Cloudy day light.
Consonant chords. Low
pitches
Anxiety cluster: sadness, misery and depression representations
Reaction StructuresReaction Structures
SADNESS
Sadness is a negative low tension emotion, so rhythm needs to decelerate, become greyer and treat emotion catalysers implicitly.
11/20
Framing Editing Sound Lighting Music Catalysers
Intensity(variation)
Medium frames,
some Close-ups
Reduce frame
durationFast tempos Brighter Explicit
Quality(adding)
Very Clear distinguishing.
Sunny day light.
Melodic. High and medium pitches
Anxiety cluster: excitement, delight or euphoria
representations
Reaction StructuresReaction Structures
HAPPINESS
Happiness is a positive high tension emotion, rhythm accelerates, becomes brighter, crystal clear picture definition, treat emotion catalysers explicitly.
12/20
Framing Editing Sound Lighting Music Catalysers
Intensity(variation)
Medium and general frames
Extended frame
duration. Very Linear
Slow tempos, decrescendo
Mellower Implicit
Quality(adding)
Sufficiently distinguishing.
Sunny end afternoon
Melodic. Low and medium pitches
Anxiety cluster: calmness or
repletion representations
Reaction StructuresReaction Structures
RELAX
Relax is an optimal state formed by low tension and positive emotion, it tends to balance everything.
13/20
Case StudyCase StudySilence of the Lambs (1991)
We’ll present a theoretical usage of how reaction structures could operate, analysing the second, 10 minutes period of the movie.
Sequence: First visit Clarisse pays to Hannibal Lecter in prison
The sequence wil be presented divided in four phases: Entrance, Confrontation, Retreat, Flashback
We can’t show the 10 minutes, so we’ll analyse only some frames 14/20
Silence of the LambsSilence of the Lambs(A) Entrance
Entrance of Clarisse into the high-security chamber
Night light, music starts slowly in a crescendo
Anticipation: Medium disgust catalyser
System could analyse emotional effects on the viewer to this disgust catalyser, and learn what levels the viewer is prepared to support
Editing accelerates, framing gives Clarisse regular close-ups
Sounds dissonant with higher pitches
The lighting is dark, difficulty in identifying prisoners’ faces.
“I can smell your cunt”night lightentrance
Tension
15/20
Silence of the LambsSilence of the Lambs(B) Confrontation – part 1
strong close up medium close upconfrontation
Confrontation starts with viewer arousal already activated
Confrontation is pure anxiety, rises the whole story arc.
Anxiety is triumphantly carried off by very strong actors’ performances
Framing is anxiety effective with extreme close-ups
Lighting could become darker
Editing rhythm could be accelerated
Music could be added to help reinforcing anxiety.
Anxiety
16/20
Silence of the LambsSilence of the Lambs(B) Confrontation – part 2
implicit explicit
Strong emotional disgust catalyser
Prisoner Miggs masturbates and then throws his semen at Clarisse’s face
To Intensify – system could show explicit masturbation
To smooth – system could use suggestion instead of explicitly show semen dripping or even could eliminate the scene from the representation.
Anxiety (false ending iniciates sadness cluster)
17/20
Silence of the LambsSilence of the Lambs(C) Retreat
Retreat sequence is the declared arc peak and it’s conducted through sadness cluster.
Clarisse is presented crying, the day light is grey and the melodic low pitched score helps the rising note of sadness.
System, could zoom to tears, add some rain or expand sequence duration with music
sadness grey light
Sadness
18/20
Silence of the LambsSilence of the Lambs(D) Flashback
Flashback constitutes the relaxation structure.
Flashback changes from a grey day in Foster’s present to a sunny end of day in her past.
Action rendered in slow motion and music is melodic.
Calmness and familiarity enters the viewer’s body and mind, relaxing and restoring composure
mellower light familiarity
Relax
19/20
We tried to develop a theoretical story computer system based in oral and cinematic storytelling able to react to emotion detection.
We developed a kind of a dynamic cinema, supporting undirected interactivity in multilinear scenarios.
The analysis we have presented here must be seen as a draft for the next stage of research in the field
Next stage – reactions to emotion detection within VR storytelling.
For that we’re starting to work, inside an european project, in developing an interactive storytelling taxonomy of emotions, to support the development of story reactions within virtual environments.
We are convinced that story reactions to emotions detected will become a field of great research in interactive storytelling
Conclusions & PerspectivesConclusions & Perspectives
20/20
Story Reaction Structures to Emotion DetectionStory Reaction Structures to Emotion Detection Nelson Zagalo1, Anthony Barker3, Vasco Branco2
1,2 Dept. of Communication and Art, University of Aveiro3810-193 Aveiro, Portugal
[email protected], [email protected]
3 Dept. of Languages and Cultures, University of Aveiro3810-193 Aveiro, Portugal
ACM WORKSHOP SRMC, 2004