audio subtitling: measuring emotional arousal in users through … · 2018. 10. 10. · audio...
TRANSCRIPT
![Page 1: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/1.jpg)
Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures
Iturregui-Gallardo, G. – Matamala, A. – Soler-Vilageliu, O.
TransMedia Catalonia Research Group
Universitat Autònoma de Barcelona
Languages & the Media – Berlin – 3-5 October
![Page 2: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/2.jpg)
Table of Contents
- Audio subtitling (AST)
- Multilingual contents
- Effects: voice-over and dubbing
- The measurement of emotional arousal
- The experiments - Objectives: Experiment 1 and Experiment 2
- Stimuli
- Instruments
- Self-report
- Psychophysiology
- Participants: Experiment 1 and Experiment 2
- Results
- Conclusions
![Page 3: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/3.jpg)
Audio Subtitling (AST)
Where?
- Countries with bigger subtitling tradition (TV broadcasting)
- Countries with a dubbing tradition -> multilingual content
- Main language -> dubbed
- Secondary languages -> subtitled
- Also: other text on screen
![Page 4: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/4.jpg)
Multilingual Contents
Different languages
Different languages
Aural and visual
Aural or visual
Various channels
Single channel
Simultaneously
Simultaneously
Multiplicity of outputs
Single output
Original content
Adapted content
![Page 5: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/5.jpg)
Effects: Voice-over and Dubbing
Audio allows for different combinations -> different forms of AST
(Iturregui-Gallardo, 2018)
Dubbing effect
• Acted (prefabricated orality)
(Baños & Chaume, 2009; Sánchez Mompeán, 2016)
• Original is not heard
• Synchronised
Voice-over effect
• Read (less changes in prosodic features)
• AST superimposed (original can be heard)
• AST before/after (imperfect isochrony)
![Page 6: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/6.jpg)
The Measurement of Emotional Arousal
Media as entertainment Flow and presence Emotion
Discrete/Dimensional
approach Valence Arousal Psychophysiology
![Page 7: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/7.jpg)
The Experiments: Objectives
- Experiment 1:
Comparison of dubbing and voice-over effect in audio subtitled contents - Self-report measurements (T-SAM questionnaire)
- Psychophysiological measurements
- EDA
- HR
- Experiment 2:
Comparison of results from blind and partially sighted participants to sighted participants
- Self-report measurements (T-SAM questionnaire)
- Psychophysiological measurements
- EDA
- HR
![Page 8: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/8.jpg)
The Experiments: Stimuli
Series
Wojenne dziewczyny
[War Girls] (TVP1, 2017)
- Clips (3)
- ≅ 3 min scenes
- Unit
- Validated online
- Fear, anger, neutral
![Page 9: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/9.jpg)
Instruments: Self-report (T-SAM questionnaire)
(Iturregui-Gallardo & Méndez-Ulrich, forthcoming)
Valence
Arousal
![Page 10: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/10.jpg)
Instruments: Psychophysiology
- EDA (Electrodermal Activity)
- HR (Heart Rate)
![Page 11: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/11.jpg)
Experiment 1: Participants
42 participants:
- Blind (13) and partially sighted (29)
- 17 ♀
- 25 ♂
- Mean=38 years old
![Page 12: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/12.jpg)
Experiment 2: Participants
42 participants:
- Sighted participants
- 27 ♀
- 15 ♂
- Mean=30 years old
![Page 13: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/13.jpg)
Conclusions: Self-report Questionnaire (T-SAM)
- Emotions targeted were induced
- Results are consistent (Experiments 1 and 2) and differences are significant.
- Fear: low valence/high arousal
- Sadness: very low valence/medium arousal
- T-SAM proved effective (more research)
- Experiment 1: Conditions (effects) not significant, only in Valence for Fear
- Dubbing effect was rated with lower valence
![Page 14: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/14.jpg)
Results: Experiment 1 Self-report Instrument: T-SAM
[VALOR]
(0.61) 3.04 (0.67)
5.38 (0.62)
3.52 (0.62)
4.5 (0.88) 5.42 (0.52)
1
3
5
7
9
Sadness Fear Neutral
Valence ratings for emotion and effect
conditions
Dubbing VO
Emotion: (F(2, 120) = 9.356; p = <.001) || Effect: (F(1, 120) = 0.918; p = 0.34)
![Page 15: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/15.jpg)
Results: Experiment 1 Self-report Instrument: T-SAM
4.85 (0.57)
6.77 (0.63) 4.71 (0.78)
[VALOR]
(0.55)
[VALOR]
(0.50)
3.9 (0.72)
1
3
5
7
9
Sadness Fear Neutral
Arousal ratings for emotion and effect conditions
Dubbing VOEmotion: (F(2, 120) = 20.578; p =< .001) || Effect: (F(1, 120) = 0.183; p = 0.67)
![Page 16: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/16.jpg)
Post hoc test with Bonferroni correction: Valence
Emotion t pbonf
Sadness Neutral -3.75 <.001
Sadness Fear -0.02 1.000
Neutral Fear 3.73 <.001
Post hoc test with Bonferroni correction: Arousal
Emotion t pbonf
Sadness Neutral 1.48 0.427
Sadness Fear -4.67 <.001
Neutral Fear -6.15 <.001
![Page 17: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/17.jpg)
Results: Experiment 2 Self-report Instrument: T-SAM
3.6 (0.39) 3.86 (0.66)
6.34 (0.34)
1
2
3
4
5
6
7
8
9
Sadness Fear Neutral
Valence ratings for emotion conditions
(F (2, 84) = 68.576; p = <.000)
![Page 18: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/18.jpg)
Results: Experiment 2 Self-report Instrument: T-SAM
4.2 (0.71)
6.27 (0.56)
3.39 (0.46)
1
2
3
4
5
6
7
8
9
Sadness Fear Neutral
Arousal ratings for emotion conditions
(F (2, 82) = 59.289; p = <.000)
![Page 19: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/19.jpg)
Post hoc test with Bonferroni correction: Valence
Emotion t pbonf
Sadness Neutral -13.19 <.000
Sadness Fear -0.83 1.000
Neutral Fear -9.87 <.000
Post hoc test with Bonferroni correction: Arousal
Emotion t pbonf
Sadness Neutral 2.88 0.019
Sadness Fear -8.01 <.001
Neutral Fear -10.98 <.001
![Page 20: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/20.jpg)
Conclusions: Psychophysiology (EDA and HR)
Experiment 1:
- EDA results not significant
- HR results significant (Emotion and Effect) - Fear causes more activation
- Voice-over effect causes more heart acceleration
- Dubbing effect causes deceleration - Attention processes (Tremayne & Barry, 2001; Thomas et al., 2012; Graham, 1992)
![Page 21: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/21.jpg)
Results: Experiment 1 Psychophysiology
-0.26 (0.09)
-0.05 (0.10)
-0.11 (0.10)
-0.2 (0.11)
-0.1 (0.11)
-0.22 (0.10)
-0,3
-0,25
-0,2
-0,15
-0,1
-0,05
0
Sadness Fear Neutral
EDA values for emotion and effect conditions
Dubbing Voice-over
Emotion: (F(2, 93) = 1.084; p = 0.34) || Effect: (F(1, 93) = 0.15; p = 0.7)
![Page 22: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/22.jpg)
Results: Experiment 1 Psychophysiology
-3.22 (1.41)
-1.41 (1.41)
-3.09 (1.45)
-3.14 (1.45)
2.76 (1.45)
0.43 (1.45)
-4-3-2-101234
Sadness Fear Neutral
Bpm
HR values for emotion and effect conditions
Dubbing Voice-over
Emotion: (F (2, 98) = 3,62; p = 0.03) || Effect: (F (1, 98) = 4.862; p = 0.03)
![Page 23: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/23.jpg)
Post hoc test with Bonferroni correction: EDA
Emotion t pbonf
Sadness Neutral -0.54 1.000
Sadness Fear -1.45 0.372
Neutral Fear -0.92 1.000
Post hoc test with Bonferroni correction: HR
Emotion t pbonf
Sadness Neutral -1.28 0.606
Sadness Fear -2.69 0.025
Neutral Fear -1.38 0.506
![Page 24: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/24.jpg)
Conclusions: Psychophysiology (EDA and HR)
Experiment 2:
- EDA not significant
- HR not significant
- Trend: higher activation for Fear - Relates to Experiment 1 results
![Page 25: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/25.jpg)
Results: Experiment 2 Psychophysiology
-1.19 (0.77) -1.07 (0.77)
-0.98 (0.77)
-1,4
-1,2
-1
-0,8
-0,6
-0,4
-0,2
0
Sadness Fear Neutral
ED
A (
µS)
EDA values for emotion conditions
(F(2,105) = 0.02; p = 0.981)
![Page 26: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/26.jpg)
Results: Experiment 2 Psychophysiology
-4.65 (1.40)
-2.59 (1.40)
-5.02 (1.42) -6
-5
-4
-3
-2
-1
0
Sadness Fear Neutral
HR (
bpm
)
HR values for emotion conditions
(F(2, 92) = 0.87; p = 0.423)
![Page 27: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/27.jpg)
Post hoc test with Bonferroni correction: EDA
Emotion t pbonf
Sadness Neutral -0.19 1.000
Sadness Fear -0.11 1.000
Neutral Fear 0.08 1.000
Post hoc test with Bonferroni correction: HR
Emotion t pbonf
Sadness Neutral -0.19 1.000
Sadness Fear -1.04 0.903
Neutral Fear -1.22 0.679
![Page 28: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/28.jpg)
References
Baños, R., & Chaume, F. (2009). Prefabricated Orality. A Challenge in Audiovisual Translation. In M. G. Marrano, G. Nadini, & C. Rundle (Eds.), inTRAlinea special issue: The Translation of Dialects in Multimedia.
Braun, S., & Orero, P. (2010). Audio description with audio subtitling – an emergent modality of audiovisual localisation. Perspectives: Studies in Translatology, 18(3), 173–188. doi: 10.1080/0907676X.2010.485687
Graham, F. K. (1992). Attention: The Heartbeat, the Blink, and the Brain. In B. A. Campbell, H. Hayne, & R. Richardson (Eds.), Attention and Information Processing in Infants and Adults. New York: Psychology Press.
Iturregui-Gallardo, G. (2018). Rendering Multilingualism through Audio Subtitles: Shaping a Categorisation for Aural Strategies. International Journal of Multilingualism. doi: 10.1080/14790718.2018.1523173
Iturregui-Gallardo, G. & Méndez- Ulrich, J. L. (forthcoming). Towards the Creation of a Tactile Version of the Self-Assessment Manikin (T-SAM) for the Emotional Assessment of Visually Impaired People. International Journal of Disability, Development and Education.
![Page 29: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/29.jpg)
References
Petrie Thomas, J. H., Whitfield, M. F., Oberlander, T. F., Synnes, A. R., & Grunau, R. E. (2012). Focused attention, heart rate deceleration, and cognitive development in preterm and full-term infants. Developmental Psychobiology, 54(4), 383–400. https://doi.org/10.1002/dev.20597
Reviers, N., & Remael, A. (2015). Recreating multimodal cohesion in audio description: A case study of audio subtitling in Dutch multilingual films. New Voices in Translation Studies, 13(1), 50–78.
Sánchez Mompeán, S. (2016). “It’s not what they said; it’s how they said it”: A corpus-based study on the translation of intonation for dubbing. In A. M. Rojo López & N. Campos Plaza (Eds.), Interdisciplinarity in Translation Studies. Theoretical Models, Creative Approaches and Applied Methods. Bern, Berlin, Bruxelles, Frankfurt am Main, New York, Oxford, Wien: Peter Lang.
Tremayne, P., & Barry, R. J. (2001). Elite pistol shooters: physiological patterning of best vs. worst shots. International Journal of Psychophysiology, 41(1), 19–29. https://doi.org/10.1016/S0167-8760(00)00175-6
![Page 30: Audio Subtitling: Measuring Emotional Arousal in Users Through … · 2018. 10. 10. · Audio Subtitling: Measuring Emotional Arousal in Users Through Psychophysiological Measures](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a95ed30849614fbf0745bb/html5/thumbnails/30.jpg)
Disclaimer
TransMedia Catalonia is a research group funded by the Secretariat for Universities and Research (Catalan Government), reference code 2017SGR113.
The project NEA (Nuevos Enfoques sobre Accesibilidad/New Approaches to Accessibility) has received funding from the Ministry of Economy, Industry and Competitiveness (Spanish Government), grant agreement No FF12015-64038-P, MINECO/FEDER.
Iturregui-Gallardo is an FI grantholder awarded by the Agency for Management of University and Research Grants of the Catalan Government, reference code 2016FI_B 00012.
Texts, marks, logos, names, graphics, images, photographs, illustrations, artwork, audio clips, video clips, and software copyrighted by their respective owners are used on these slides for personal, educational and non-commercial purposes only. Use of any copyrighted material is not authorized without the written consent of the copyright holder. Every effort has been made to respect the copyrights of other parties. If you believe that your copyright has been misused, please direct your correspondence to: [email protected] stating your position and we shall endeavour to correct any misuse as early as possible.
This document and its contents reflect the views only of the authors. TransMedia Catalonia and the Ministry of Economy, Industry and Competitiveness cannot be held responsible for any use which may be made of the information contained therein.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.