simon et al dynamic clips 060914 - université de...
TRANSCRIPT
-
Dynamic Clips 1
Running head: DYNAMIC EMOTIONAL FACIAL EXPRESSIONS
Dynamic Clips of Emotional and Pain Facial Expressions - Validation of a Stimulus Set -
Daniela Simona,e, Kenneth D. Craigb, Frederic Gosselinc, Pascal Belind, and Pierre Rainvillee*
a) Department of Clinical Psychology, Humboldt University of Berlin, Germany
b) Department of Psychology, University of British Columbia, Vancouver / Canada
c) Departement de Psychologie, Université de Montréal, Montréal / Canada
d) Department of Psychology, University of Glasgow, Glasgow / UK
e) Departement de Stomatologie, Université de Montréal, Montréal / Canada
* Corresponding author : Pierre Rainville, Ph.D., Professeur agrégé
Département de stomatologie
Faculté de médecine dentaire
Pav. Paul-G-Desmarais (local 2123)
Université de Montréal
CP 6128, Succursale Centre-ville
Montréal, QC H3C3J7
Canada
Fax: +001 / 514 / 343-2111
Phone: +001 / 514 / 343-6111 ext. 3935
E-mail: [email protected]
-
Dynamic Clips 2
Acknowledgements
This research was supported by grants and salary support from the Natural Sciences and
Engineering Research Council (NSERC) of Canada, the Fonds Québécois de la recherche sur la
nature et les technologies (FQRNT), the Canadian Fund for innovation (CFI), the Fonds de la
recherche en santé du Québec (FRSQ), and a scholarship to DS from the German Academic
Research Exchange Service (DAAD). We thank Julie Senécal, David Wong, and Annie L. for their
help in the realisation of this study.
-
Dynamic Clips 3
Abstract
Emotional facial expressions have been intensively investigated using photographs. However,
because facial expressions are highly dynamic, there is a need for stimuli that include this important
temporal dimension. Moreover, the facial expression of pain is not included in any available
stimulus set. Here, we describe 1-sec film clips recorded in 8 actors who displayed facial
expressions of six emotions (anger, disgust, fear, happiness, sadness and surprise), or pain,
compared to neutral expression. Prototypicality and distinctness of all stimuli were confirmed by
two trained judges using the Facial Action Coding System. Discrimination by naïve volunteers and
subjective evaluations of the clips on expression intensity, arousal and valence, confirmed the
validity and specificity of the 7 expressions. This set of dynamic facial expressions provides unique
material to further explore the perception of facial expressions of emotion and pain and their role in
the regulation of social behaviour and cognition.
-
Dynamic Clips 4
Dynamic Clips of Emotional and Pain Facial Expressions - Validation of a Stimulus Set -
The importance of the face in interpersonal communication is widely recognized in
anthropology, psychology and the social sciences. Facial expressions are powerful social signals,
which impart information about a person’s emotional state and might provide survival value by
signalling potential threats (Darwin, 1872). Nowadays, scholars agree on the account of at least
‘minimal universality’ of facial expressions, which predicts a certain amount of cross-cultural
similarity in expressing and recognizing emotional faces, without necessarily postulating innateness
(e.g. Russell & Fernandez-Dols, 1997). Although facial expressions in naturalistic contexts might
contain a blend of different emotions, considerable evidence has been provided for the existence of
at least six prototypical basic emotions (anger, disgust, fear, sadness, happiness and surprise; see
Table 1) (Ekman, Friesen, & Elsworth, 1982). According to Ekman & Friesen (1975) each basic
emotion corresponds to a specific activation pattern of facial muscles (action units = AU),
measurable by the Facial Action Coding System (FACS) (Ekman & Friesen, 1978). The facial
expressions of the six basic emotions have been subject to research in a variety of disciplines
ranging from cognitive, social, and clinical science to computer science. Research works
encompassing topics from automatic facial feature tracking (e.g. Lievin & Luthon, 2004) to neural
correlates of empathic responses to emotional facial expressions (e.g. Carr, Iacoboni, Dubeau,
Mazziotta, & Lenzi, 2003) reflect the breadth of scientific interest in emotional facial expressions
(EFEs).
In parallel to those developments, pain research has provided evidence that the facial
expression of pain is unique and distinct. According to the International Association for the Study
of Pain, pain is “an unpleasant sensory and emotional experience associated with actual or potential
tissue damage, or described in terms of such damage” (see http://www.iasp-pain.org/terms-p.html).
-
Dynamic Clips 5
Modern conceptions of pain typically separate secondary emotional responses associated with the
appraisal of the cause and future consequences of pain from the immediate affective response
(including facial expression) intrinsic to acute pain experiences (e.g. Price, 2000). Moreover,
evolutionary theories propose that the facial expression of pain following acute injury might play a
role in survival by alarming onlookers in situations of direct threat and/or by eliciting empathy and
solicitous behavior towards the individual experiencing pain (Williams, 2002). While fearful and
angry faces may rather signal an impending danger, pain faces indicate an actual noxious event
immediately affecting the body’s integrity. Therefore, the facial expression of pain might arguably
be more important for the survival of the species than other basic facial expressions (Williams,
2002). However, comparatively little work has been done on pain expression and this research has
had relatively little impact on emotion research.
The facial expression of pain has been investigated in the past decades especially in the
context of pain assessment (Craig, Prkachin, & Grunau, 2001) and has recently become an area of
interest in neuroscientific research (e.g. Botvinick et al., 2005; Simon, Craig, Miltner, & Rainville,
in press). Several studies using FACS reliably identified the co-occurrence of certain facial actions
in a pain face across various clinical pain conditions (e.g. LaChapelle, Hadjistavropoulos, & Craig,
1999; Prkachin, 1992) (see Table 1). Although research is this field is still scant, this facial pattern
has been described to be unique and distinct from the facial expressions of basic emotions
(Kappesser & Williams, 2002; Williams, 2002). Part of the reason pain has not been investigated as
much as other basic expression maybe related to the lack of appropriate stimuli set to study the
facial expression of pain. Furthermore, the integration of pain into emotion research requires that
specific stimuli be developed including both pain and basic emotion expressions. The stimulus set
described in the present report may provide a tool to establish bridges between those currently
independent domains.
-
Dynamic Clips 6
Most studies on facial expressions have been conducted using static facial displays; for
example, the photographs set provided by Ekman & Friesen (1976). However, facial expressions
occurring in real-life interpersonal communication are highly dynamic. Moreover, the identification
of EFEs seems to be partly influenced by motion (Harwood, Hall, & Shinkfield, 1999; O'Toole,
Roark, & Abdi, 2002; Roark, Barrett, Spence, Abdi, & O'Toole, 2003). A recent study showed that
the dynamic properties of facial expressions significantly improve discrimination (Ambadar,
Schooler, & Cohn, 2005). Moreover, brain regions implicated in processing facial affect, including
the amygdala and fusiform gyrus, show greater responses to dynamic versus static emotional
expressions; and activity in the superior temporal sulcus discriminate dynamic faces expressing
emotions from dynamic face not expressing emotion (LaBar, Crupain, Voyvodic, & McCarthy,
2003; see also Haxby, Hoffman & Gobbini et al., 2002). Hence, to improve the ecological validity
of the investigation of the social communicative functions of emotional facial expressions, there is a
need for dynamic stimuli. To date no standardized and validated set of such stimuli containing all
basic emotions has been published. Studies investigating responses to dynamic stimuli of EFEs
either used computer-based morphs (e.g. LaBar et al., 2003; Sato, Kochiyama, Yoshikawa, Naito, &
Matsumura, 2004) or sets of movie clips comprising some but not all basic emotions (Kilts, Egan,
Gideon, Ely, & Hoffman, 2003; Wicker et al., 2003). Furthermore, the facial expression of pain is
not included in any validated set of static or dynamic facial expression stimuli. Recent studies
examining brain responses to the observation of pain expression used movie clips showing patients
in pain (e.g. Botvinick et al., 2005). It is not clear whether the stimuli developed in this study may
be contaminated by contextual information and motion resulting in non-uniform visual complexity.
More importantly, this stimulus set does not allow for the study of the similarity and differences
between responses to pain expression and responses to the expression of basic emotions. This
-
Dynamic Clips 7
constitutes a serious limitation, especially for neuroscientific research on the facial expression of
pain and emotions.
The aim of the present study was to produce and validate a comprehensive, standardized set
of dynamic clips of facial expressions of the six basic emotions for future research on EFEs. In
addition, expressions of acute pain were included to assess further the specificity of this expression
compared to basic emotions and to provide material to establish additional bridges between emotion
and pain research.
Method
Participants
Fifteen healthy volunteers (11 males and 4 females, mean age: 24.1 ± 3.4) were recruited on
the campus of the University of Montreal to participate in a study on the perception of facial
expression. Each participant provided informed consent and received monetary compensation for
their participation (25 CA$). All procedures were approved by the local Ethics Committee.
Stimuli and Stimuli Production
Drama students of different theatrical schools in Montréal were hired for the production of
the stimuli. Initially 11 actors took part in the recording but only the best 8 (4 males and 4 females;
mean age: 24.4 ± 7.5 y.o.) were used to create this set of stimuli. They were selected based on the
results of the observers’ judgements as those who produced the most unambiguous facial
expressions. All actors provided written informed consent, transferring the copyright of the
produced material to the research group. Their participation was compensated with 100 CA$. The
recording sessions took place with a seamless chroma-key blue paper screen background and two
diffuse tungsten light sources. In order to minimize contamination of the stimuli by special facial
-
Dynamic Clips 8
cues, selected actors did not have facial piercing, moustache or beard, and did not wear earrings and
make-up during the recording. If necessary they were asked to put on a hairnet. The actors sat
comfortably about 1.5 m away from the camera lens.
Actors were required to express the six basic emotions (happiness, disgust, fear, anger,
sadness and surprise) in three intensities (moderate, strong, extreme). Actors also generated facial
displays expressing acute pain experiences in four intensities (mild, moderate, strong, extreme), and
produced a neutral face as a control. Actors were asked to produce each expression in about 1 sec
starting with a neutral face and ending at the peak of the expression. The actor’s performance was
monitored online by the filming team positioned outside of the cabin.
Prior to the recordings, the actors were given an instruction guide encouraging them to
imagine personal situations to evoke each emotion or pain. In addition, descriptions of short scenes
were provided as examples to support vivid imaginations of the different emotional states they were
asked to display (see appendix). The large majority of the clips (about 90%) were produced using
this method based on mental imagery of an emotional situation. If necessary, the actors were also
shown photographs of prototypical emotional facial expressions. If the filming team still saw
discrepancies between depicted and expected facial expressions, discrete muscles were trained as
described by Ekman & Friesen (1975) and Ekman, Friesen & Hager (2002) (Fig. 1). The
performance was repeated until the filming team was convinced that the criteria described by
Ekman & Friesen (1975) were met for each facial expression of emotions and by Williams (2002)
for pain expression. At least 2-3 good streams were recorded for each level and in each condition.
Given the considerable volume of film clips produced with each actor, a thorough FACS analysis
could not be performed online and the filming team primarily relied on their immediate recognition
of the target emotions. However, their decision to include a clip in the set was also informed by a
list of easily detectable AUs for the online identification of ‘expected emotion expressions’ (Happy:
-
Dynamic Clips 9
AU 6/12; Anger: AU 4/7/23; Disgust: AU 10; Fear: AU 1/4/5/25; Surprise: AU 1/2/26; Sadness:
AU 1/4/15 and Pain: AU 4/6/7/10). The movie streams were captured in colour by a Canon XL1S
video camera and directly transferred and saved to a Pentium computer for off-line editing using
Adobe Premiere 6.5. Each recording session lasted about 1.5 hours. (Figure 1 about here)
In this report we selected stimuli of the “strong” intensity level, as the emotions displayed
appeared to be unambiguous and rarely judged to be exaggerated at that level. In a pre-rating
session three naïve judges (1 male, mean age: 25.3 ± 0.6) independently selected the three best
sequences per actor for each face category. Using Adobe Premiere 6.5 and QuickTime Pro, the clips
were cut backward from the peak of the expression in order to assure that the peak was captured
within 1 sec (Image size: 720 x 480 pixel, Frame rate= 29.97 frames per second; mean inter-ocular
distance = 100.6 ± 4.5 pixels). Those features were imposed to facilitate their use in experimental
studies in which the duration of the stimuli may be critical (e.g. event-related activation in brain
imaging experiments). Admittedly, the disadvantage of this choice is that the 1-sec duration does
not capture the full extent of the dynamical expression. More specifically, in some clips, the onset
of the expression may not match precisely with the onset of the clip (i.e. in some clips the onset to
apex slightly exceeded 1 sec), and the offset of the expression was excluded from the clips.
Validation
Subjective ratings. The rating session took place in a meeting room at the University of
Montreal. Participants viewed each film clip twice and were asked to judge it on three different
scales. After the first presentation participants evaluated valence and arousal on a 9 - point Likert
scale. Participants were instructed to “rate how the person in the clip might feel: with respect to
valence: - 4 = clearly unpleasant to + 4 clearly pleasant; and arousal: -4 = highly relaxed to +4 =
high level of arousal”. Information about valence and arousal was included to allow for the
-
Dynamic Clips 10
stimuli’s use in experimental studies inspired by the dimensional model of emotion (e.g. Lang,
Greenwald, Bradley & Hamm, 1993). Neuroimaging studies have shown that responses of some
brain areas are crucially influenced by valence and/or arousal of stimuli (e.g. the amygdala; see
Zald, 2003), underlining the importance of controlling for those dimensions in studies investigating
emotions. After the second presentation, participants rated each facial expression with respect to the
intensity of happiness, disgust, fear, anger, sadness, surprise and pain on a scale of 6 - point Likert
scale. Participants were instructed to “rate the intensity of each emotion in the clip from 0 = not at
all to 5 = the most intense possible”. Each clip was therefore rated on all emotional categories.
Based on the emotion intensity ratings, the clip with the lowest mean intensity on all non-target
emotions was selected for each actor and emotion. The final set comprised 64 one-second clips with
each of the 8 actors contributing one clip to each of the 8 conditions (Fig. 2). (Figure 2 about here)
Facial Action Coding System. FACS coding was performed on the 64 selected clips (Fig. 1).
This procedure offered the opportunity to compare the results of these facial expressions with the
prototypes reported in the literature (chapter 12, p. 174, Table 1 Ekman et al., 2002; Williams,
2002). All 64 clips were evaluated by two independent accredited coders (1 male and 1 female),
who were blind to the target expression in each clip. The FACS ratings comprise information about
the occurrence and intensity (a = trace of the action to e = maximum evidence) of each AU. In the
first stage of coding, the two coders independently identified the AU’s present in each clip and rated
their intensity. In the second stage, the differences in coding were reconciled across the two coders
to provide a single set of AU’s with their intensity for each clip. (Table 1 about here)
Data Analysis
Subjective ratings. Average ratings were first calculated across subjects for each clip and
each rating scale, and the mean and SEM ratings were then computed across clips within each
-
Dynamic Clips 11
expression condition. The frequency of correct classification was calculated for each expression
category based on the highest emotion intensity rating to provide an index of sensitivity [Hit rate =
% (# of correct classification) / (# of correct classification + # of “miss”)]. The frequency of correct
rejection of stimuli not in the target expression was also determined to provide an index of the
specificity of judgments for each expression category [= % (# of correct rejection) / (# of correct
rejection + # of “false alarms”)]. Additionally, the presence of non-target expression was assessed
by calculating a “mixed-emotion index” (mean of the ratings for the non-target expressions) for
each condition.
A cluster analysis was computed to insure that each clip was adequately recognized and that
clear boundaries existed between the different categories of facial expression (Squared Euclidean
distances; Ward-Method). The analysis was performed on the 64 clips using the 7 intensity ratings
(one for each possible expression) averaged across participants. Inter-judge reliability was
determined by computation of Cronbach’s Alpha. In order to determine whether a portrayed
emotion was rated higher on the corresponding scale than on the non-target scales, Fisher’s
protected least significance difference test was calculated for each face category. Subsequent
analyses included repeated measures analysis of variance (ANOVA) on the participant’s ratings
(intensity, mixed-emotion index, valence and arousal) with factors emotion (pain and six basic
emotions) and sex of actor (male / female). A Greenhouse-Geisser correction was used for
computation of the statistical results (p ≤ .05). Main effects of emotion underwent post hoc analysis
(multiple comparisons on eight emotional categories) using a Bonferroni correction (p ≤ .05/36 =
.0013).
Facial Action Coding System. The evaluation of the FACS coder was performed in two
successive stages. During the first stage both coders provided independent lists and intensity ratings
of the observed AUs. The frequency reliability was determined by 2x the number of agreements
-
Dynamic Clips 12
divided by the total number of AUs coded by both raters. The intensity reliability was determined
by 2x the number of intensity agreements ±1, divided by the total number of AUs on which the
coders agreed. During the second stage an agreed set was determined by reconciling the
disagreements. The criteria for the reconciliation are provided by the FACS manual (Ekman et al.,
2002, p. 88). The reported results refer to this agreed set.
The number of actors showing some involvement of an AU was determined for each
expression condition. Additionally, the mean number and % of target AUs observed in each
expression category was determined. All AUs that were activated at an intensity level of ‘b’
(indicating slight evidence of facial action) or more in at least 50% of the actors were considered to
be representative of the expression condition (note that level ‘a’ indicates only a trace). The lists of
observed AUs in each category were compared with those previously reported for prototypical
emotional facial expression or their major variants (Ekman et al., 1982). While a prototype refers to
a configuration of AUs commonly associated with a certain emotion, major variants constitute
partial expressions of a prototype (Smith & Scott, 1997). Moreover, in order to evaluate sex
differences in facial expression, repeated measurement ANOVAs with within-subjects factor ‘AU’
and between group-factor ‘sex of the actor’ were computed for each emotion (p ≤ .05/7 = .007).
Results
Subjective Ratings
Analysis of inter-rater reliability revealed a very high agreement of the 15 observers’ for
intensity, valence and arousal ratings (Cronbach’s Alpha = .97).
Intensity. The average ratings across participants for each portrayed expression on the
corresponding target scales are reported in Table 2. Pairwise comparisons revealed that each
emotional facial expression was perceived as more intense than neutral faces (all p’s < .001). The
-
Dynamic Clips 13
analysis of the mean intensity ratings of the target emotional category (see asterisks in Table 2) also
revealed a main effect of emotional expression (F (7, 98) = 58.20; p < .001; ε = .57). Facial
expressions of happiness and surprise were judged as more intense than sad faces (happiness Vs
sadness: p < .001; surprise Vs sadness: p < .001) while happy faces were perceived as more intense
than fear faces (p < .001). The remaining emotions did not significantly differ from each other.
Intensity ratings were not influenced by the sex of the observer (F (1, 13) = 1.24; p = .285) or of the
actor (F (1, 14) = 0.03; p = .864; ε = 1.00) (see Table 3). Moreover, each category of facial
expressions yielded significantly higher average ratings on the corresponding scale than to the 6
non-target scales (Fisher’s protected LSD; all p’s < .001; see Table 3). (Table 2 and Table 3 about
here)
Discrimination. The cluster analysis identified 8 different subgroups of clips corresponding
to each of the 8 expression conditions. According to the observed clustering, all clips were
adequately assigned to the target category (i.e. the distance between clips of the same target
emotion was always smaller than between clips of different target emotions). The analysis also
revealed second-order combinations of clusters for neutral and sadness, as well as pain and fear.
Pain and fear also showed some proximity to disgust as reflected by a third-order combination of
these clusters.
The sensitivity (Hit rate) and specificity (Correct rejection rate) indices confirmed that
participants clearly recognized the target emotion shown in the clips, and that the presence of non-
target expression was negligible (Table 2). Sensitivity was higher than 70% for all conditions (mean
sensitivity: 86%), well above the 14% chance rate. Specificity was above 94% for all expression
(mean specificity: 97%). However, the presence of non-target expression differed between
emotions as revealed by a main effect of emotional expression on the mixed-emotion index (F (7,
98) = 13.89; p < .001; ε = .44). Pairwise comparisons revealed that fear, pain and disgust faces
-
Dynamic Clips 14
showed slightly more traces of non-target emotions than happy, angry and neutral faces (all p’s ≤
.001). Fear clips additionally showed more traces of non-target emotions than sadness and surprise
clips (p’s ≤ .001). While the mixed-emotion index was not influenced by the sex of the observer, a
main effect of the sex of the actor was detected reflecting slightly higher reports of non-target
expression in female actors (F (1, 14) = 4.87; p = .044; ε = 1.00). Taken together, these findings
indicate that the expressions were easily discriminated and included only traces of non-target
expressions.
Valence. Analysis of the average valence scores clearly indicate that participants perceived
happy faces as “pleasant”, neutral and surprise faces as “neutral”, and the remaining emotional
faces (pain, disgust, sadness, fear, anger) as unpleasant (Table 3). Thus, the analysis of these ratings
resulted in a main effect of emotional expression (F (7, 98) = 216.40; p < .001; ε = .47). As
demonstrated by pairwise comparisons, participants judged happy faces as significantly more
pleasant than all other face categories (all p’s < .001). Furthermore, all facial expressions that had
been rated as “unpleasant” were significantly more unpleasant than both neutral and surprised faces
(all p’s < .001). Additionally, within the group of unpleasant facial displays, pain faces were rated
significantly higher on unpleasantness than fear, sad and anger faces (pain Vs fear: p < .001; pain
Vs sadness: p = .001; pain Vs anger: p < .001). Furthermore, disgust clips also yielded higher
unpleasantness ratings than anger clips (p < .001).While valence ratings were not influenced by the
observer’s sex, an interaction between emotional facial expression and sex of the actor was found (F
(7, 98) = 5.55; p < .001; ε = .64). This was due to higher levels of pleasantness to female than male
faces expressing surprise (p = .001).
Arousal. The statistical analysis of the average arousal ratings for all emotional facial
expressions revealed a main effect of emotional expression (F (7, 98) = 24.91; p < .001; ε = .39).
Pairwise comparisons showed that participants perceived neutral and sad faces as significantly less
-
Dynamic Clips 15
arousing than pain, surprise, fear, anger, as well as disgust faces (all p’s < .001), while sadness did
not differ significantly from neutral. Furthermore, pain clips were rated as the most arousing and
differed significantly from fear, disgust and happy faces (p’s ≤ .001). While arousal ratings were not
influenced by the observer’s sex, a main effect of the actor’s sex was observed (F (1, 14) = 6.32; p =
.025; ε = 1.00). Moreover, a trend of an interaction between the actor’s sex and emotion was
detected (F (7, 98) = 2.31; p = .065; ε = .60). Post-hoc comparisons reveal higher levels of arousal
to female than male faces expressing happiness (t (14) = 3.21; p = .006) (see Table 3).
FACS
The reliability of the FACS coding of the 64 clips across the two coders was excellent
(frequency reliability = 82.3%; intensity reliabilities = 90.6%). Table 1 lists all AUs observed in the
different condition and shows the overlap with prototypical AUs, as reported in the FACS literature.
Considering the FACS investigators guide (chapter 12, p.174, Table, Ekman et al., 2002), the
detected activation patterns of AUs for each emotion generally corresponded well with the
prototypes or major variants of the intended emotional facial expressions. The selected stimuli
always included several of the target AUs in various combinations, and only few non-target AUs
(Table 1). More specifically, all happiness expressions included the target AUs 6 and 12; anger
expressions included 8 out of 10 target AUs; disgust expression included all 7 target AUs; fear
expressions included 7 out of 8 target AUs; surprise expressions included all 5 target AUs; sadness
expressions included 7 out of 8 target AUs; and pain expressions included 9 out of 10 target AUs.
Moreover, few non-target AUs were observed. For example, anger stimuli included an overall
average of 3.38 target AUs and an only 0.38 non-target AUs (see # of target AU in Table 1). This
meant that among all the AUs observed in the anger clips, 89.6% were target AUs. Happiness and
-
Dynamic Clips 16
disgust showed the lowest relative rates of target AUs (see Table 1) indicating the presence of
additional non-target AU mainly in those conditions.
There was no main effect of the sex of the actor, and no interaction between the emotion
condition and the sex of the actor on FACS data, suggesting that both males and females
contributed to the observed patterns of facial expression across conditions.
Discussion
The aim of the present study was to validate a set of dynamic stimuli of prototypical facial
expressions of basic emotions and pain. The film clips were comprehensively validated by FACS
coding and subjective ratings. The facial expressions of the different basic emotions and pain
corresponded well with the prototypes and major variants reported in the literature (chapter 12,
p.174, Table 1, Ekman et al., 2002; Williams, 2002). Additionally, the results of the cluster analysis
performed on the observer’s ratings of emotions and pain demonstrated that the 64 film clips were
correctly classified (100% accuracy) into 8 different groups of EFEs, in accordance with the
intended target emotions. Hence, in line with the literature, specificity and distinctness of the
investigated facial expressions was confirmed by both FACS coding and subjective ratings of naïve
observers (Kappesser & Williams, 2002; Smith & Scott, 1997).
Taking the magnitude of the subjective evaluations into account, the EFEs selected here
were perceived as expressing a comparable level of intensity (range: 2.40 to 3.34/5). This is
consistent with the instructions given to actors and the selection of clips to obtain a final set of
stimuli displaying each target expression at a consistent intensity level across conditions. However,
differences were observed with surprise rated as slightly more intense than sad faces, and happy
faces rated as slightly more intense than sad and fear faces.
-
Dynamic Clips 17
FACS prototypes and the dynamical expression of emotions
Results clearly demonstrated an excellent correspondence with FACS prototypes as shown
in Table 1. A detailed comparison of the FACS responses observed here was further performed with
those previously reported in a reference study using a similar methodology with actors performing
short scenes with the instruction to feel or not feel the emotions expressed (Gosselin et al., 1995). In
general, our stimuli were found to be highly consistent with the felt condition reported in that study.
The concordance with FACS prototypes was also significantly better in the present set of stimuli.
More specifically, Gosselin et al. (1995) reports that felt happiness is characterized by target
AUs 12 (lip corner puller; mean ± SEM rate: 0.92 ±0.16) and AUs 6 (cheek raiser; 0.94 ±0.08) with
the latter reduced to a rate of 0.77 ± 0.30 in the unfelt condition. Both AUs occurred in 100% of the
happiness clips selected in our stimulus set.
Similarly, the anger stimuli proposed here showed an overall higher rate of target AUs and a
lower rate of non-target AUs suggesting better sensitivity and specificity than those of Gosselin et
al. (1995). Only three target AUs reached a rate of 0.25 in that study, compared to six in the present
stimulus set (rate >2/8). In their unfelt condition, Gosselin et al. further reported a decreased
occurrence of target AU26 (jaw drop; unfelt: 0.24 ±0.20; felt: 0.49 ±0.30), an AU that we did not
observe here. However, unfelt anger was also associated with an increase occurrence of target
AU17 (chin raiser; 0.29 ±0.34; felt: 0.06 ±0.04), an AU that we did observe in 3/8 clips.
Disgust stimuli also included more target AUs than those reported in the reference study.
There was one exception with target AU26 (jaw drop) occurring less frequently in our stimuli (1/8)
than in those of Gosselin et al. (1995) (rate = 0.40 ±0.35). Target AU17 (chin raiser) and non-target
AU7 (lid tightener) were also observed in most clips described in the present study. In Gosselin et
al. (1995), the occurrence of target AU17 increased in the unfelt condition while the rate of non-
-
Dynamic Clips 18
target AU7 increased in the felt condition. This mismatch between the target/non-target AUs and
the unfelt/felt conditions makes those observations difficult to interpret.
In Gosselin et al., felt fear was characterized mainly by 3 target AUs (4-brow lowerer: 0.33
±0.40; 5-upper lid raiser; 0.55 ±0.41; and 26-jaw drop: 0.75 ±0.17). Several additional target AUs
occurred in less than 30% of their stimuli. In the proposed stimuli, 6 target AUs occurred in more
than 50% of the fear clips as reported in Table 1. The only slight difference between the felt and
unfelt fear in Gosselin et al. was observed with the non-target AU7 (lid tightener) occurring more in
the felt (0.29 ±0.45) than unfelt condition (0.08 ±0.17). Again, the higher occurrence of a non-target
AUs in the felt condition is difficult to interpret. Nevertheless, this non-target (AU7) also occurred
in 2/8 cases in our study, a rate consistent with the felt condition of the reference study.
Felt surprise was characterized by two target AUs (5-upper lid raiser: 0.76 ±0.33; and 26-
jaw drop: 0.72 ±0.41), while other main target AUs occurred rarely (AU1-inner brow raiser and
AU2-outer brow raiser: rate < 0.20) in the stimuli of Gosselin et al. (1995). Again, our stimuli
provided a better match with the prototypes with four target AUs (1, 2, 5, and 26) occurring in all or
almost all surprise clips (Table 1). Consistent with the felt emotions reported by Gosselin et al.,
target AUs 5 and 26 and the non-target AU25 (lip part) were highly frequent.
Finally, our sadness stimuli also demonstrated a better concordance with the prototypes than
those of Gosselin et al. (1995). The expression of felt sadness reported by Gosselin et al. (1995) was
characterized mainly by 3 target AUs (1-inner brow raiser; 4-brow lowerer; and 26-jaw drop),
although each occurred in less than 25% of cases. Several additional target AUs also occurred less
frequently (AU15-lip corner depressor; AU17-chin raiser; and AU25-lips part; mean rate ≤ 0.17). In
our stimulus set, four target AUs (1, 4, 15, and 17) were observed in 4/8 or more clips and three
more (25, 26 and 6) occurred at a lower rate (1-3/8). The target AUs observed here included those
that were only, or mainly, present in the felt condition (AUs 1, 4, 25, and 26) in Gosselin et al.
-
Dynamic Clips 19
(1995). The only minor difference with felt sadness is our observation of AU15 (lip corner
depressor), which was found mainly in the unfelt sadness condition (0.34 ±0.23) in the reference
study.
It should be noted that the increased occurrence of some AUs in the unfelt condition
reported by Gosselin et al. (1995) (e.g. AU17 in anger and disgust) may be confounded with
differences in the intensity of the expression (i.e. possibly stronger in the unfelt conditions) rather
than the genuineness of the expression (e.g. Hill & Craig, 2002). Therefore, it is possible that those
cases in the present stimuli simply reflected stronger genuine expressions. With a few minor
exceptions, the detailed comparison of the proposed stimulus set with the FACS prototypes and
with the observations of Gosselin et al., (1995) in the felt emotion condition supports the validity of
the present stimulus set as representative of the genuine and prototypical expression of felt
emotions.
Perceiving dynamical expressions of emotions and pain
The discrimination of the different emotions and pain based on the overall subjective ratings
of the different emotion/pain categories led to an overall perfect classification rate (cluster analysis).
This means that each stimulus received its highest mean rating (across observers) for the target
emotion/pain category. Global correct recognition rates were well above chance, and in most cases
higher than 85%, while correct rejection rates were always above 95%, consistent with the excellent
sensitivity and specificity of the stimuli. Traces of non-target emotions occasionally lead to
misattribution by some observes, mainly in pain and fear clips (both recognition rates of 74.2%).
Furthermore, the range of correct recognition rates (74% to 100%; see Hit rate in Table 2) was
comparable to the recognition rates reported for the original set of stimuli used by Ekman & Friesen
(1976) to develop the FACS coding system and to determine the prototypical facial expressions of
-
Dynamic Clips 20
basic emotions. Similar recognition rates are also reported by Kappesser & Williams (2002) in
health care professional identifying pain (hit rate = 70%) and negative emotions (70-97%) from
photographs of facial expressions based on FACS prototypes.
The data also suggests that participants perceived some ambiguity or a mixture of several
emotions in some cases. Errors of attribution and mixed emotions were observed mainly in fear and
pain. Traces of surprise were reported in fear expression which sometimes led to misattribution
(Table 2). Previous studies have reported some confusion between fear and surprise when expressed
on human faces in some cultures (Ekman, Friesen, & Ellsworth, 1972) as well as by a model
observer classifying affective faces (Smith, Cottrell, Gosselin, & Schyns, 2005). This is consistent
with the similarity of the facial display associated with fear and surprise as demonstrated by the
overlap of target AUs (1, 2, 5, and 26) involved in those emotions (Table 1). Based on the
prototypical AUs and the observed FACS results, AU4 (brow lower) may be the most distinctive
feature between fear and surprise. In addition, AU25 (lip part) was observed in both fear and
surprise although it constitutes a prototypical AU only in fear. The high rate of AU25 in surprise
clips might explain the detection of mixed emotions and/or misattributions observed in those
categories.
Similarly, pain faces contained traces of disgust, fear and to a lesser extent surprise, and
were occasionally misattributed to these non-target categories (Table 2). In the present stimulus set,
most of those expressions contained AU25 (lip part), a target AU for pain, disgust, and fear
according to the prototypes. Additional AUs commonly observed in both pain and disgust included
AUs 4, 6, 7, 9, 10, and 20. Those AUs are part of the prototypes for pain expression but only AUs 9
and 10 are part of the prototypes for disgust expression. However, it is interesting to note that in
spite of the presence of several non-target AUs, disgust faces were well discriminated. This is
possibly due to the presence of target AU15 (lip corner depressor), a feature that was only observed
-
Dynamic Clips 21
in disgust (and sadness) and may be essential to discriminate disgust from pain and fear. The
circumstances for more ambiguous (or mixed) expression and/or perception of certain emotional
facial expressions should be further investigated in future research. Recognition studies using
random masking of parts of the facial displays (Gosselin & Schyns, 2001) might contribute to the
identification of features that are necessary and sufficient for the adequate discrimination of fear
and surprise.
Valence and arousal
In line with the literature, the investigated expressions can be divided into pleasant
(happiness), neutral (surprise, neutral) and unpleasant (pain, anger, fear, disgust, sadness) facial
expressions according to their valence (e.g. Feldman Barrett & Russell, 1998). From the cluster of
unpleasantly rated facial displays, pain was perceived as most unpleasant. These findings are in line
with the bio-social significance of pain (Williams, 2002). Moreover, subjects also judged that
disgust faces were as unpleasant as pain faces and more unpleasant than fear faces. This may reflect
the functional role of pain and disgust expression in signalling immediate noxious events (tissue
damage or food poisoning) resulting in an actual threat to the integrity of the body (e.g. Wicker et
al., 2003). This clearly contrasts with fearful and angry faces that may signal impending threats
rather than actual noxious events.
With respect to arousal, sad and neutral faces were perceived as least arousing in contrast to
all other EFEs, in accordance to factorial analytic findings (e.g. Feldman Barrett & Russell, 1998).
Again pain clips yielded the highest arousal ratings, however not significantly different from facial
expressions of anger and surprise. This may again reflect a fundamental functional role of pain
expression in communicating actual tissue damage.
-
Dynamic Clips 22
Sex differences
While the observer’s sex did not show any influence on the ratings, the actor’s sex seemed
to modulate the participant’s judgements in some cases. This may reflect a perceptual-cognitive
effect in the observer rather than differences in expression as the FACS results did not show any
significant effect of the sex of the actor. More traces of non-target emotions and higher ratings of
arousal were given to female expression in general. Moreover, females expressing surprise were
perceived as being slightly more pleasant than male actors. However, taking the intensity ratings
into account, judgements did not seem to be strongly influenced by gender stereotypes as suggested
by previous research (Algoe, Buswell, & DeLamater, 2000; Hess, Adams & Kleck, 2004; Plant,
Kling, & Smith, 2004).
Some limitations and future directions
As a final note, some limitations have to be underscored. Although the film clips of EFEs
corresponded generally well with the FACS prototypes, fear and disgust faces may not be optimal.
The imperfect match with FACS prototypes might have affected the recognition of fear faces.
However, the facial expression of disgust was well recognized in spite of the more complex facial
configuration observed in the present stimulus set compared to the FACS prototypes. Variability
between actors in the FACS results and between the present results and the previously reported
prototypes may reflect differences in the specific situations evoked to elicit the emotion or
individual differences in expressive style. Those differences also occur in naturalistic contexts
(Smith & Scott, 1997) and are consistent with the ecological validity of the present stimulus set.
Future studies may examine in more details the symmetry and the smoothness of the full temporal
unfolding of the expression (including the offset) as those factors may be affected by the
genuineness of the expression (e.g. Craig et al., 1991; Gosselin, Kirouac, & Dore, 1995). Those
-
Dynamic Clips 23
factors were not assessed here as they are typically undetected by the observers (Gosselin et al.,
1995; Hadjistavropoulos, Craig, Hadjistavropoulos, & Poole, 1996). Therefore, although the stimuli
may not be “pure” emotion/pain expressions and might include some spatio-temporal irregularities,
the possible presence of those features did not compromise the ability of normal volunteers to
recognize the expressions. The development of pattern-recognition algorithms for the analysis of
facial expression (Hammal, Eveno, Caplier & Coulon, 2006) will likely contribute to future
developments in this field by allowing for an unbiased and automatized analysis of dynamical facial
expression including indices of asymmetries and precise spatio-temporal characterization. Random
spatio-temporal masking procedures (Gosselin & Schyns, 2001) may also contribute to the
identification of facial features necessary and sufficient for the correct discrimination of felt Vs
unfelt emotions and associated with the concealment of felt emotions or pain.
Finally, one has to emphasize that this set of stimuli is consistent with prototypical facial
expressions but not exhaustive, and that perceptual data were obtained in a small homogeneous
group of observers. Spontaneous real-life expressions are of course not restricted to such small sets
and both the expressions and their perception by external observers are also determined by
individual factors, the social surrounding and the cultural background. Furthermore, since the
present stimulus set based on Western prototypes the generalization to other cultural contexts might
be limited. To achieve a more comprehensive and valid investigation of facial expressions of
different emotional states, considerable work remains to be done. In particular, validated sets of
more subtle and mixed emotional expressions, including the social emotions must be developed.
Studies must also be performed in larger populations to investigate more specifically the effects of
individual characteristics (e.g. sex) of the observers.
-
Dynamic Clips 24
Conclusion
We present a set of dynamic, prototypical facial expressions of the six basic emotions, pain
and a neutral display, which can be reliably recognized and distinguished by normal individuals.
Consistent with previous studies, results confirmed the discrimination of facial expressions of basic
emotions. Importantly, the results further demonstrate the distinctiveness of pain facial displays and
the reliable discrimination of pain faces from those of basic emotions. For a comparable intensity of
expression, pain was also perceived as the most unpleasant and arousing, possibly reflecting its
higher bio-psychosocial significance. This standardized set of stimuli, including the comprehensive
rating data and FACS coding, can be widely used in cognitive, social, clinical, and computer
science/neuroscience studies on facial expressions and their social communicative functions.
-
Dynamic Clips 25
References
Algoe, S. B., Buswell, B. N., & DeLamater, J. D. (2000). Gender and Job Status as Contextual Cues
for the Interpretation of Facial Expression of Emotion - Statistical Data Included. Sex roles,
42(3/4), 183-208.
Ambadar, Z., Schooler, J., & Cohn, J. (2005). Deciphering the enigmatic face: the importance of
facial dynamics in interpreting subtle facial expressions. Psychological Science, 16(5), 403-
410.
Botvinick, M., Jha, A. P., Bylsma, L. M., Fabian, S. A., Solomon, P. E., & Prkachin, K. M. (2005).
Viewing facial expressions of pain engages cortical areas involved in the direct experience
of pain. Neuroimage, 25(1), 312-319.
Carr, L., Iacoboni, M., Dubeau, M. C., Mazziotta, J. C., & Lenzi, G. L. (2003). Neural mechanisms
of empathy in humans: a relay from neural systems for imitation to limbic areas.
Proceedings of the National Academy of Sciences of the U S A, 100(9), 5497-5502.
Craig, K. D., Hyde, S. A., & Patrick, C. J. (1991). Genuine, suppressed and faked facial behavior
during exacerbation of chronic low back pain. Pain, 46(2), 161-171.
Craig, K. D., Prkachin, K. M., & Grunau, R. V. E. (2001). The facial expression of pain. In D. C.
Turk & R. Melzack (Eds.), Handbook of Pain Assessment (Vol. 2nd ed, pp. 153-169). New
York: Guilford Press.
Darwin, C. (1872). The expression of the emotions in man and animals. London: Albemarle.
Ekman, P., Friesen, W. V., & Ellsworth, P. (1972). Emotion in the human face: Guidelines for
research and an integration of findings. Oxford: Pergamon Press.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face. Englewoods Cliffs, NJ, USA: Prentice
Hall.
-
Dynamic Clips 26
Ekman, P., & Friesen, W. V. (1976). Pictures of Facial Affect. Palo Alto (CA): Consulting
Psychologist Press.
Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System. Palo Alto (CA): Consulting
Psychologist Press.
Ekman, P., Friesen, W. V., & Elsworth, P. (1982). Research foundations In P. Ekman (Ed.),
Emotions in the Human Face. London Cambridge University Press.
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System (FACS). Salt Lake
City: A Human Face.
Feldman Barrett, L. F., & Russell, J. A. (1998). Independence and bipolarity in the structure of
current affect. Journal of Personality and Social Psychology, 74, 967-984.
Gosselin, P., Kirouac, G., & Dore, F. Y. (1995). Components and recognition of facial expression in
the communication of emotion by actors. Journal of Personality and Social Psychology, 68,
83-96.
Gosselin, F. & Schyns, P. G. (2001). Bubbles: A technique to reveal the use of information in
recognition. Vision Research, 41, 2261-2271.
Hadjistavropoulos, H. D., Craig, K. D., Hadjistavropoulos, T., & Poole, G. D. (1996). Subjective
judgments of deception in pain expression: accuracy and errors. Pain, 65(2-3), 251-258.
Hammal, Z., Eveno, N., Caplier, A.& Coulon, P-Y (2006). Parametric models for facial features
segmentation, Signal Processing, 86, 399-413.
Harwood, N., Hall, L., & Shinkfield, A. (1999). Recognition of facial emotional expressions from
moving and static displays by individuals with mental retardation. American Journal on
Mental Retardation, 104(3), 270-278.
Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2002). Human neural systems for face recognition
and social communication. Biological Psychiatry, 51(1), 59-67.
-
Dynamic Clips 27
Hess, U., Adams, R. B. Jr. & Kleck, R. E. (2004). Facial appearance, gender, and emotion
expression. Emotion , 4, 378-388.
Hill, M.L. & Craig, K.D. (2002). Detecting deception in pain expressions: The structure of genuine
and deceptive facial displays. Pain, 98, 135-144.
Kappesser, J., & Williams, A. C. (2002). Pain and negative emotions in the face: judgements by
health care professionals. Pain, 99(1-2), 197-206.
Kilts, C. D., Egan, G., Gideon, D. A., Ely, T. D., & Hoffman, J. M. (2003). Dissociable neural
pathways are involved in the recognition of emotion in static and dynamic facial
expressions. Neuroimage, 18(1), 156-168.
LaBar, K., Crupain, M., Voyvodic, J., & McCarthy, G. (2003). Dynamic perception of facial affect
and identity in the human brain. Cerebral Cortex, 13(10), 1023-1033.
LaChapelle, D. L., Hadjistavropoulos, T., & Craig, K. D. (1999). Pain measurement in persons with
intellectual disabilities. Clinical Journal of Pain, 15(1), 13-23.
Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures:
affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261-273.
Lievin, M., & Luthon, F. (2004). Nonlinear color space and spatiotemporal MRF for hierarchical
segmentation of face features in video. IEEE Transactions on Image Processing, 13(1), 63-
71.
O'Toole, A. J., Roark, D. A., & Abdi, H. (2002). Recognizing moving faces: a psychological and
neural synthesis. Trends in Cognitive Science, 6(6), 261-266.
Plant, E. A., Kling, K. C., & Smith, G. L. (2004). The influence of gender and social role on the
interpretation of facial expressions. Sex roles, 51 (3/4), 187-196.
Price, D. D. (2000). Psychological and neural mechanisms of the affective dimension of pain.
Science, 288(5472), 1769-1772.
-
Dynamic Clips 28
Prkachin, K. M. (1992). The consistency of facial expressions of pain: a comparison across
modalities. Pain, 51(3), 297-306.
Roark, D., Barrett, S. E., Spence, M. D., Abdi, H., & O'Toole, A. J. (2003). Psychological and
neural perspectives on the role of facial motion in face recognition. Behavioral and
Cognitive Neuroscience Reviews, 2(1), 15-46.
Russell, J. A., & Fernandez-Dols, J.-M. (1997). The Psychology of Facial Expression. New York
Cambridge University Press.
Sato, W., Kochiyama, T., Yoshikawa, S., Naito, E., & Matsumura, M. (2004). Enhanced neural
activity in response to dynamic facial expressions of emotion: an fMRI study. Cognitive
Brain Research, 20(1), 81-91.
Simon, D., Craig, K. D., Miltner, W. H. R., & Rainville, P. (in press). Brain responses to dynamic
facial expressions of pain. Pain.
Smith, C. A., & Scott, H. S. (1997). A Componential approach to the meaning of facial expressions.
In J. A. Russell, & Fernandez-Dols, J.-M. (Ed.), The Psychology of Facial Expression (pp.
229-254). New York: Cambridge University Press.
Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns, P. G. (2005). Transmitting and decoding
facial expressions. Psychological Science, 16(3), 184-189.
Wicker, B., Keysers, C., Plailly, J., Royet, J. P., Gallese, V., & Rizzolatti, G. (2003). Both of us
disgusted in My insula: the common neural basis of seeing and feeling disgust. Neuron,
40(3), 655-664.
Williams, A. C. (2002). Facial expression of pain: an evolutionary account. Behavioral and Brain
Sciences, 25(4), 439-488.
Zald, D. H. (2003). The human amygdala and the emotional evaluation of sensory stimuli. Brain
Research Reviews, 41(1), 88-123.
-
Dynamic Clips 29
Appendix
Instructions to the actors: "We will record movie-clips while you express different basic emotions
and pain in different levels of intensity. If you want, you may also include the interjection "Ah!".
This guide will help you to generate vivid imaginations of these different events. Of course, the
instructions are only suggestions and please feel free to use appropriate personal situations to
facilitate the production of the target expression."
Emotion Proposed vignettes
Surprise I. Slightly surprised: „It’s early October. You step out of the door and detect that
it snowed overnight and everything outside is completely white. “
II. Surprised: “A friend is back from his trip to Australia and tells you that he
met your former classmate on a hike through the outback.”
III: Extremely surprised: “You meet a woman on the street about which you
were told that she was killed in a car accident. Later you find out that it was her
twin sister. “
Fear I. Slightly frightened: “After watching a horror movie you hear strange noises
everywhere in your flat.”
II. Frightened: “Imagine you walk home alone at night. You have to walk
through a very dark alley. You hear foot steps following you. ”
III. Terrified: „You are climbing a mountain over a very high cliff with a river
underneath. You are almost at the top when your foot slips, and you slide down
towards the cliff. Just you go off the cliff, you grap onto a small tree with one
hand. All of your body is hanging over the cliff and you are holding on with
-
Dynamic Clips 30
only one hand. Your friend is coming close to save you, but your arm is giving
up and you are about to let go and fall. Your life is hanging by a thread. ”
Happiness I. Slightly happy: “You are walking through the park. It’s a sunny nice day. You
sit down on a park bench. Right beside you two puppies are playing on the
meadow. They are very cute and move clumsily. You enjoy observing their
play.”
II. Happy: “You walk along the street downtown. You have to stop at a red
traffic light. While waiting there you detect a good friend on the other side of
the road. Imagine the moment, after your first surprise is gone, when you and
the friend look each other in the eyes. “
III. Very happy: “You are having a funny party with close friends.
Disgust I. Slightly disgusted: „You are having a mug of coffee and recognize after
taking a sip that your coffee cream is curdled. “
II. Disgusted: „You take a walk in the park and step into a fresh dog dropping.”
III. Extremely disgusted: „You are on your way to the rest room. When you
enter the room you detect that the toilet has completely overflowed. “
Anger I. Annoyed: „You have an appointment with a friend, that is already 30 minutes
too late and still not in sight.“
II. Angry: „After a long day of work you walk home. You are really stressed,
because you have to carry two heavy bags with vegetables that you bought at the
market. Suddenly, a rout of teenagers is coming from behind, having a race with
their bicycles on the sidewalk and hit one of your bags, which almost fall
down.“
-
Dynamic Clips 31
III. Furious: “You are on your way to an important meeting wearing your best
clothes, when a car at lightning speed passes through a slop of dirty water that
now runs all over your clothes.” or “Imagine your cat or dog is killed by the
neighbours dog.”
Sadness I. Saddened: “A very close friend that you haven’t seen for a very long time is
going to visit you tonight. You are really looking forward to it. But then your
friend is calling and telling you that his car broke down and he has to cancel his
visit.”
II. Sad: “You are in love with a wonderful person. But then this person tells you
that he/she will never feel the same for you. ”
III. Very sad / on the verge of tears / crying: “A close friend or relative dies. “
Pain I. Barely painful: “Imagine a mild pinprick!”
II. Slightly painful: “You scratch your finger slightly on ingrain wallpaper.”
III. Painful: “You caught your finger in a drawer.”
IV. Very painful: “Imagine you cut your finger.”
-
Dynamic Clips 32
Table 1. Occurrence rate of AUs relative to the prototypical expression (■ and □) of basic emotions and pain
Emotion AU
1 2 4 5 6 7 9 10 11 12 15 16 17 20 22 23 24 25 26 27
Inne
r br
ow ra
iser
Out
er b
row
rai
ser
Bro
w lo
wer
Upp
er li
d ra
iser
Che
ek ra
iser
Lid
tight
en
Nos
e w
rinkl
er
Upp
er li
p ra
iser
Nas
o. fu
rrow
de
epen
Lip
corn
er p
ulle
r
Lip
corn
er
depr
esso
r
Low
er li
p de
pres
sor
Chi
n ra
iser
Lip
stre
tch
Lip
funn
el
Lip
tight
en
Lip
pres
sor
Lips
par
t
Jaw
dro
p
Mou
th st
retc
h
# of
targ
et A
Us b
% o
f tar
get A
Us c
Happiness Prototype ■ ■ 2.00 59.2% Observed 1/8 1/8 0/8 0/8 8/8a 3/8 0/8 0/8 0/8 8/8 a 0/8 0/8 0/8 3/8 0/8 0/8 0/8 4/8 a 1/8 0/8 (0.00) Anger Prototype ■ ■ ■ □ □ ■ ■ □ ■ ■ 3.38 89.6% Observed 0/8 0/8 6/8 a 3/8 1/8 5/8 a 1/8 1/8 0/8 0/8 0/8 0/8 3/8 0/8 0/8 6/8 a 3/8 1/8 0/8 0/8 (0.53) Disgust Prototype ■ and/or ■ □ □ □ □ □ 4.38 59.7% Observed 0/8 0/8 7/8 a 0/8 4/8 a 7/8 a 3/8 7/8 a 0/8 0/8 8/8 a 2/8 7/8 a 5/8 a 0/8 1/8 0/8 7/8 a 1/8 0/8 (0.32) Fear Prototype ■ ■ ■ ■ □ ■ and/or ■ and/or ■ 4.63 85.4% Observed 4/8 a 3/8 5/8 a 8/8 a 1/8 2/8 0/8 1/8 0/8 0/8 0/8 4/8 a 0/8 2/8 0/8 0/8 0/8 8/8 a 5/8 a 0/8 (0.42) Surprise Prototype ■ ■ ■ □ □ 3.50 81.3% Observed 7/8 a 8/8 a 0/8 5/8 a 0/8 0/8 0/8 0/8 0/8 0/8 0/8 0/8 0/8 0/8 1/8 0/8 0/8 7/8 a 6/8 a 1/8 (0.27) Sadness Prototype ■ ■ □ □ ■ □ □ □ 4.25 83.8% Observed 6/8 a 2/8 7/8 a 0/8 1/8 2/8 0/8 3/8 0/8 0/8 8/8 a 0/8 7/8 a 2/8 0/8 0/8 0/8 3/8 2/8 0/8 (0.49) Pain Prototype ■ ■ ■ ■ ■ □ □ □ □ □ 6.88 92.3% Observed 2/8 0/8 8/8 a 0/8 8/8 a 8/8 a 3/8 8/8 a 0/8 3/8 0/8 3/8 1/8 8/8 a 0/8 0/8 0/8 8/8 a 0/8 1/8 (0.30)
Note. ■ AU which occurs with prototype; □ may occur with prototype and / or major variant [according to FACS investigators’ guide and Kappesser &
Williams (2002]; /8: number of actors who activated a certain AU with an intensity of ≥ b (b=slight evidence of facial action);
a occurrence rate ≥ 4/8;
b mean number (±SEM) of target AUs activated per clip in each condition;
c percentage of the observed AUs that were part of the prototypes or its major variants in each condition.
-
Dynamic Clips 33
Table 2. Matrix of Subject’s Intensity Rating, Sensitivity (Hit rate) and Specificity (Correct
rejection rate) by Emotion Category
Expression Perceived
Target
Expression
Hap
pine
ss
Ang
er
Dis
gust
Fear
Surp
rise
Sadn
ess
Pain
Hit
rate
(%)
Cor
rect
re
ject
ion
rate
(%
)
Neutral 0.06 (0.02)
0.06 (0.03)
0.06 (0.03)
0.03 (0.02)
0.02 (0.01)
0.14 (0.05)
0.01 (0.01)
71.67 (5.31)
100 (0.00)
Happiness 3.34 * (0.14)
0.00 (0.00)
0.00 (0.00)
0.02 (0.02)
0.18 (0.07)
0.01 (0.01)
0.00 (0.00)
100 (0.00)
96.98 (1.62)
Anger 0.00 (0.00)
2.97* (0.18)
0.13 (0.07)
0.04 (0.03)
0.08 (0.04)
0.09 (0.06)
0.03 (0.02)
97.50 (1.22)
98.84 (0.42)
Disgust 0.04 (0.02)
0.02 (0.02)
2.84 * (0.18)
0.32 (0.09)
0.20 (0.09)
0.13 (0.05)
0.43 (0.12)
85.83 (3.66)
95.58 (0.54)
Fear 0.00 (0.00)
0.08 (0.05)
0.43 (0.10)
2.48* (0.16)
1.08 (0.17)
0.08 (0.04)
0.38 (0.15)
74.17 (5.55)
96.69 (0.59)
Surprise 0.17 (0.08)
0.00 (0.00)
0.00 (0.00)
0.34 (0.19)
3.27 * (0.14)
0.01 (0.01)
0.00 (0.00)
95.00 (1.67)
94.45 (0.96)
Sadness 0.00 (0.00)
0.08 (0.04)
0.24 (0.07)
0.16 (0.07)
0.11 (0.05)
2.40 * (0.14)
0.34 (0.22)
87.50 (6.60)
96.59 (0.71)
Pain 0.11 (0.06)
0.03 (0.01)
0.62 (0.15)
0.53 (0.21)
0.43 (0.13)
0.27 (0.11)
2.68 * (0.25)
74.17 (4.95)
96.18 (1.00)
Note. Values correspond to the mean (±SEM) intensity rating (ranging from 0 to 5), as well as %
(±SEM) correct recognition (Hit rate) and % (±SEM) correct rejection of each expression category
(Expression Perceived) for each target expression condition averaged across actors and observers.
SEM reflects the variability between actors. * Mean rating of the target emotion is significantly
higher than the mean rating on the other scales (Fisher’s protected least significance difference test;
all p’s < .001).
-
Dynamic Clips 34
Table 3. Mean (SEM) Ratings by Emotional Category for Male and Female Actors
Intensity of the
target expression
Valence of the
target expression
Arousal of the
target expression
Target expression
Male actors
Female actors
Male actors
Female actors
Male actors
Female actors
Neutral 0.06 (0.02) 0.05 (0.02) 0.02 (0.03) -0.18 (0.09) -1.23 (0.38) -1.18 (0.36)
Happiness 3.22 (0.15) 3.47 (0.13) 2.80 (0.16) 3.18 (0.12) 0.18 (0.44) 0.73 (0.43)
Anger 3.00 (0.18) 2.93 (0.21) -1.57 (0.22) -1.80 (0.21) 1.20 (0.37) 1.28 (0.30)
Disgust 2.78 (0.20) 2.90 (0.20) -2.57 (0.17) -2.65 (0.15) 0.73 (0.31) 1.23 (0.28)
Fear 2.45 (0.20) 2.50 (0.18) -2.23 (0.12) -2.20 (0.14) 1.37 (0.29) 1.58 (0.30)
Surprise 3.37 (0.17) 3.17 (0.13) -0.27 (0.12) 0.25 (0.16) 1.55 (0.35) 1.48 (0.34)
Sadness 2.42 (0.14) 2.38 (0.19) -2.08 (0.18) -2.30 (0.18) -0.83 (0.32) -0.38 (0.41)
Pain 2.68 (0.23) 2.67 (0.30) -2.77 (0.19) -3.00 (0.15) 2.25 (0.28) 2.03 (0.18)
Note: Intensity ratings ranged from 0 to 5. Valence and arousal ratings ranged from +4 to -4.
-
Dynamic Clips 35
Figure captions
Figure 1. Flow chart illustrating the method of stimuli production and validation.
Figure 2. Examples of four female and four male actors expressing the six basic emotions, pain and
neutral. Five time points in the clip (1ms, 250 ms, 500 ms, 750 ms, 1000ms) are displayed.
-
Dynamic Clips 36
Figure 1
-
Dynamic Clips 37
Figure 2