2014 11 18 wearable computing lecture 05 affective ... 1 18 th november, 2014 assistant professor...

1

18th November, 2014 Assistant Professor Dr. Bert ARNRICH

Wearable Computing

Lecture 5: Affective Computing

Assistant Professor Dr. Bert ARNRICH 218th November, 2014

Intro

� If we want computers to interact naturally with us, then we

need Affective Computing, i.e. computers which recognize

and express emotions

2


Intro

� Sometimes we believe that computers notice our emotional

state� We repeatedly type the same wrong command or click on something

that does not work, as if the computer would notice our frustration and react in some way


Intro

� Sometimes we even have a conversation with a computer

program which has almost no information about human

thought or emotion

3


Eliza

� In 1966, Joseph Weizenbaum

implemented ELIZA

� Simple parsing and substitution

of key words into phrases� "Who is your favorite composer?“

� "What about your own favorite composer?" or

� "Does that question interest you?“

� ELIZA effect: tendency to

assume that computer behave

like humans


Application Scenarios I

� Call center: � If a user is frustrated/confused, connects him to a human operator

� Emotional music player: � Music selection according to user’s mood

� Teleconference support: � Display information about participants’ emotional states

� Affective electronic communication: � Automatically tag messages with Emoticons ☺ � �

� Computer-aided learning: � Adapt teaching in case the student is bored or frustrated

4


Application Scenarios II

� Media content tagging: � Identify and tag emotional pictures/sequences automatically

� Affective disorders: � Monitor daily variations of patient’s state to improve treatment

� Personal Stress Assistant:� Keep track of stressful situations and help creating a better “Work-Life-

Balance”

� “Out-of-the-lab” psychological studies� Understand the role of emotions in daily life, e.g. development of

relationships or group behavior


What are emotions?

� Consensual definition derived from 92 different notions:

� Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can � Give rise to affective experiences such as feelings of arousal,

pleasure/displeasure

� Generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes

� Activate widespread physiological adjustments to the arousing conditions

� Lead to behavior that is often, but not always, expressive, goal-directed, and adaptive.

[Kleinginna et al., 1981] A categorized list of emotion definitions, with suggestions for a consensual definition

5


Darwin was one of the first to study emotions

[Darwin, 1872] Expression of the emotions


Emotion Categorization into Valence – Arousal

6


Physiological adjustments to arousing conditions

� Apparent to others:� Facial expression

� Voice intonation

� Gestures, Movement

� Posture

� Pupillary dilation

� Less apparent to others:� Respiration

� Heart rate

� Temperature

� Electrodermal response

� Muscle action potentials

� Blood pressure

[Picard, 2000] Affective Computing

Most of the physiological adjustments can be monitored with wearable devices.


Activation of physiological adjustments means …

� … that we can measure and model emotions

HR = heart rate, FT = finger temperature

[Setz, 2012] Multimodal Emotion and Stress Recognition

7


Emotion Recognition



Content of this Lecture

Sensing of Physiological adjustments

Emotion Elicitation Ground Truth

8


Content of this Lecture




Facial Expression

� Paul Ekman et al. have shown that facial

expressions across the globe fall roughly into few

categories:

� SADNESS: The eyelids droop as the inner corners of the brows rise and, in extreme sadness, draw together. The corners of the lips pull down, and the lower lip may push up in a pout.

� SURPRISE: The upper eyelids and brows rise, and the jaw drops open.

� ANGER: Both the lower and upper eyelids tighten as the brows lower and draw together. Intense anger raises the upper eyelids as well. The jaw thrusts forward, the lips press together, and the lower lip may push up a little.

9


Facial Expression

� Paul Ekman et al. have shown that facial

expressions across the globe fall roughly into few

categories:

� DISGUST: The nose wrinkles and the upper lip rises while the lower lip protrudes.

� FEAR: The eyes widen and the upper lids rise, as in surprise, but the brows draw together. The lips stretch horizontally.

� HAPPINESS: The corners of the mouth lift in a smile. As the eyelids tighten, the cheeks rise and the outside corners of the brows pull down.


� Specify how to recognize and score action units (AUs) which

represent the muscular activity that produces momentary

changes in facial appearance

� Computer Vision methods available for automatic coding

Facial Action Coding System

AU1: Inner Brow Raiser AU2: Outer Brow Raiser AU4: Upper Lid Raiser

10


Voice

� The most important is often not what is said but how it is

said.

� Even pets are able to “understand” how something is said

� Try it by your own: infer emotions from pseudo sentences � “Hätt sandig prong nju wentsie”

� “Vi gott leich jean kill gos terr”

Happiness Interest Anger Boredom

I22103

F22205

L22203

K11111


Voice

� The most important is often not what is said but how it is

said.

� Even pets are able to “understand” how something is said

� Try it by your own: infer emotions from pseudo sentences � “Hätt sandig prong nju wentsie”

� “Vi gott leich jean kill gos terr”

Happiness Interest Anger Boredom

I22103

F22205

L22203

K11111

11


Voice Feature Calculation



Body Movement

� Body movements are related to the level of arousal or the

degree of intensity of an affective experience

� Happiness: relatively jerky, loose, fast, hard, expanded, and full of action.

� Anger: very jerky, stiff, fast, hard, expanded, and full of action.

� Sadness: very smooth, loose, slow, soft, contracted, inactive.

� Neutral: relatively smooth, loose, slow, soft, very contracted, and inactive

12


Body Movement: Stress vs. Cognitive Load

� Motivation:� Body movements carry

information about affective states.

� Chair Sensor:� Tekscan pressure mat

� 32x32 senor elements

� Sampling 25Hz

� Two conditions under study� Stress

� Cognitive Load

[Arnrich et al., 2010] What does your chair know about your stress level?


Body Movement: Center of Pressure

5 10 15 20 25 30

51

02

03

0

1:dim.x

1:d

im.y

5 10 15 20 25 30

51

02

03

0

1:dim.x

1:d

im.y

5 10 15 20 25 30

51

02

03

0

1:dim.x

1:d

im.y


Compute Center of Pressure (CoP) out of each pressure frame and

investigate CoP movements.

13


Body Movement: Measure of Nervousness

� Variation of the x-

CoP is higher during

stress for most of the

subjects (p<0.05)

� Might be a measure

of nervousness

� 6 subjects behave

differently to stress



Posture

� Naturally occurring postures convey information about

affective states and interest level

� Symbolic postures convey a specific meaning about the

actions of a user, e.g., leaning forward toward a computer

screen might be a sign of attention

14


Respiration

� Normal respiration rate at rest: 12-18 breaths per minute

� Increase of respiration rate is often observed during anxiety

� Often used measurement principle: strain sensor


Respiration: Measurement via Seat Belt

15


Respiration

� Respiration signal is sometimes affected by motion artifacts



Respiration

� Usually, the feature of interest is the respiration rate

� Peak detection allows to identify inhalation/exhalation events

16


Heart Rate

� The electrocardiogram (ECG)

measures the electrical activity of the

heart muscle

� The characteristic segments and

peaks of the ECG pattern are named

by the letters P to T

� The R-peak represents the most

prominent attribute of the ECG and is

used to compute the heart rate


Heart Rate

� Nowadays a broad variety of wearable chest belt exists

which allow to record ECG data

� Usually, the chest belts provide additional measures on body

acceleration (important for artifact detection), breathing rate

and skin temperature

17


ECG integrated into Airplane Seat

[Schumm et al., 2012] ECG Monitoring in an Airplane Seat


Heart Rate: RR intervals

� Important feature of interest is the time duration between

successive heart beats, called RR interval

18


Heart Rate: RR intervals

� Important feature of interest is the time duration between

successive heart beats, called RR interval

� RR intervals over time reveal heart rate variability

[Schumm et al., 2012] ECG Monitoring in an Airplane Seat


� Heart rate variability decreases with increasing load levels

RR intervals during workload load levels

[Cinaz et al., 2013] Monitoring of Mental Workload Levels during an Everyday Life Office-Work Scenario

19


Heart Rate Variability: Frequency Domain

� Compute power spectral density of the RR-intervals

� Usually, the Lomb-Scargle Periodogram is used for

computation in order to account for then uneven sampled

heart beat events



Heart Rate Variability: Frequency Domain

� Two important frequency ranges are often considered� Low frequency (LF) range: 0.04-0.15 Hz

� High frequency (HF) range: 0.15-0.4 Hz

20


Common used Heart Rate Variability Features



Temperature

� Peripheral body temperature is affected by emotional states

� Usually measured with a temperature sensor attached to the

finger or the arm

� It was shown that average finger temperature � increases between 0.1 and 0.2 degree Celsius due to anger

� decreases between 0.01 and 0.08 degree Celsius due to fear

21


Electrodermal Response

� Psychological or physiological arousal results in increased

sweat gland activity



Electrodermal Response

� Electrodermal response measurement is a method of

measuring the electrical conductance of the skin which

varies with its moisture level

� Also known as skin conductance, galvanic skin response

(GSR), electrodermal response, electrodermal activity

(EDA), psychogalvanic reflex

� Electrodermal response is usually measured at the palmar

sites of the hands or the feet where the density of sweat

glands is highest (> 2000/cm2).


22


Skin Conductance Reaction

� Ideal Skin Conductance Reaction to a stimulus



EDA signal example

� Smoothed EDA signal consists of � a slowly changing part (Skin Conductance Level: SCL)

� overlaid by short, fast conductance changes (Skin Conductance Responses SCRs)


23


EDA signal example

� High-pass filtered EDA signal with threshold and detected

peaks used for detection of skin conductance responses



EDA and Affective State

� Slowly changing skin conductance level (SCL) is a measure

for general psychophysiological activation

� Fast conductance changes (Skin Conductance Responses

SCRs) as reaction to a stimulus or general activation


24


Emotional sweating under stress

10 20 30 40 50 60 70 800

0.5

1

1.5

2

2.5

3

3.5

x 10−6 EDA level data (stress condition)

Time [min]

Co

nd

ucta

nce [

S]

Baseline (Instruction) MIST

C+SF2C2C1 F1

Recovery

[Setz et al., 2009] Discriminating stress from cognitive load using a wearable EDA device


Electromyogram

� Electrical activity of a muscle can be measured using

electromyography (EMG)

� Usually measured via surface electrodes applied to the skin,

e.g. on the “smiling muscle”


25


Electrooculogram

� Electrooculography (EOG) is a technique

to measure the resting potential of the

retina

� Used for eye movement analysis

� Measured between two electrodes

� Attached at the right and left side of the eye for the horizontal eye movements

� Attached below and above the eye for the vertical eye movements


Electrooculogram and Affective State

� In the vertical EOG, the blinks are visible

� Eye blinks are related to startle events

� The blink rate can be affected by the emotional state of a

person


26


Lecture Content




Emotion Elicitation: Pictures

27


Emotion Elicitation: Pictures

� Applicable to elicite a range of emotions, simple, highly

standardized experiment setup allows for replication

� Only short duration of emotions (~6 s) can be induced

� Some emotions (e.g. fear, anger) are difficult to induce with

pictures because subjects are not personally threatened.

� International Affective Picture System (IAPS)� Database of pictures used to elicit a range of emotions.

� Contains over 1,000 colored pictures


Emotion Elicitation: Movies

28





� Wide range of emotions which develop over time (~1-10 min)

� Simple, highly standardized experiment setup allows for replication

� Allows to elicit cognitively sophisticated emotional states such as nostalgia

� Some emotions (fear, anger) are difficult to induce because subjects are not personally threatened

� Recommended film clips for eliciting discrete emotions (amusement, anger, disgust, fear, plain neutral, pleasant neutral, sadness, surprise) are available


29


Emotion Elicitation: Music

� Emotions develop over time (15-20 min.)

� Simple, highly standardized experiment setup allows for

replication

� Music taste might influence experienced emotions.

� Moods (positive or negative) rather than discrete emotions

� Recommended classical music pieces for inducing happy

and sad moods are available



Emotion Elicitation: Psychological Approaches

� Masking of experiment purpose fosters realistic responses

� Emotions occur in social context

� Effective technique for eliciting anger and stress

� Time-consuming and complex experiment setup� development of convincing cover story

� training of experimenters

� thorough debriefing

� Might be ethically critical


30


Psychological Approach: Load + Social Evaluation

Arithmetic

Task

Time

Pressure

Performance of

norm collectivePerformance of

subject (always

less than norm)

Answer field


Psychological Approach: Social Stress

1. Mild� Do you have problems with the keyboard?

� Can you read the tasks on the screen well?

2. Moderate� Did you sleep bad?

� Are you feeling bad today?

3. Strong� Do you take drugs?

� Did you already had math problems in school?

31


� 5 minute presentation in front of a jury

� 5 minutes mental arithmetic in front of the jury:

count backwards from 1,022 in steps of 13. If a mistake is

made, then start again from the beginning.

Psychological Approach: Trier Stress Test


Emotion Elicitation: Social Interaction

� Recommended approach to invoke a conflict discussion in

interacting subjects (e.g. couples)

� Elicited emotions closely resemble the emotions that occur in

everyday life

� Temporal characteristics of emotions can be investigated

� Time-consuming and complex experiment setup, limited

standardization since experimenter can not totally control

interactions and participants may change the topic


32


Emotion Elicitation: Daily Life

� Lab stress, movie and soccer

[Wilhelm et al., 2010] Emotions beyond the laboratory


Emotions Elicitation during Soccer


33


Lecture Content




� When eliciting emotions through pictures, movies or music a

ground truth is given by the content of the media

� However, self-assessment questionnaires is the preferred

method to gather ground truth

� There exists a variety of self-assessment questionnaires

developed by psychologists

� Depending on the research question, one has to decide

when and what to ask

Groundtruth

34


When to ask?

� In standardized experiments when using

pictures/movies/music, usually after each stimulus the

subject is asked for a self-assessment on experienced

emotions

� In more real-life experiments, more practical self-

assessments are used� Experience Sampling

� Day Reconstruction


Experience Sampling

� Ask the subjects at random time

points how they feel right now

� Allows to obtain experienced

feelings in the current situation

� Subjects should not be aware

when the next questionnaire is

given

� Consider a minimal time span

between successive time points

35


� Retrospective self-assessment, usually done at the end of a

day

� Ask the subjects to define episodes of daily activities and

rate their feelings within the episodes

� Self-assessments might be biased, e.g. in the evening you

might forget a stressful event occurred during the day

Day Reconstruction

Episode Name Time Start Time End How did you feel?

Preparing breakfast 7:30 8:00 Tired

Commuting 8:00 8:30 Boring

Read email 8:30 9:00 Exited

… … … …


What to ask?

� Mood and Feelings Questionnaire� Available for child and adults

� Long (33 items) and short versions (13 items) available

[http://devepi.duhs.duke.edu/mfq.html]

36


What to ask?

� Positive Affect Negative Affect Scale� Up to 60 items

� Measures the two scales positive/negative affect and specific affects

[http://www2.psychology.uiowa.edu/faculty/Clark/PANAS-X.pdf]


� Geneva Emotion Wheel� Emotions are arranged in the arousal – valence space

What to ask?

[http://www.affective-sciences.org/gew]

37


What to ask?

� Self-assessment manikin� First row: valence, second row: arousal

� Useful to allow fast and simple response, e.g. when collecting self-assessments with the smart phone


� Biological marker like Cortisol useful in stress experiments

� Experts label video sequences recorded from the subjects� Often it is required that several experts rate the same sequences in

order to estimate inter-rater reliability

Stress

Control

Alternative Groundtruths

38


Lessons learned

� Emotions activate widespread physiological adjustments which can be monitored with wearable devices

� Sensing of Physiological adjustments to arousing conditions

� Emotional face expressions can be analyzed via Facial Action Coding

� Voice feature calculation helps to understand how something is said

� Body movements carry information about level of arousal

� Postures convey information about interest level

� Respiration rate is usually measured with strain sensors

� Heart rate and its variability is a powerful modality in emotion recognition

� Body temperature is affected by emotional states

� Arousal results in increased sweat gland activity observable via electrodermal response measurement

� Electromyogram to measure electrical activity of a muscle, e.g. “smiling muscle”

� Electrooculography for measuring eye blinks that are affected by emotional state


Lessons learned

� Emotion Elicitation

� Pictures: short duration, wide range of emotions

� Movies: wide range of emotions which develop over time

� Music: eliciting mood rather than discrete emotions

� Psychological Approaches: complex setup, effective technique for

eliciting anger and stress

� Social interaction: closely resemble emotions that occur in everyday

life, limited standardization

� Daily life: ultimate goal

39


Lessons learned

� Ground Truth

� When to ask: Experience Sampling, Day Reconstruction

� What do ask: Mood and Feelings Questionnaire, Positive Affect

Negative Affect Scale, Geneva Emotion Wheel, etc.

� Biological marker

� Video labeling

2014 11 18 wearable computing lecture 05 affective ... 1 18 th november, 2014 assistant professor...

Documents