05 JUNE, 2015

SANGSUN PARK

Sungkyunkwan Univ. Dept of., Human ICT Convergence

The Emotion Recognition Using Wearable Device Basis on Bio-Data Analysis.

of 24

Motivation

“The Emotion is the way of the representation to express their inner face.”

- Bertand Russell, Philosopher, 1872

Jerome Kagan, What is Emotion?: History, Measures, and Meanings, ISBN 978-0-300-12474-3, Yale University Press

2

Outline

1. Introduction

2.Subject definition

3.Related works

4.Difference of other works

5.Emotion analysis demensions

6.Experiment I

7.Experiment II

1) Experimental equipments

2) Experiment process

3) Wearable Prototype

4) Data Analysis

8.Proposal for Service

9.Conclusion

Introduction

of 24

Introduction

•Motivation :- How can we see the inner face of other person?- The Feelings transmission is due to the smooth, Human in group can feel more happiness or feel less fear. Garriy Shteynberg et al, Feeling More Together: Group Attention Intensifies Emotion, American Psychological Association 2014, Emotion, Vol. 14, No. 6, 1102–1114

• Object :

- The Emotion of Human is qualitative amount that invisible and uncounterable level. -> Quantitative &

Countable.- Wearable Device Prototype to express emotion in face-to-face situation opponent between the human.

•Method :

- The human emotion is estimated by utilizing the biometric data, the development of devices for the

classification model and the real-time data acquisition feelings.

• Result :

- The human emotion analysis basis on Bio-data and wearable prototype, design prototype

5

of 24

Subject definition

6

ExcitingSad Unhappy Happy

Neutral

Junwoong Lee et al, Classification of Emotion Terms in Korean, Korea media communication research, 2008, pp.85-116, 504 1229-7526 KCI

Stimulation from external

Perception Response Emotion express

Visual

Sound

Temperature

Cause

Scene

Evironment

Episodic Memory

Remind

Process

Factors Gesture

Laugh

Action

Sweat

Cognition from internal

: How can we feel the emotion?

Exciting

Happy

Neutral

Unhappy

Sad

of 24

Related works : Biometric based

7

1. F. Nasoz et al, Emotion Recognition from physiological signals using wireless sensors for presence technologies, 2003 2. A. Haag et al, Emotion Recognition using Bio-sensors : First steps towards an Automatic system, 2004 3. K. H. Kim er al, Emotion recognition system using short-term monitoring of physiological signals, 2004 4. L. Li, Emotion Recognition using physiological signals, 2006 5. C. Jing, The Research on Emotion Recognition from ECG Signal, 2009

Summary

ReferenceExp. Device Bio-Data Experimental Method

Emotion Factors

Analysis Result

1 Sense Wear armband

G S R ( H e a r t rate, Temp.)

- Show the movie have the emotion context 5 Min.

6 K N N , D F A , MBP

MBP 83%, KNN 71%, DFA 74%

2 Wireless Device for Bio-Sensing

EMG , E C G , Resp i rat ion , S C , T EMP . , BVP

- Show the 800 pictures concerned emotion expression from IAP Organization. 2 Mean, STD Arousal 96.58%

3 BIOPAC MP100P P G , S K T , ECG, EDA(Ag/AgCl)

- Tell the story telling to participant using interaction doll 2 Min 4 Fisher-KNN

Emotion detection 78.43% In 50 samples

4

AD Instruments ML870, MLT409 ~ ML132 BioAMP

EEG, ECG, HR, SKT, SC

- Show the movie have emotion 5 Min

3 SVM, CCAEmotion detection 85.3% In 60 samples

5 BIOPAC MP150 ECG

- Show the movie have emotion 4 Min

2 Fisher-KNNSadness detection in Fisher-KNN 94.59%

The most of all Emotion researches are using the biological method that HRV. ECG, EEG, fMRI. GSR(SC).

Reference

of 24

Related works : Device or Service based: Wearable Device & Service based studies.

Yu Hao et al, A Visual Feedback Design based on a Brain-Computer Interface to Assist Users Regulate their Emotional State, CHI 2014, One of a CHInd, Toronto, ON, Canada, Work-in-Progress

: The emotion detection from EEG signal

Sara Ferraro et al, TWINY emotional logging, ISWC '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

: Emotion detection wearable device

: Emotion cartographyChristian Nold, Emotion caltography, http://emotionalcartography.net

8

ETC : Foursquare, IM iN(Only SNS)

Work Definition

of 24

Work Proposal

10

1. Easy to get Bio-data thru Wearable Device: Wearable device to collect Bio-data(GSR, HRV)

2. Emotion detection with Color based Classification

Model and Algorithm: Emotiom circumplex model for Classification

3. Heartmap : Proposal Service design prototype: context awareness proposal service using emotion data

Negative Positive

High Arousal

Low Arousal

4

5

1

2

3

of 24

Differences of other works

11

Study

FactorThis work Biometric based Device or Service

based

Bio-data HRV / GSR ECG, EEG, Resperation EEG, HRV

Device Development Wearable Experimental Equipments Wearable

Emotion classification 5 6 ~ 2 Color & Visual

External factors Sound & Humidity Temperature .

Analyized Data Transfer WiFi & BLE Laboratory, Wired Data

of 24

Emotion analysis demensions

1. Demention I : HRV, GSR

• Internal Factors : for the stresss level

12

K. H. Kim et al, Emotion Recognition system using short-term monitoring of physiological signals, 2004

2. Demension II : Temperature, Sound

• External Facters for emotion

3. J. Russell`s The circumplex model of affect

: The emotion classification

• X / Axis = Positive / Negative

• Y / Axis = Arousal high / Arousal low

• 2 Axis can determine to emotion level

Jihye Yeo et al, Developing and Adapting an Emotion Model Using Colors for an Emotion Expression, HCI-K 2008

Sangmin Hwang, Kyongmin Kim, Study on the development of Color Sensibility Scale and its application, Korean Society for

Emotion & Sensibility 1999

Unhappy Happy

High Arousal

Low Arousal

4

5

1

2

3

Sad ExcitingNeutral HappyUnhappy

of 24

Emotion Detection

13

HRV

GSR

Hum.

Arousal High

Arousal Low

Internal

External

Internal

External

Happy

UnHappy

Emotion Model Emotion DB

Sound

Place

Temp.

Bio-data Analysis Emotion Feedback

of 24

Service PrototypeHeartmap Service

14

HRV

GSR, Temp., Sound

EmotionAnalysis 1

Emotion Analysis 2

Emotion Analysis 3

USER

Profile GPS Emotion

Database

Profile

GPSEmotion

Emotion DB Cloud

Emotion Analysis 5

Emotion Analysis 4

GeeYoung Noh, DongNyuk Jeong, SangSun Park, Heart Map : Shared Emotions based on Map Service, CHI 2015 work in progress

Pilot Experiment

of 24

Pilot Experiment

16

Arduino Develoment Board

T34 : Heart rate transfer

Sangsun Park et al, A study on the human emotion inference in meaningful place using HRV signal and text mapping. HCI 2015, 253-260.

1. Equipments

1 2

3

4 5

Sad ExcitingNeutral HappyUnhappy

1 2 3 4 5

of 24

Pilot Experiment

17

EXP. Picture HRV Verbal Emotion Value Inferred Emotion Survey Emotion

1 Kyogbok Palace 4.50 2.50 3.50 Neutral Neutral

2 National cemetery 4.50 1.00 2.75 Sad Sad

3 Amusement Park 4.25 4.00 4.13 Exciting Very Exciting

4 Mt. Beakdoo 2.00 2.75 2.38 Sad Sad.

5 Airplane crash 3.75 -0.25 1.75 Sad Very Sad

Sangsun Park et al, A study on the human emotion inference in meaningful place using HRV signal and text mapping. HCI 2015, 253-260.

3. Result

Emotion Level

Pictures

Equipment

of 24

Sensors

19

HRV(Heart Rate Variability): BPM, IBI, R-R Interval, ECG(Electrocardiography)

GSR(Galvanic skin reflex) : EDR(Electrodermal response), PGR (psycho galvanic reflex), SCR(skin conductance

response)

Sound

•Enviromental sound may occure to effect the emotion of human. •The most bad noise to effect the emotion is animal bark.

Humidity

•If someone exposed in extreme hot weather, It could effect to emotion. •When the natural temperature is growing, The people in online have more progressive.

Junggi Yang et al, A change of the public's emotion depending on Temperature & Humidity index. The Society of Digital Policy & Management, 12(10), 243-252.

Schuller, B., Hantke, S., Weninger, F., Han, W., Zhang, Z., & Narayanan, S. (2012, March). Automatic recognition of emotion evoked by general sound events. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on (pp. 341-344). IEEE.

of 24

Equipment : Wearable Prototype

•Arduino WiDo & Sensors : GSR, HRV(IR Type), Temperature & Humidity, Sound Sensor, LED Bar, SDHC memory data logger

20

Atmega Development : Arduino Leonardo WIDO

Shiled Prototype : Easy to convertable, reenginner

HRV & GSR Sensors : Necklace, Chest, Fingers

of 24

Experimental processIAPS(Affective Picture System) •It could be more responsibility using the Emotion pictures from IAP organization.

•The Experiment pictures have reflected the 3 kinds of normal context emotion.

•5 Participants result, 20Min, Compare Means

•The Experimental situation represented by 2 cases.

•Humidity Case 1 : Natural below 50% / Case 2 : Above 80%(Use humidifier)

•Sound Case 1 : Natural noise in laboratory ( Below 50db) / Case 2 : White Noise(Above 00db)

21

소음측정 및 평가, 산업안전보건연구원 안정위생연구센터, 2007. 04

Experiment II Result

of 24

Experimental result : Data Analysis - Each 10 seconds using HRV+GSR (+Sound+Humidity)

23

60

68

76

84

92

100

1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177

0

175

350

525

700

2 11 20 29 38 47 56 65 74 83 92 101 110 119 128 137 146 155 164 173

0

1

2

3

4

5

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171

HRV

GSR

Emotion Level

70

76

82

88

94

100

1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177

0

175

350

525

700

2 11 20 29 38 47 56 65 74 83 92 101 110 119 128 137 146 155 164 173

25

40

55

70

85

100

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171

0

175

350

525

700

2 11 20 29 38 47 56 65 74 83 92 101 110 119 128 137 146 155 164 173

0

1

2

3

4

5

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 1710

1

2

3

4

5

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171

Sad Neutral Exciting

Unhappy Happy Happy

High Arousal Low Arousal High Arousal

High

Time(s) Time(s) Time(s)

Time(s)Time(s)Time(s)

High High High

High High High

of 24

ConclusionThe Study : The Emotion Recognition Using Wearable Device Basis on Bio-Data Analysis.

•This study presented easy to get emotion & level analysis using wearable device

•HRV is utilized for Low Arousal and High Arousal.

•GSR is utilized for Happy and Unhappy.

Future works

•Considering external Environment factors, Sound and Humidity.

•This research may use for the new value between men on men.

•Using Wearable device could make new mobile service and interaction in realtime.

24

Thank you for your attention.

References1. Jerome Kagan, What is Emotion?: History, Measures, and Meanings,

ISBN 978-0-300-12474-3, Yale University Press

2. Garriy Shteynberg et al, Feeling More Together: Group Attention Intensifies Emotion, American Psychological Association 2014, Emotion, Vol. 14, No. 6, 1102–1114

3. Junwoong Lee et al, Classification of Emotion Terms in Korean, Korea media communication research, 2008, pp.85-116, 504 1229-7526 KCI

4. F. Nasoz et al, Emotion Recognition from physiological signals using wireless sensors for presence technologies, 2003

5. A. Haag et al, Emotion Recognition using Bio-sensors : First steps towards an Automatic system, 2004

6. K. H. Kim er al, Emotion recognition system using short-term monitoring of physiological signals, 2004

7. L. Li, Emotion Recognition using physiological signals, 2006

8. C. Jing, The Research on Emotion Recognition from ECG Signal, 2009

9. Yu Hao et al, A Visual Feedback Design based on a Brain-Computer Interface to Assist Users Regulate their Emotional State, CHI 2014, One of a CHInd, Toronto, ON, Canada, Work-in-Progress

10. Sara Ferraro et al, TWINY emotional logging, ISWC '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

11. C h r i s t i a n N o l d , E m o t i o n c a l t o g r a p h y , h t t p : / /emotionalcartography.net

12. K. H. Kim et al, Emotion Recognition system using short-term monitoring of physiological signals, 2004

13. Jihye Yeo et al, Developing and Adapting an Emotion Model Using Colors for an Emotion Expression, HCI-K, 2008

14. Sangmin Hwang, Kyongmin Kim, Study on the development of Color Sensibility Scale and its application, Korean Society for Emotion & Sensibility 1999

15. Sangsun Park et al, A study on the human emotion infer in meaningful place using HRV signal and text mapping. HCI 2015, 253-260.

16. Hagit, C. et al. (1998). "Analysis of heart rate variability in posttraumatic stress disorder patients in response to a trauma-related reminder". Biological Psychiatry 44 (10): 1054–1059. doi:10.1016/S0006-3223(97)00475-7. PMID 9821570.

17. Castaldo, R., et al. "Acute mental stress assessment via short term HRV analysis in healthy adults: A systematic review with meta-analysis." Biomedical Signal Processing and Control 18 (2015): 370-377.

18. Schuller, B., Hantke, S., Weninger, F., Han, W., Zhang, Z., & Narayanan, S. (2012, March). Automatic recognition of emotion evoked by general sound events. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on (pp. 341-344). IEEE.

19. Junggi Yang et al, A change of the public's emotion depending on Temperature & Humidity index. The Society of Digital Policy & Management, 12(10), 243-252.

of 24

Related works

27

Castaldo, R., et al. "Acute mental stress assessment via short term HRV analysis in healthy adults: A systematic review with meta-analysis." Biomedical Signal Processing and Control 18 (2015): 370-377.

The research trend in HRV Analysis: BPM, IBI, R-R Interval, ECG(Electrocardiography)

of 24

Experimental equipments

28

Hagit, C. et al. (1998). "Analysis of heart rate variability in posttraumatic stress disorder patients in response to a trauma-related reminder". Biological Psychiatry 44 (10): 1054–1059. doi:10.1016/S0006-3223(97)00475-7. PMID 9821570.

HRV(Heart Rate Variability) : BPM, IBI, R-R Interval, ECG(Electrocardiography)

Recall Rest

of 24

Experimental equipmentsGSR(Galvanic skin reflex) : EDR(Electrodermal response), PGR (psycho galvanic reflex), SCR(skin conductance response)

29

http://www.seeedstudio.com/wiki/Grove_-_GSR_Sensor

Deep Breath

Starve

of 24

Experimental equipments Addtional data - Sound/Humidity

30

Sound •Enviromental sound may occure to effect the emotion of human.

•The most bad noise to effect the emotion is animal bark.

Humidity •If someone exposed in extreme hot weather, It could effect to emotion.

•When the natural temperature is growing, The people in online have more progressive.

Junggi Yang et al, A change of the public's emotion depending on Temperature & Humidity index. The Society of Digital Policy & Management, 12(10), 243-252.

Schuller, B., Hantke, S., Weninger, F., Han, W., Zhang, Z., & Narayanan, S. (2012, March). Automatic recognition of emotion evoked by general sound events. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on (pp. 341-344). IEEE.