modeling driver/passenger behavior garching, 12.july.2019 … · 2019-07-13 · modeling...

Kim, Kumar. Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment

Modeling Driver/Passenger BehaviorAccording to Emotional States for In-CabinEnvironment

Eesha Kumar

Pooreumoe Kim

Technische Universität München

Department of Informatics

Garching, 12.July.2019

1

Kim, Kumar. Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin EnvironmentKim, Kumar. Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment

Outline

Introduction Behavior Modeling Future Work Conclusion

2


Outline


3


Research Question(s)

How can we create/model human behavior from emotional states for In-Cabin Environments?

- What is human behavior?

- What are the major challenges in modeling behavior?

- Why should be model behavior?

- What are some possible approaches?

4


What is Human behavior?

“In a driving environment, particularly level 3 and 4 semiautonomous, a passenger's behavior is

defined by an action performed within the observable environment resulting from an emotional

state.”

5


Why should we model human behavior?

- Human Perspective

- Safety for human-in-the-loop systems

- Improve reliability and trust

- Personalization and adaptability

- Autonomous Perspective

- Actions as a Quantifiable uncertainty (Human Aware)

6


What are the major challenges in accurately modeling human behavior?

- Unpredictability in human nature

- One Emotional State can influence Multiple behaviors and/or vice-versa

- Too many unknowns in an environment

- Difficulty in accurately capturing environment data

7


What are some possible approaches?

- Machine Learning Techniques

- Variational Methods

- Probabilistic Models

- Self Assessments

8


Outline


9

Kim, Kumar. Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin EnvironmentKim, Kumar. Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment 10

The Big Picture


behavior Modeling

Sensors Emotion recognition

Interaction Domain

11

Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment 12

1. Channels

a. Questionnaire

b. Electroencephalography(EEG)

c. Audio

d. Video

e. Simulation

2. Techniques

a. Machine Learning

i. Conventional

ii. Deep Learning

b. Probabilistic Models

Emotion & Affection Recognition

behavior Modeling

Sensors

Emotion recognition

Interaction Domain

Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment

Emotion & Affection Recognition - Channels

13

Questionnaires

- Generating initial labeled data

- Preparing personalized data



14

Simulation

- A popular methods to acquire drivers’ data.

- A simulator may involve monitors, and camera to record subjects.

- It also generates artificial sounds to mimic driving environment [17].



15

Electroencephalography(EEG)

- It measures brain activity.

- Bryan James Higgs collected EEG data and use it to his car-following model[2].

- M Soleymani, M Pantic, T Pun measured EEG as well as eye gaze to generate multi-modal model for emotion recognition [14].


Facial Recognition

- Informative component in emotion prediction.

- Eye Movement/ Gaze analysis improves accuracy[21].

- Others measure facial thermometer[12].


16

[A Behavior-based Emotion Recognition System in Intelligent Cars]



17

Speech Corpus

- A strong tool to measure emotion.

- Hard to collect natural verbal data from simulation (Privacy, etc)

- Researchers classifiers to identify emotions from IEMOCAP and SEMAINE [18].



18

Body Features

- Ishan Behoora and Conrad S. Tucker quantified body parts.

- Labels such as head, right, left arm

- Measured velocity and acceleration of movements for each part [9].


1. Conventional a. Support Vector Machinesb. Random Forestsc. k-Nearest Neighbors

2. Deep Learninga. Deep Neural Networksb. LSTM Modeling

Emotion & Affect Recognition - Techniques

19

Machine Learning


1. Conventional a. Support Vector Machinesb. Random Forestsc. k-Nearest Neighbors

2. Deep Learninga. Deep Neural Networksb. LSTM Modeling


20

Machine Learning


1. Bayesian Networks[5]

2. Probabilistic Product Rule[5]

3. Hidden Markov Models[21]


21

Probabilistic Models

Modeling Driver/Passenger Behavior According to Emotional States for In-Cabin Environment 22

1) Machine Learning

2) Variational Methods

3) Probabilistic models

4) Self Assessment

Behavior Modeling

behavior Modeling

Sensors

Emotion recognition

Interaction Domain


Behavior Modeling - Machine Learning

23

- Why?

- Personalization

- Implicit association of data

- How?

- e.g. K-means classification[1]

- Predicted trajectory sets by driver’s

mental state

- Match prediction against User Input


Behavior Modeling - Variational Methods

24

- Why?

- No overfitting

- Various methodologies available

- How?

- e.g. Model Cost functions to represent

disturbance range[11]

- Minimise Risk Cost Function

- Maximise Precision Function


Behavior Modeling - Probabilistic Models

25

- Why?

- Mimics Human decision making

- Quantify Uncertainty in actions

- How?

- e.g. Maximum Expected Utility[5]

- Calculate action which maximises utility

- Generate random actions as follow up


- Why?

- Ground Truth

- Generation of Accurate Scenario Data

- How?

- Questionnaires[11,17]

- Personal Interviews[4]

Behavior Modeling - Self Assessment (Manual)

26


Outline


27


Future Work

- Non Intrusive Ways to record/sense data

- Further explore Unsupervised Learning and Reinforcement Learning techniques

- Hybrid Integration of Environment’s input sources

- Online Learning

- Portability and Personalization

28


Outline


29


Conclusion

- Behavior modeling remains largely ambiguous

- Past research and implementations

- Machine Learning Techniques

- Variational Methods

- Probabilistic Models

- Self Assessments

- Tradeoffs

- Intrusive vs Non Intrusive measurement of data

- Generalization vs Personalization

30


Questions

31


References[1] K Driggs-Campbell, V Shia, R Bajcsy,Improved Driver Modeling for Human-in-the-Loop Vehicular Control,2015.

[2] BJ Higgs,Emotional Impacts on Driver Behavior: An Emo-Psychophysical Car-Following Model,2014.

[3] Christelle Pcher, Cline Lemercier, Jean-Marie Cellier,The Influence of Emotions on Driving Behavior,2011.

[4] Christelle Pcher, Cline Lemercier, Jean-Marie Cellier,Emotions drive attention: Effects on drivers behavior,2009.

[5] Javier G. Rzuri, Decision-making content of an agent affected by emotional feedback provided by capture of humans emotions through a Bimodal System,2010.

[6] Kamaruddin, N., Wahab, A.,Driver Behavior Analysis through Speech Emotion Understanding,2010.

[7] YH Yang, JY Liu Quantitative Study of Music Listening Behavior in a Social and Affective Context,2013.

[8] Changchun Liu, Pramila Rani, Nilanjan Sarkar An empirical study of machine learning techniques for affect recognition in human-robot interaction 2015.

[9] Ishan Behoora, Conrad S. Tucker Machine learning classification of design team members body language patterns for real time emotional state detection 2015.

[10] Katherine Driggs-Campbell, Ruzena Bajcsy Identifying Modes of Intent from Driver Behaviors in Dynamic Environments 2015.

[11] Mesut Kuscu Driver-Behavior Modeling Based on Deep Learning for Automated Emotion Recognition 2019.

[12] Vijay Govindarajan, Ruzena Bajcsy Human Modeling for Autonomous Vehicles: Reachability Analysis, Online Learning, and Driver Monitoring for Behavior Prediction 2017.

[13] S Wang, Q JiVideo affective content analysis: a survey of state-of-the-art methods 2015.

[14] M Soleymani, M Pantic, T Pun MultiModal Emotion Recognition in Response to Videos 2011.

[15] Martin Wollmer, Moritz Kaiser, Florian Eyben, Bjorn Schuller, Gerhard Rigoll LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework 2012.

[16] J Han, Z Zhang, N Cummins, F Ringeval, B Schuller Strength modelling for real-world automatic continuous affect recognition from audiovisual signals 2017.

[17] Tahir Hajizada A Behavior-based Emotion Recognition System in Intelligent Car 2018.

[18] S Mariooryad, R Lotfian, C Busso Building A Naturalistic Emotional Speech Corpus by Retrieving Expressive Behaviors From Existing Speech Corpora 2014.

[19] Y Gao, N Bianchi-Berthouze, H Meng What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay?2012.

[20] Tim Chuk , Kate Crookes , William G. Hayward , Antoni B. Chan , Janet H. Hsiao Hidden Markov model analysis reveals the advantage of analytic eye movement patterns in face recognition across cultures 2017.

32


Thank you for your time! :)

33

modeling driver/passenger behavior garching, 12.july.2019 … · 2019-07-13 · modeling...

Documents