Nao Robot as a Social Mediator

Supervisor: Asst Prof Justin DauwelsSchool of Electrical and Electronic Engineering, NTU

PhD Student: Yasir Tahir

Institute for Media Innovation (IMI), EEE, NTU

1

Co-Supervisor: Dr Daniel ThalmannInstitute for Media Innovation (IMI), NTU

Outline

SOCIOFEEDBACK SYSTEM OVERVIEW MOTIVATION

APPROACH

RESULTS

NAO as a SOCIAL MEDIATOR RELATED WORK

EXPERIMENTS

DEMO VIDEO

2

3

Sociofeedback System An Overview

Motivation When people communicate, their behavior and personality traits are

reflected in their speaking mannerism and body language.

Our objective is to analyze behavior in real-time and provide feedback based on user’s social state.

Approach We use audio-visual and depth data to extract behavioral cues. We

conduct data collection experiments and then get the recordings annotated. We then extract behavioral cues and train a machine learning classifier.

We determine levels of Dominance, Interest and Agreement and estimate social state using these values.

3

Functional Blocks of the System

4

Results

Numerical tests through the leave-one-out cross-validation indicate that the accuracy for inferring the sociometrics is in the range of 80-85%.

5

Rasheed, Umer, Yasir Tahir, Shoko Dauwels, Justin Dauwels, Daniel Thalmann, and Nadia Magnenat-Thalmann. "Real-time comprehensive sociometrics for two-person dialogs." In Human BehaviorUnderstanding, pp. 196-208. Springer International Publishing, 2013.

Possible Feedback Platforms

• Social Mediator via Nao

• Retrospective Feedback via Avatar Animation

• Socio-Feedback via Skype Application

• Socio-Feedback via Android Application

6

Feedback via Nao Robot

7

The following steps are discussed in this section:-

Related work in social robotics Concept of Nao as social mediator User studies conducted Demo video

Related Work: Human Robot Interaction

8

“Human beings seem to perceive affect from

robot motion, regardless of its shape and

outlook. Harris and Sharlin conducted a study to

understand different emotions that humans

perceive by movements of an abstract looking

robot”

Exploring the affect of abstract motion in social human-robotInteraction.J. Harris and E. Sharlin

“There is an experiment conducted to understand the affect humans elicit from the movement of a non-human like robot. The robots in this experiment were iCat robot and Roomba vacuum cleaning robot.

Perception of affect elicited by robot motion.M. Saerbeck and C. Bartneck

Can human beings identify emotions expressed by a humanoid robot using gestures? To address that question, a study was performed to explore whether humans can recognize emotions displayed by humanoid robot (Nao).”

Towards an affect space for robots to display emotional body language.A. Beck, L. CanameroInterpretation of emotional bodylanguage displayed by robots.A. Beck, A. Hiolle

Related Work: Social Robotics Applications

9

“In the recent past, many social robots

have been designed for real world

interactions e.g. Kismet, Mel, Pearl,

Robovie, and Robota , and Paro.”

Kismet: A context-dependent attention system for a social robot.Mel: A first experiment in engagement for human-robot interaction in hosting activities.Pearl: A mobile robotic assistantfor the elderly.Robovie: an interactivehumanoid robot

“Nowadays, social robots are successfully helping children in their social, emotional and communication deficits. They create interesting, appealing, and meaningful interplay situations that compel children to interact with them. One of the emerging applications of social robotics is the therapy of children with autism.”

An approach to the design of sociallyacceptable robots for children with autism spectrum disorders.K. C. Welch, U. LahiriMimicking and evaluating human motion to improve the imitation skill of children with autism through a robot.I. Fujimoto, T. Matsumoto

“Similarly, social robots are being actively deployed in nursing homes for assistance of the elderly. Those studies typically investigate what different social functions the device can play in the living environment of the elderly.”

A sociable robot to encourage social interaction among the elderlyC. D. Kidd, W. Taggart, and S. Turkle,

Nao as Social Mediator

SOCIAL MEDIATOR The role of social mediator means that Nao robot is interfaced with

sociofeedback system. The sociofeedback system analyses an ongoingconversation.

REAL-TIME FEEDBACK on social behavior Using the inference generated by sociofeedback system the robot can

provide relevant feedback using gestures and audio message.

10

Monitoring of Speakers + Feedback

Behavior Detection: Dominance, Interest, Discord, Consistency,

Mirroring etc

Feature Extraction

Visual Cues

Speech Cues

Audio and Video Data Acquisition

Feedback based on

SocialBehavior

Please carry on!Kindly wait for your

turn…

Please speak louder…

User Studies

12

So far we have conducted two user studies:-

First one to get user opinion about the feedback being delivered. Also to determine the most appropriate way of delivering feedback.

In the second study we interfaced Nao with sociofeedback system and asked participants to interact in certain manner. User speech was analysed and feedback was delivered.

User Study for appropriate Feedback

13

We called 20 people to receive feedback from Nao robot.

The participants did not have to participate in a conversation.

We provided feedback with only audio, only gestures and then by combining audio along with gestures.

The participants first identified the feedback and then rated the robot on Godspeed questionnaire.

• Yasir Tahir, Umer Rasheed, Keng Hui, Shoko Dauwels, Justin Dauwels. "`Nao as social mediator: A user study." In Robots in public spaces:towards multi-party, short-term, dynamic human-robot interaction, 2013.

Godspeed Questionnaire

14

AnthropomorphismMachinelike 1 2 3 4 5 HumanlikeMoving rigidly 1 2 3 4 5 Moving elegantly

AnimacyMechanical 1 2 3 4 5 OrganicInert 1 2 3 4 5 Interactive

LikabilityDislike 1 2 3 4 5 LikeUnfriendly 1 2 3 4 5 Friendly

Perceived IntelligenceIgnorant 1 2 3 4 5 KnowledgeableUnintelligent 1 2 3 4 5 Intelligent

Perceived SafetyCalm 1 2 3 4 5 AgitatedQuiescent 1 2 3 4 5 Surprised

• Bartneck, Christoph, et al. "Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots." International Journal of Social Robotics 1.1 (2009): 71-81.

Results

15

Speak Louder

Too noisy

Stop Interrupting

Calm down

Slowdown

Uninterested

Audio 84.2 68.4 89.4 89.4 89.4 78.9

Gestures 63.1 15.7 21 57.8 31.5 36.8

Combined 84.2 100 100 94.7 100 100

Results

16

Characteristics P-values Averagevalue (audio)

Average value(combined)

AnthropomorphismMachine/human like

0.017 2.94 3.52

Moving rigidly/elegantly 0.002 2.36 3.26

AnimacyMechanical/organic

0.129 2.89 3.26

Inert/interactive 0.046 3.15 3.63

LikabilityDislike/like

0.110 4.26 4.63

Unfriendly/friendly 0.008 3.73 4.26

Perceived IntelligenceIgnorant/knowledgeable

0.009 3.31 3.63

Unintelligent/intelligent 0.186 3.42 3.57

Perceived SafetyCalm/agitated

0.741 2.36 2.26

Quiescent/surprised 1.00 2.84 2.84

Human Perception of Nao as social mediator

17

We called 20 people to receive feedback from Nao robot.

The participants engaged in a conversation. The conversation was analyzed in real time and then feedback was provided at the end.

Based on the conclusions of previous studies we provided feedback using audio and gestures.

The participants rated the experiment on two questionnaires. One questionnaire was to rate the feedback and the second one to rate human perception using Godspeed questionnaire.

• Yasir Tahir, Umer Rasheed, Shoko Dauwels, Justin Dauwels. "`Perception of Humanoid Social Mediator in Two-Person Dialogs." Accepted as Late braking report at Human Robot Interaction (HRI 2014)

Questionnaire for Feedback assessment

18

Questions Average

valueQ1 Did you notice when the socio-feedback system was addressing you? 4.5

Q2 Did you notice when the socio-feedback system was addressing others? 4.5

Q3 Was the timing of socio-feedback appropriate? 3.3

Q4 Did the socio-feedback system interrupt the conversation? 2.3

Q5 Was the interaction natural? 4.2

Q6 Did you understand the message given by the socio-feedback? 4.8

Q7 Do you agree with the given feedback? 4.7

Q8 Did you enjoy using the socio-feedback system? 4.8

Demo Video

19

Q & A

Q&A

20