Sarah Cosentino, Ph.D.

Graduate School of

Advanced Science and Engineering

Waseda University, Tokyo

Interactive Musical Humanoid Robots – seen by an European researcher in Japan

EUROPA HOUSE, TOKYO, JAPAN

DECEMBER, 11, 2015

OUTLINE

1 . M Y R ES EA R C H WO R K • musical interacting robots

2 . M Y E X P E R I E N C E I N J A PA N • why, how

• grants

• plus and minus of international mobility

• personal feelings, working atmosphere

3 . M Y R E L AT I O N TO E U RO P E W H I L E I N J A PA N • cooperation going on, or planned

• next employment

• short comparison between work environments in Japan and Europe

EUROPA HOUSE, TOKYO, JAPAN

DECEMBER, 11, 2015

3

MY RESEARCH – MUSICAL PERFORMANCE ROBOTS

WASEDA University – Flutist humanoid robot WF4

4

ENTERTAINMENT INDUSTRY

Entertainment

• provides amusement or enjoyment

• holds the attention of an audience

• intrinsically motivating

Social interaction = Entertainment • Users not satisfied with the role of pure observer

• want to be involved and have an influencing role

Applications: • Educational

• Healthcare – Rehabilitation

– Psychological support

[1] Oxford English dictionaries

[2] J. Bryant and P. Vorderer (2006)

[3] E. Moyer-Gusé (2008)

[4] A. Silayan-Go (1990)

[5] E. Flores, et al. (2008)

[6] D. Zillmann (1988)

[7] Statista (online, 2013)

[1]

[2]

[3]

[4]

[5]

[6]

Estimated global entertainment market value from 2014 to 2018 (in trillion U.S. Dollars)

1.9 2.0 2.1 2.2 2.3

2014 2015 2016 2017 2018

5

ENTERTAINMENT ROBOTICS

Problems

1. Non-humanlike interaction abilities

2. No feedback from audience

3. Fixed performance

Aldebaran – Nao

Toyota – Trumpet robot

Flutist WF-4RVI

KOBIAN

WASEDA entertainment robots

Saxophonist WAS-3

GeorgiaTech – Marimba

[1] Camprodon (2015), Gruebler(2011)

[2] Toyota partner robot (2005)

[3] Cicconet (2012, 2013)

[4] Petersen, et al. (2010)

[5] Solis (2012)

[6] Kishi (2013)

[1]

[3]

[2] [4]

[5]

[6]

•Vocal commands recognition few languages •Wave and Point recognition rate ~ 50% • Smile and Frown recognition rate ~ 100% but

10s continuous signal non-natural

•No interaction •Coordination

signals expression

•Creative interaction with visual signals recognition

• Facial and body emotional expression

6

HUMAN-HUMAN INTERACTION

Human social signals

Conscious

signals Unconscious

signals

Super conscious interaction

NATURAL INTERACTION

Human interaction: 1. MULTIMODAL 2. CONSCIOUS and UNCONSCIOUS 3. ADAPTIVE - changes depending on feedback

Humans Media: • Social and natural • Like interaction in real life [1] [1] Nass and Reeves (1996)

7

RESEARCH FOCUS

Problem 1: non-humanlike interaction abilities • Limited to simple direct verbal commands Solution

– Complex language

Problem 2: not multimodal • Limited to one communication channel Solution

– Several simultaneous communication channels

Problem 3: no emotional adaptation • Limited to conscious, direct commands Solution

– Emotional communication of the human partner

8

Non verbal communication

Verbal communication

HUMAN COMMUNICATION ANALYSIS

Conscious signals

Unconscious signals

Kinesics .

Vocal paralanguage

Codified gestures

Emotional

Creative gestures

1. MULTIMODAL 2. CONSCIOUS and UNCONSCIOUS 3. ADAPTIVE - changes depending on feedback

9

COGNITIVE AND EMOTIONAL HUMAN INTERACTION

MULTIMODAL Problem: synchronized detection and measurement

Unconscious

[unaware] [uncontrolled]

Conscious

[aware] [active control]

[invisible] [internal] Physiological

Interaction

Face expression

Vocal emission

Body movement

Respiration

Brain activity

Heart rate

Skin conductance

Blood pressure

Physical [external] [visible]

10

HUMAN PARAMETERS SENSING TECHNOLOGIES

Interaction

Face expression

Vocal emission

Body movement

Respiration

Brain activity

Heart rate

Skin conductance

Blood pressure Camera

Microphone

IMU

EMG

Physiological

Physical

• Audiovisual sensors PROS Low cost Background Ecological

CONS Not robust Multiple tracking Noise affected

• Wearable sensors PROS Multiple tracking High precision Light computation Robust

CONS Not portable Expensive Non ecological Difficult to integrate

11

HUMAN PARAMETERS MONITORING SYSTEMS

• Development of a multi-sensor system to obtain real-time objective data on human state:

– Inexpensive – compared to current solutions

– Portable – Untethered and lightweight, allows ecological human data monitoring

– Reconfigurable – Use interchangeable modular sensors that can be added or removed as needed.

12

REAL-TIME NON-VERBAL ARTISTIC INTERACTION

13

DIRECT INTERACTION

Copyright

Takanishi Laboratory

14

INDIRECT INTERACTION

15

CREATIVE INTERACTION

Copyright

Takanishi Laboratory Copyright

Takanishi Laboratory

16

PERCEPTION RESULTS

*** : p<0.005

***

Result of questionnaire

• Perception survey

– questionnaire Likert scale 10 points

• Performance comparison

– robot playing “as is”

– robot following the artist using the interaction system

– professional musician following the artist

*** *** ***

17

DIRECT HUMAN COMMUNICATION SIGNALS

Problem 1: non-humanlike interaction abilities Experiment 1: Direct signaling gestures Experiment 2: Indirect conducting gestures – Complex non-verbal language: codified, symbolic

• Learn and recognize

The system can be used to detect and recognize human conscious gestural communication

SUCCESS FAILURE

– Non-verbal language recognition – Real-time interaction – No equal previous system

– Performance perceived not completely natural

– Language not completely modelled

18

CREATIVE HUMAN COMMUNICATION SIGNALS

Problem 2: not multimodal

Problem 3: no emotional adaptation Experiment 3: Dance interaction – Two channels: body posture and movement velocity – Creative gestures: not codified, intuitive

• Perceive and interpret

The system can be used to detect and recognize human emotional creative communication

SUCCESS FAILURE

– Multimodal non-verbal social communication

– Real-time interaction

– Performance perceived not completely natural

– Language not completely modelled

19

RESEARCH ACHIEVEMENTS 1

[1] S. Cosentino, K. Petersen, Z. Lin, L. Bartolomeo, S. Sessa, M. Zecca, and A. Takanishi, “Natural human–robot musical interaction: understanding the music conductor gestures by using the WB-4 inertial measurement system,” Advanced Robotics, pp. 1–12, 2014.

[2] T. Kishi, H. Futaki, G. Trovato, N. Endo, M. Destephe, S. Cosentino, K. Hashimoto, and A. Takanishi, “A Robotic Head that Displays Japanese ‘Manga’ Marks,” in Advances on Theory and Practice of Robots and Manipulators, vol. 22, M. Ceccarelli and V. A. Glazunov, Eds. Cham: Springer International Publishing, 2014, pp. 245–253.

[3] D. Zhang, S. Sessa, W. Kong, S. Cosentino, D. Magistro, H. Ishii, M. Zecca, and A. Takanishi, “Development of subliminal persuasion system to improve the upper limb posture in laparoscopic training: a preliminary study,” Int J CARS, pp. 1–9, Apr. 2015.

Journal articles (3):

International conferences (16). Main papers: [4] S. Cosentino, Y. Sugita, M. Zecca, S. Sessa, Z. Lin, K. Petersen, L. Bartolomeo, H. Ishii, K. Saito, and A. Takanishi,

‘Musical robots: towards a natural joint performance’, presented at the ICIES 2012 - International Conference on Innovative Engineering Systems, Alexandria, Egypt, 2012, pp. 19 – 24.

[5] S. Cosentino, Y. Sugita, M. Zecca, S. Sessa, Z. Lin, K. Petersen, H. Ishii, and A. Takanishi, ‘Music conductor gesture recognition by using inertial measurement system for human-robot musical interaction’, presented at the IEEE ROBIO 2012 - International Conference on Robotics and Biomimetics, Guangzhou, China, 2012, pp.

30–35. Best Student Paper Award

[6] S. Cosentino, T. Kishi, M. Zecca, S. Sessa, L. Bartolomeo, K. Hashimoto, T. Nozawa, and A. Takanishi, ‘Human-humanoid robot social interaction: Laughter’, in 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 1396–1401.

[7] T. Kishi, T. Kojima, N. Endo, M. Destephe, T. Otani, L. Jamone, P. Kryczka, G. Trovato, K. Hashimoto, S. Cosentino, and others, ‘Impression survey of the emotion expression humanoid robot with mental model based dynamic emotions’, in Robotics and Automation (ICRA), 2013 IEEE InternationalConference on, 2013, pp. 1663–1668.

[8] T. Kishi, N. Endo, T. Nozawa, T. Otani, S. Cosentino, M. Zecca, K. Hashimoto, and A. Takanishi, ‘Bipedal humanoid robot that makes humans laugh with use of the method of comedy and affects their psychological state actively’, in Robotics and Automation (ICRA), 2014 IEEE InternationalConference on, 2014, pp. 1965–1970.

[9] T. Kishi, H. Futaki, G. Trovato, N. Endo, M. Destephe, S. Cosentino, K. Hashimoto, and A. Takanishi, ‘Development of a comic mark based expressive robotic head adapted to Japanese cultural background’, in Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on, 2014, pp. 2608–2613.

Best paper award 2012 IEEE International

Conference on Robotics and Biomimetics - ROBIO

20

RESEARCH ACHIEVEMENTS 2

[1] S. Cosentino, “Human-robot laughter interaction”, 2014 ITALY-JAPAN workshop – Robot

coworker, Waseda University, Tokyo, Japan. Student competition finalist Award

Workshop (1):

Invited Talks (6):

[1] Advanced Human-Robot Interaction: goals, challenges and approaches. Invited talk at University of Genova – DIBRIS laboratory, Genova, ITALY, February, 2015

[2] Advanced Human-Robot Interaction: goals, challenges and approaches. Invited talk at Facebook Inc. HQ, Menlo Park, Palo Alto, USA, October, 2014

[3] Advanced Human-Robot Interaction: goals, challenges and approaches. Invited talk at Aix-Marseille Universite’ Institut de Neurosciences de la Timone, Marseille, FRANCE, September 2014

[4] Advanced Human-Robot Interaction: goals, challenges and approaches. Invited talk at University of Birmingham IRLAB – Intelligent Robotics Lab, Birmingham, UK, September 2014

[5] Advanced Human-Robot Interaction: goals, challenges and approaches. Invited talk at University of Genova – DIBRIS laboratory Genova, ITALY, February 2015

[6] WAraBLE Waseda Bioinstrumentation for Laughter and Emotions. Invited talk at Carnegie Mellon University – InterACT laboratory, Pittsburgh, PA, USA, March 2014

Student competition finalist award

2014 ITALY-JAPAN workshop robot coworker

OUTLINE

1 . M Y R ES EA R C H WO R K • musical interacting robots

2 . M Y E X P E R I E N C E I N J A PA N • why, how

• grants

• plus and minus of international mobility

• personal feelings, working atmosphere

3 . M Y R E L AT I O N TO E U RO P E W H I L E I N J A PA N • cooperation going on, or planned

• next employment

• short comparison between work environments in Japan and Europe

EUROPA HOUSE, TOKYO, JAPAN

DECEMBER, 11, 2015

22

WHY JAPAN

• ROBOTICS!!!

23

HOW JAPAN – INTERNATIONAL EXCHANGE PROGRAMS

• Vulcanus in Japan program

– http://www.eu-japan.eu/vulcanus-japan-0

24

HOW JAPAN – GRANTS

• StudyJapan: general information to study in Japan

– http://www.studyjapan.go.jp/en/

• JASSO: information on scholarships to study in Japan

– http://www.jasso.go.jp/study_j/scholarships_e.html

• MEXT and JSPS scholarships

– Information on the website of Japanese embassy in one own country

– https://www.jsps.go.jp/english/programs/index.html

• EURAXESS: general information for researchers mobility

– http://ec.europa.eu/euraxess/

25

PLUS AND MINUS OF INTERNATIONAL MOBILITY

• PLUS

• MINUS

– ?

Stability

Meaningful cultural exchanges Broader knowledge

Exciting life experiences

26

PERSONAL FEELINGS

• Ph.D. in Japan

– Independent work

– Exciting and interesting

– Frustrating and insane

• Working atmosphere

– It depends on the environment and coworkers

– It depends on one’s own disposition

OUTLINE

1 . M Y R ES EA R C H WO R K • musical interacting robots

2 . M Y E X P E R I E N C E I N J A PA N • why, how

• grants

• plus and minus of international mobility

• personal feelings, working atmosphere

3 . M Y R E L AT I O N TO E U RO P E W H I L E I N J A PA N • cooperation going on, or planned

• next employment

• short comparison between work environments in Japan and Europe

EUROPA HOUSE, TOKYO, JAPAN

DECEMBER, 11, 2015

28

RELATION TO EUROPE WHILE IN JAPAN

• Cooperation

– Research IS cooperation

– Cooperation is networking

– Networking might result in interesting offers

• Comparison

– Japan

• More precise

• Stricter

– Europe

• More easy-going

• More tutored

THANK YOU VERY MUCH FOR

YOUR ATTENTION!

EUROPA HOUSE, TOKYO, JAPAN DECEMBER, 11, 2015