socially intelligent robots cynthia breazeal mit media lab robotic life group cynthia breazeal mit...

Socially Intelligent RobotsSocially Intelligent Robots

Cynthia BreazealMIT Media Lab

Robotic Life Group

Cynthia BreazealMIT Media Lab

Robotic Life Group

Robots turn 85 years oldPosted May 31st 2006 11:38PM by Ryan BlockFiled under: RobotsDear Robots,

We're very sorry. It appears we missed your 85th birthday two days ago -- the anniversary of which is marked by the date Czech writer Karel Capek debuted his play R.U.R. (Rossum's Universal Robots) to its first audience in Prague. Yes, we know the concept of the automaton dates back much further, but we think it's well agreed upon that Capek's play marks the robot's entry into mass consciousness (as well as marking the first use of the word "robot"). No matter, we're just saying happy birthday, robots -- not because we fear you'll one day you'll subsume us in some dystopian nightmare of artificial intelligence gone terribly wrong, but because from Asimov to AIBO, from Roomba to Ri-Man, from QRIO to ASIMO, we just love ya. So happy birthday, happy birthday, happy birthday, robots, and when the day of reckoning comes, please remember: Engadget and its readers are your friends.

All our love,Engadget

Robots have… explored ocean depths,mapped subterranean mines, rescued natural disaster victims,assisted surgeons with operations,driven autonomously across the desert,

And even been to Mars…

Robots have… explored ocean depths,mapped subterranean mines, rescued natural disaster victims,assisted surgeons with operations,driven autonomously across the desert,

And even been to Mars…

What’s Next?What’s Next?

The next big frontier…society at largeThe next big frontier…society at large

Everyday Life with People and Robots

…and its implication for design

Everyday Life with People and Robots

…and its implication for design

People and RobotsPeople and Robots

Robots are not perceived as pure tools or appliances, but often as social actors -- over a wide range of morphologies and behaviors

Robots are not perceived as pure tools or appliances, but often as social actors -- over a wide range of morphologies and behaviors

Robots Evoke Human Social Responses

Robots Evoke Human Social Responses

New Scientist, 2005

“The Kismet Effect”

Newsmaker: My friend, the robot

CNET news.com, May 24, 2006

Newsmaker: My friend, the robot

CNET news.com, May 24, 2006

The PackBots have almost become members of military units, Angle said, recalling an incident when a U.S. soldier begged iRobot to repair his unit's robot, which they had dubbed Scooby Doo. "Please fix Scooby Doo because he saved my life," was the soldier's plea, Angle told the Future in Review conference last week in Coronado, Calif. For many reasons, people bond with robots in a way they don't bond with their lawn mowers, televisions or regular vacuum cleaners. At some point, this could help solve the looming health care problem caused by an enormous generation of aging people. Not only could robots make sure they take their medicine and watch for early warning signs of distress, but they could also provide a companion for lonely people and extend their independence.

Social Robots Socio-emotive Factors

Social Robots Socio-emotive Factors

Interactive Toys

NEC “babysitters” OMRON “pets”

BANDAI “elder toys”Professional ServiceRobots

the socio-emotive and psychologicalaspects of people, in long-term relations

Future applicationsrequire robots to address

“Social as interface”

“Social as entertainment”

“Social as relationship”

HRI, An Emerging DisciplineHRI, An Emerging Discipline

An important goal of Human-Robot Interaction (HRI) is synergy of the human-robot system. Robots bring their own abilities that complement human strengths. It is not about equivalence (replacement), but compatibility with a typical human partner

Lastin

g R

ela

tion

ship

Four Cornerstones of Social Robotics in HRI

Four Cornerstones of Social Robotics in HRI

Team

work

Socia

l Learn

ing

Socia

l Inte

lligen

ceInterdependence

Transparent Communication

Cognitive Compatibility

Perspective Taking

User Studies,Psychology &Social Development

Today’s FocusToday’s Focus

Robots, like humans, should leverage the social and environmental constraints in the real world to foster learning new skills and knowledge from anyone.

Animal training techniques{Stern, Frank, Resner, Virtual Petz, Agents 1998}{Blumberg et al. Integrated learning for interactive characters,

SIGGRAPH 2002}{Kaplan et al., Robot clicker training, RAS 2002}

Reinforcement Learning with humans{Isbell et al. Cobot: a social reinforcement learning agent, UAI

1998}{Evans, Varieties of Learning, AI Game Programming Wisdom,

2002}{Clouse, Utgoff, Teaching a Reinforcement Learner, ICML 1992}

Active Learning, Learning with Queries{Cohn, Ghahramani, Jordan, Active learning with statistical

models, 1995}{Cohn et al., Semi-supervised clustering with user feedback,

2003}

Personalization agents, Adaptive user interfaces

{Lashkari, Metral, Maes, Collaborative Interface Agents, AAAI 1994}

{E. Horovitz et al., The Lumiere project, UAI 1998}

Learning by Demonstration, Programming by Example

{Voyles, Khosla, Programming robotic agents by demonstration, 1998}

{Lieberman, Your Wish is my Command, 2001}{A. Billard, Special Issue of RAS on Robot Programming by

Demonstration, 2006}

Learning by Imitation {S. Schaal review in TICS 1999} {K. Dautenhahn & C. Nehaniv, Imitation in Animals and Artifacts,

2002}

… and many more

Most people don’t have experience with Machine Learning techniques, they have a lifetime of experience with social learning interactions that they bring to the table.

We emphasize the need to consider and design to support the ways that people naturally approach teaching.

And then design algorithms and systems that take better advantage of this

How Do Ordinary People Teach a RL Agent?

How Do Ordinary People Teach a RL Agent?

Experiments inSophie’s KitchenExperiments in

Sophie’s Kitchen

QuickTime™ and aAnimation decompressor

are needed to see this picture.

A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface.

A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface.

Sophie learns via Q-Learning

30 steps~10,000 states2-7 actions/state

Allows us to run many subjects on-line

Experiments inSophie’s KitchenExperiments in

Sophie’s Kitchen

A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface.

A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface.

QuickTime™ and aAnimation decompressor

are needed to see this picture.

An object specific reward is about a particular part of the world

Initial ExperimentInitial ExperimentThomaz & Breazeal RO-MAN 2006

18 people trained Sophie They are given a description of the cake task,

and told they can’t do actions but can help Sophie by sending FEEDBACK messages with the mouse

System logs time of state changes, agent actions, and any human feedback. We analyze games logs to understand people’s teaching behavior

18 people trained Sophie They are given a description of the cake task,

and told they can’t do actions but can help Sophie by sending FEEDBACK messages with the mouse

System logs time of state changes, agent actions, and any human feedback. We analyze games logs to understand people’s teaching behavior

Findings: GuidanceFindings: Guidance

People tried to use the object specific rewards as FUTURE directed guidance.

People tried to use the object specific rewards as FUTURE directed guidance.

QuickTime™ and aAnimation decompressor

are needed to see this picture.

0 20 40 60 80 100%%%%%

Each player’s %Object Rewards about last object

Never About Most Recent Object

Always About Most Recent Object

Many object rewards not about the last object usedMany object rewards not about the last object used

0

2

4

6

8

10

12

14

16

Number of People

Zero rewardsto Empty Bowl

At least 1 rewardto Empty Bowl

Almost everyone gave rewards to the bowl or tray sitting empty on the shelf...a guidance reward.

Almost everyone gave rewards to the bowl or tray sitting empty on the shelf...a guidance reward.

Findings: People Adapt Teaching to their Mental

Model of Sophie

Findings: People Adapt Teaching to their Mental

Model of Sophie

People gave more rewards after realizing their feedback made a difference

Interpreted Sophie’s behavior as being a “staged” learner

Adapted their teaching strategy accordingly

People gave more rewards after realizing their feedback made a difference

Interpreted Sophie’s behavior as being a “staged” learner

Adapted their teaching strategy accordingly

human rewards : agent actions

(Avg)Individual (Avg)IndividualIndividual (Avg)

Initial Experiment

Transparency

Asymmetry

Guidance

}

Using Guidance in Sophie’s Kitchen

Using Guidance in Sophie’s Kitchen

Interactive Q-Learning Algorithm, baseline system}

slight delay to animate act and receive human reward

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

Using Guidance in Sophie’s Kitchen

Using Guidance in Sophie’s Kitchen

QuickTime™ and aAnimation decompressor

are needed to see this picture.

GuidanceExperimentGuidance

ExperimentThomaz & Breazeal, AAAI 2006

Hypothesis: Non-expert teachers can use guidance to improve agent’s performance

27 subjects trained Sophie in two groups:Using feedback only Using both feedback and guidance

Again, system logs game play and logs are analyzed to understand teaching behavior

Hypothesis: Non-expert teachers can use guidance to improve agent’s performance

27 subjects trained Sophie in two groups:Using feedback only Using both feedback and guidance

Again, system logs game play and logs are analyzed to understand teaching behavior

Effects of GuidanceEffects of Guidance

+ >> only

1-tailed T-tests show logs in guidance condition are significantly better than non-guidance 1-tailed T-tests show logs in guidance condition are significantly better than non-guidance

feedback only

guidance + feedback

effect size

Number of Trials 28.5 14.6 49%

Number of Actions 816.4 368 55%

Number of Failures 18.89 11.8 38%

Number Fails before 1st Goal

18.7 11 41%

Number Unique States Visited

124.44 62.7 50%

Initial Experiment

Transparency

Asymmetry

Guidance

}

TransparencyTransparency

How can machine learners be Transparent? How can machine learners be Transparent?

Teachers structure the environment and the task to help a learner succeed. Learners contribute by revealing internal state; helping the teacher maintain a mental model to make guidance more appropriate.

Sophie’s Gaze BehaviorSophie’s Gaze Behavior

Interactive Q-Learning Algorithm modified toincorporate Guidance

Sophie’s Gaze BehaviorSophie’s Gaze Behavior

QuickTime™ and aAnimation decompressor

are needed to see this picture.

TransparencyExperiment

TransparencyExperiment

52 subjects trained Sophie in an online version:

Feedback and guidance, no gaze Feedback and guidance, Sophie gazing

Hypothesis:

Learners can help shape their learning environment by communicating aspects of the internal process -- gaze will improve the human’s guidance instruction

52 subjects trained Sophie in an online version:

Feedback and guidance, no gaze Feedback and guidance, Sophie gazing

Hypothesis:

Learners can help shape their learning environment by communicating aspects of the internal process -- gaze will improve the human’s guidance instruction

Thomaz et al., ICDL 2006

Sophie’s Gaze BehaviorSophie’s Gaze Behavior

Results: Sophie’s gaze significantly improves the guidance received - more when uncertainty high and less when uncertainty is low.

Uncertainty high: 3 or more choices

Uncertainty low: 3 or less

0

10

20

30

40

50

60

70

80

90

uncertainty low uncertainty high

gaze

no-gaze

LessonsLessonsPeople bring their own teaching and learning experience to the task

Social factors of guidance and transparency

Collaborative process between teacher and learner improves performance

Agent can use transparency cues to improve its own learning environment by helping teacher form a better mental model

Adding gaze significantly improves the human’s Guidance

SummarySummary