Improving Improving Human-Robot InteractionHuman-Robot Interaction

Jill Drury, The MITRE CorporationJill Drury, The MITRE Corporation

Collaborators:Collaborators:

Holly Yanco, UMass LowellHolly Yanco, UMass LowellJean Scholtz, NISTJean Scholtz, NIST

Mike Baker, Bob Casey, Dan Hestand, Mike Baker, Bob Casey, Dan Hestand, Brenden Keyes, Phil Thoren (UML)Brenden Keyes, Phil Thoren (UML)

Methodology for Evaluating HRIMethodology for Evaluating HRI

Two approaches: field and laboratoryTwo approaches: field and laboratoryField work: so far, robotics competitionsField work: so far, robotics competitions See many different user interfaces but have no See many different user interfaces but have no

control over what operator doescontrol over what operator does Difficult to collect dataDifficult to collect data Can see what they did – but there isn’t time to Can see what they did – but there isn’t time to

determine whydetermine why Best used to get an idea of the difficulties in the real Best used to get an idea of the difficulties in the real

worldworld Can identify “critical events” but don’t know for certain Can identify “critical events” but don’t know for certain

whether operator was aware of themwhether operator was aware of them

Methodology for Evaluating HRIMethodology for Evaluating HRI

Laboratory studiesLaboratory studies Take what we learned in the real world and Take what we learned in the real world and

isolate factors to determine effects isolate factors to determine effects Repeatability is still difficult to achieve due to Repeatability is still difficult to achieve due to

fragile nature of robotsfragile nature of robots

Analysis FrameworksAnalysis Frameworks

Taxonomy to define human/robot systemTaxonomy to define human/robot system

Detailed definition of human-robot Detailed definition of human-robot interaction awarenessinteraction awareness

Coding scheme/metrics for analyzing dataCoding scheme/metrics for analyzing data

Scholtz’ evaluation guidelinesScholtz’ evaluation guidelines

TaxonomyTaxonomyAutonomyAutonomy

Amount of interventionAmount of intervention

Human-robot ratioHuman-robot ratio

Level of shared interactionLevel of shared interaction

Composition of robot teamsComposition of robot teams

Available sensorsAvailable sensors

Sensor FusionSensor Fusion

CriticalityCriticality

TimeTime

SpaceSpace

Awareness as a concept from Awareness as a concept from CSCWCSCW

CSCW software:CSCW software: ““...makes the user aware that ...makes the user aware that

he is part of a group, while he is part of a group, while most other software seeks to most other software seeks to hide and protect users from hide and protect users from each other” [Lynch et al. 1990]each other” [Lynch et al. 1990]

HRI software:HRI software: Makes humans aware of Makes humans aware of

robots’ status and activities via robots’ status and activities via the interfacethe interface

What is “awareness”?What is “awareness”?

No standard definition in the CSCW fieldNo standard definition in the CSCW field We’ve seen at least 16 different definitions!We’ve seen at least 16 different definitions!

Are many different types of awareness, e.g.Are many different types of awareness, e.g. Concept awarenessConcept awareness Conversational awarenessConversational awareness Group-structural awarenessGroup-structural awareness Informal awarenessInformal awareness Peripheral awarenessPeripheral awareness

Common thread: understanding that participants have of Common thread: understanding that participants have of each other in a shared environmenteach other in a shared environment

Situation awarenessSituation awareness Social awarenessSocial awareness Task awarenessTask awareness Workspace Workspace

awarenessawareness

But CSCW systems are But CSCW systems are different from robotic systems…different from robotic systems…

CSCW:CSCW: Multiple humans interacting via a CSCW systemMultiple humans interacting via a CSCW system

Robotics:Robotics: Single or multiple humans interacting with a single or Single or multiple humans interacting with a single or

multiple robotsmultiple robots Non-symmetrical relationships between humans and Non-symmetrical relationships between humans and

robots; e.g., differences in robots; e.g., differences in Free willFree will

CognitionCognition

Tailoring an awareness Tailoring an awareness definition for HRI: a base casedefinition for HRI: a base caseGiven one human and one robot...Given one human and one robot...

... HRI awareness is the understanding that the human ... HRI awareness is the understanding that the human has of the has of the location, location, activities, activities, status, and status, and surroundings of the robot; and surroundings of the robot; and

the knowledge that the robot has ofthe knowledge that the robot has of the human’s commands necessary to direct its activities and the human’s commands necessary to direct its activities and the constraints under which it must operatethe constraints under which it must operate

An awareness framework: An awareness framework: General caseGeneral case

Given n humans and m robots working together Given n humans and m robots working together on a synchronous task, HRI awareness consists on a synchronous task, HRI awareness consists of five components:of five components: Human-robot awarenessHuman-robot awareness Human-human awarenessHuman-human awareness Robot-human awarenessRobot-human awareness Robot-robot awarenessRobot-robot awareness Humans’ overall mission awarenessHumans’ overall mission awareness

General case: a detailed lookGeneral case: a detailed look

Given n humans and m robots working together Given n humans and m robots working together on a synchronous task, HRI awareness consists on a synchronous task, HRI awareness consists of five components:of five components: Human-robotHuman-robot: the understanding that the humans : the understanding that the humans

have of the locations, identities, activities, status and have of the locations, identities, activities, status and surroundings of the robots. Further, the surroundings of the robots. Further, the understanding of the certainty with which humans understanding of the certainty with which humans know this information.know this information.

Human-humanHuman-human: the understanding that the humans : the understanding that the humans have of the locations, identities and activities of their have of the locations, identities and activities of their fellow human collaboratorsfellow human collaborators

General case, concludedGeneral case, concluded Robot-humanRobot-human: the robots’ knowledge of the humans’ : the robots’ knowledge of the humans’

commands needed to direct activities and any human-commands needed to direct activities and any human-delineated constraints that may require command delineated constraints that may require command noncompliance or a modified course of actionnoncompliance or a modified course of action

Robot-robotRobot-robot: the knowledge that the robots have of the : the knowledge that the robots have of the commands given to them, if any, by other robots, the commands given to them, if any, by other robots, the tactical plans of the other robots, and the robot-to-robot tactical plans of the other robots, and the robot-to-robot coordination necessary to dynamically reallocate tasks coordination necessary to dynamically reallocate tasks among robots if necessary.among robots if necessary.

Humans’ overall mission awarenessHumans’ overall mission awareness: the humans’ : the humans’ understanding of the overall goals of the joint human-robot understanding of the overall goals of the joint human-robot

activities and the measurement of the moment-by-momentactivities and the measurement of the moment-by-moment progress obtained against the goalsprogress obtained against the goals..

Coding Scheme: Problems Coding Scheme: Problems Relating to Critical IncidentsRelating to Critical Incidents

Critical incident: Robot has, or could, cause Critical incident: Robot has, or could, cause harm or damage harm or damage

Types of problems:Types of problems: Local navigationLocal navigation Global navigationGlobal navigation Obstacle encounterObstacle encounter Vehicle stateVehicle state Victim identification (specific to search and Victim identification (specific to search and

rescue)rescue)

Some Metrics for HRISome Metrics for HRI

Time spent navigating, on UI overhead and Time spent navigating, on UI overhead and avoiding obstaclesavoiding obstaclesAmount of space coveredAmount of space coveredNumber of victims foundNumber of victims foundCritical incidentsCritical incidents Positive outcomesPositive outcomes Negative outcomesNegative outcomes

Operator interventionsOperator interventions Amount of time robot needs helpAmount of time robot needs help Time to acquire situation awarenessTime to acquire situation awareness Reason for interventionReason for intervention

Scholtz’ Guidelines (tailored)Scholtz’ Guidelines (tailored)Is sufficient status and robot location information available so that the operator knows the robot is operating correctly and avoiding obstacles?

Is the information coming from the robots presented in a manner that minimizes operator memory load, including the amount of information fusion that needs to be performed in the operators’ heads?

Are the means of interaction provided by the interface efficient and effective for the human and the robot (e.g., are shortcuts provided for the human)?

Does the interface support the operator directing the actions of more than one robot simultaneously?

Will the interface design allow for adding more sensors and more autonomy?

Design GuidelinesDesign Guidelines

Enhance awarenessEnhance awareness Provide a map of where the robot has beenProvide a map of where the robot has been Provide more spatial information about the robot in Provide more spatial information about the robot in

the environment to make the operators more award of the environment to make the operators more award of their robot’s immediate surroundingstheir robot’s immediate surroundings

Lower cognitive loadLower cognitive load Provide fused sensor information to avoid making the Provide fused sensor information to avoid making the

user fuse data mentallyuser fuse data mentally Display important information near or fused with the Display important information near or fused with the

video imagevideo image

Design Guidelines, concludedDesign Guidelines, concluded

Increase efficiencyIncrease efficiency Provide user interfaces that support multiple Provide user interfaces that support multiple

robots in a single window, if possiblerobots in a single window, if possible In general, minimize the use of multiple In general, minimize the use of multiple

windows and maximize use of the primary windows and maximize use of the primary viewing areaviewing area

Provide help in choosing robot modalityProvide help in choosing robot modality Give the operator assistance in determining Give the operator assistance in determining

the most appropriate level of robot autonomy the most appropriate level of robot autonomy at any given timeat any given time

Fusing InformationFusing Information

Victims can be missed in video imagesVictims can be missed in video images

Fusing Infrared and Color VideoFusing Infrared and Color Video

Fusing Infrared and Color VideoFusing Infrared and Color Video

Fusing Infrared and Color VideoFusing Infrared and Color Video

Other Sensor Modalities for USAROther Sensor Modalities for USAR

COCO22 detection detection

AudioAudio

Overlay of four sensor modalitiesOverlay of four sensor modalities

Overlay of four sensor modalitiesOverlay of four sensor modalities