Human-in-the-Loop Planning and

Decision Support

Subbarao KambhampatiArizona State University

1

CSIG Speaker Series

Adapted from a AAAI 2015 Tutorialrakaposhi.eas.asu.edu/hilp-tutorial/index.htm

Kartik TalamadupulaIBM T.J. Watson Research Center

AI’s Curious Ambivalence to Humans..

You want to help humanity, it is the people that you just can’t stand…

• Our systems seem happiest ..

• …either far away from humans

• …or in an adversarial stance with humans

What happened to Co-existence?

• Whither McCarthy’s advice taker?

• ..or Janet Kolodner’s house wife?

• …or even Dave’s HAL? • (with hopefully a less sinister voice) THIS TALK

HAAI in Planning

AI Planning: The Canonical View

4

A fully specifiedproblem--Initial state--Goals

(each non-negotiable)--Complete Action Model

The Plan

Human-in-the-Loop Planning

• In many scenarios, humans are part of the planning loop, because the planner:

• Needs to plan to avoid them• Human-Aware Planning

• Needs to provide decision support to humans• Because “planning” in some scenarios is too important to be

left to automated planners

• “Mixed-initiative Planning”; “Human-Centered Planning”; “Crowd-Sourced Planning”

• Needs help from humans• Mixed-initiative planning; “Symbiotic autonomy”

• Needs to team with them• Human-robot teaming; Collaborative planning

5

A Brief History of HILP• Beginnings of significant interest in the 90’s

• Under the aegis of ARPA Planning Initiative (and NASA)

• Several of the critical challenges were recognized

• Trains project [Ferguson/Allen; Rochester]

• Overview of challenges [Burstein/McDermott + ARPI Cohort]

• MAPGEN work at NASA

• At least some of the interest in HILP then was motivated by the need to use humans as a “crutch” to help the planner

• Planners were very inefficient back then; and humans had to “enter the land of planners” and help their search..

• In the last ~15 years, much of the mainstream planning research has been geared towards improving the speed of of plan generation

• Mostly using reachability and other heuristics

• Renaissance of interest in HILP thanks to the realization that HILP is critical in many domains even with “fast” planners

6

Dimensions of HIL Planning

7

Cooperation

Modality

Communication

Modality

What is

CommunicatedKnowledge Level

CrowdsourcingInteraction

(Advice from

planner to humans)

Custom Interface Critiques, subgoalsIncomplete Preferences

Incomplete Dynamics

Human-Robot

TeamingTeaming/Collaborat

ion

Natural Language

Speech

Goals, Tasks, Model

information

Incomplete Preferences

Incomplete Dynamics

(Open World)

“Grandpa Hates

Robots”

Awareness (pre-

specified

constraints)

Prespecified

(Safety / Interaction

Constraints)

No explicit

communication

Incomplete Preferences

Complete Dynamics

MAPGEN

Interaction

(Planner takes

binding advice from

human)

Direct Modification of

Plans

Direct modifications,

decision alternatives

Incomplete Preferences

Complete Dynamics

How do we adapt/adopt

modern planning technology for HILP?

8

PlanningThe Canonical View

9

Plan (Handed off

for Execution)

Full

Problem

Specification

PLANNER

Fully Specified

Action Model

Fully Specified

Goals

Completely Known

(Initial) World StateAssumption: Complete Action DescriptionsFully Specified PreferencesAll objects in the world known up frontOne-shot planning

Allows planning to be a pure inference problem

But humans in the loop can ruin a really a perfect day

Violated Assumptions:Complete Action Descriptions (Split knowledge)Fully Specified Preferences (uncertain users)Packaged planning problem (Plan Recognition)One-shot planning (continual revision)

Planning is no longer a pure inference problem

0000000000000000

000

0

0000000000000000

000

0

0000000000000000

000

0

0000000000000

Human-in-the-Loop Planning

10

Coordinate with Humans[IROS14, PlanRob15]

Replan for the Robot

[AAAI10, DMAP13]

Communicate with

Human(s) in the Loop

Open World Goals

[IROS09, AAAI10, TIST10]

Action Model Information

[HRI12]

Handle Human Instructions

[ACS13, IROS14]

Assimilate Sensor

Information

Full

Problem

Specification

PLANNER

Fully Specified

Action Model

Fully Specified

Goals

Completely Known

(Initial) World State

Sapa Replan

Problem Updates

[TIST10]

Human-in-the-Loop

Planning and

Decision Support

Goal

Manager

Challenges for the Planner1. Interpret what humans are doing

• Plan/goal/intent recognition

2. Decision Support• Continual planning/Replanning

• Commitment sensitive to ensure coherent interaction• Handle constraints on plan

• Plan with incompleteness• Incomplete Preferences• Incomplete domain models

• Robust planning with “lite” models• (Learn to improve domain models)

3. Communication• Explanations/Excuses

• Excuse generation can be modeled as the (conjugate of) planning problem

• Asking for help/elaboration• Reason about the information value

Challenge: Interpretation

•Recognize and interpret what the human needs

(goal, intent) and is currently doing (plan)

•Structure of “plans”:

• Assume Structure: Plan/Goal Recognition• Exploit/assume structured representation (plan)

• Easier to match planner’s expectation of structured input

• Restricts flexibility of humans; less knowledge specified

• Extract/Infer Structure: Infer Human Plans• Allow humans to use natural language

• Semi-structured and unstructured text

• Extract information from human-generated input

• Validate against (partial) model

• Iteratively refine recognized goals and plan

12

Plan/Goal Recognition for

Human-Robot Teaming

Talamadupula et al. – Arizona State University & Tufts University

Coordination in Human-Robot Teams Using Mental Modeling & Plan Recognition

BELIEF IN GOAL

Planning Conversation Robot Task Plans

Go here

Inferring Human (Team) Task Plans

• Integrate robots seamlessly in time-critical domains

• Lessen burden of programming and deploying robots

• Leverage the use of web-based planning tool (NICS)

14

Inferring Robot Task Plans from Human Team Meetings

Been Kim, Caleb Chacha and Julie Shah – MIT

Challenge: Decision Support

• Need to generate (and maintain) plans in the presence of human quirks and real-world considerations

– Partially specified preferences• Diverse plans

– Fully specified constraints / preferences• “Grandpa hates robots”

– Partially specified model of the world• Robust plans

– Changing world state and goals• Replanning

15

Problem Statement:

Given

the objectives Oi,

the vector w for convex combination of Oi

the distribution h(w) of w,

Return a set of k plans with the minimum ICP value.

Solution Approaches:

Sampling: Sample kvalues of w, and approximate the optimal plan for each value.

ICP-Sequential: Drive the search to find plans that will improve ICP

Hybrid: Start with Sampling, and then improve the seed set with ICP-Sequential

[Baseline]: Find kdiverse plans using the distance measures from [IJCAI 2007] paper; LPG-Speed.

1616

Generating diverse plans to handle unknown and

partially known user preferencesNguyen, Do, Gerevini, Serina, Srivastava & Kambhampati, 2012

Handling Partially Specified Preferences

17

Grandpa Hates RobotsKöckemann, Karlsson & Pecora

Fully Specified Constraints

• Compilation approach: Compile into a (Probabilistic) Conformant Planningproblem

• One “unobservable” variable per each possible effect/precondition

• Significant initial state uncertainty

• Can adapt a probabilistic conformant planner such as POND [JAIR, 2006; AIJ 2008]

• Direct approach: Bias a planner’s search towards more robust plans

• Heuristically assess the robustness of partial plans

• Need to use the (approximate) robustness assessment procedures

[Bryce et. Al., ICAPS 2012; Nguyen et al; NIPS 2013; Nguyen & Kambhampati, ICAPS 2014

Handling Partial Models of the World

19

Replanning for Changes in the World

A Theory of Intra-Agent ReplanningTalamadupula, Smith, Cushing & Kambhampati

20

Challenge: Communication

• Planner has to communicate with humans to:• Provide explanations

• Ask for help

• Providing Explanations• Excuses for failures

• Explanations for state facts

• Asking for Help• Symbiotic autonomy – humans doing what robots

cannot

• Generating detailed utterances to elicit effective help

21

Coming up With Good ExcusesGöbelbecker, Keller, Eyerich, Brenner & Nebel (2010)

Generating Excuses

22

Preferred ExplanationsSohrabi, Baier & McIlraith (2011)

Carnegie Mellon University Stephanie Rosenthal and Manuela Veloso 23

[Rosenthal, Veloso ,Dey: Ro-Man 2009, IUI 2010, JSORO 2012]

People are great sources

of information for robots‘Humans as Sensors and Effectors’

Symbiotic AutonomyRosenthal, Veloso et al.

Asking for Help – Symbiotic Autonomy

“Can you hold the elevator door open for me?”

“Can you point to where on this map?”

“Can you tell me when we reach the 4th floor?”

“Can you transfer this item from my basket?”

24

Asking for Help Using Inverse SemanticsTellex, Knepper, Li, Rus & Roy

Generating Utterances to Elicit Help

Case StudyHuman-Robot Teaming + Crowdsourcing

PLANNING FOR

HUMAN-ROBOT TEAMING

PLANNING FOR

CROWDSOURCING

INTERPRETATION

Open World Goals

Continual (Re)Planning

Plan & Intent Recognition

Activity Suggestions

Activity Critiques

Ordering Constraints

DECISION SUPPORTContinual (Re)Planning

Plan & Intent Recognition

Sub-goal Generation

Constraint Violations

Continual Improvement

COMMUNICATION Model Updates Sub-goal Generation

25

Dimensions of HIL Planning

26

Cooperation

Modality

Communication

Modality

What is

CommunicatedKnowledge Level

CrowdsourcingInteraction

(Advice from

planner to humans)

Custom Interface Critiques, subgoalsIncomplete Preferences

Incomplete Dynamics

Human-Robot

TeamingTeaming/Collaborat

ion

Natural Language

Speech

Goals, Tasks, Model

information

Incomplete Preferences

Incomplete Dynamics

(Open World)

“Grandpa

Hates Robots”

Awareness (pre-

specified

constraints)

Prespecified

(Safety / Interaction

Constraints)

No explicit

communication

Incomplete Preferences

Complete Dynamics

MAPGEN

Interaction

(Planner takes

binding advice from

human)

Direct Modification of

Plans

Direct modifications,

decision alternatives

Incomplete Preferences

Complete Dynamics

Recap

27

• HILP raises several open challenges for planning systems, depending on the modality of interaction between human and planner

• This talk:

– Discussed dimensions of variation of HILP tasks

– Identified the planning challenges posed by HILP scenarios

• Interpretation, Decision Support and Communication

– Outlined a few current approaches for addressing these challenges

rakaposhi.eas.asu.edu/hilp-tutorial