chapter 8. situated dialogue processing for human-robot interaction greert-jan m. kruijff, pierre...

Chapter 8. Situated Dialogue Processing for Human-Robot Interaction

Greert-Jan M. Kruijff, Pierre Lison, Trevor Benjamin, Henril Jacobsson, Hendrik Zender, Ivana Kruijff-Korbayova, and Nick Hawes

Kim, Jong In

Oct,2, 2015

College of

Interdisciplinary Program in Cognitive Science

Seoul National University

2

Contents8.4. Talking about What You Can See(Small Scale)

What you see and what you mean What you All know, and What you are Saying Using What you see to Rank Alternative Interpretations Referring to What you see

8.5. Talking about places You can Visit(Large Scale) Talking about places Representing Places to Talk about Referring to Elsewhere Determining the Appropriate contrast set Understanding References to Elsewhere

8.6 Talking about things you can do

8.7. Conclusions

3

Overview

4

Talking about What you Can see Talking about What You Can See

1) A small-scale space or closed context

2) Perception – Symbol

3) Grounding – determines the meaning of the linguistic symbol 4) Old information(common ground) vs New information

(Guide Learning)

“The red mug is Big”

© 2015, SNU CSE Biointelligence Lab., http://bi.snu.ac.kr 5

Guide Learning

6

Translation of Spatial expressions.

Talking about What you Can see


Ontology

Talking about What you Can see What you see and What you mean

1) Semantic – Ontologically Sorted

2) Delimitation , Quantification (Grounding)

3) Dealing with instances, update

4) Evaluation whether the new information is added

Motivation Working memory Binding Working memory

• Prompt to take action

• Intentional contents

• Operating modality

• Visuo-Spatial information

• Indexical content

Talking about What you Can see Using What you see to Rank Alternative Interpretations

<The discriminative model>:assign a score to each possible semantic interpretation of a given spoken input

1) Linguistic feature

-The acoustic feature (ASR score<auto speech recognition>)

-The syntactic Level (Parsing)

-The semantic Level (Logical form)1) Contextual Feature

-Situated context(the objects in the visual scene)

-Dialogue context(previously referred entities in the dialogue history)

2) Current data set -195 individual utterance -Check exact match, partial match, error match

-Depending on Grammar relaxation, Activated feature

Talking about What you Can see Using What you see to Rank Alternative Interpretations


1) Indexical Ambiguity

2) “Put the ball near the mug to the left of the box”

<Disambiguation effects – Visuo-Spatial Scene provides support>

Using What you see to figure out what is meant


1) Robot could refer to objects and the spatial rela-tions

2) Q1. If the number of Object is too much?

3) Problem) Inter-object Spatial Relation

4) Define the set of objects –function as landmark –

Referring to What you see(small scale)

Talking about Places you can visit Talking about Places(Large Scale)

1) How people tend to employ many different strategies to introduce new location to Robot?

(“Space which cannot be perceived at once”)

2) How human presents a familiar indoor environment to a robot(Guided tour scenario)

-Human guided the robot around and names and objects

Talking about Places you can visit Talking about Places(Large Scale)

Strategies to introduce new locations.1) Naming whole rooms (“this it the kitchen” –referring to the room itself)2) Naming specific locations (“this is the kitchen” –referring to cooking area)3) Naming specific locations by the objects (“this is the coffee machine”)

->Personalizing the representation of the environment that robotConstructs)

15

Multi-Layer Spatial Map <Metric Map>1)The first layer of SLAMSLAM: Simultaneous localization and mapping

<Topological map>1) Abstract form.2) Topological segmentation is repre-sented by the coloring of the nodes3) In order to determine category of an area, we take a majority vote approach of the classification

<Conceptual map>1) On the highest level of Abstraction2) Spatial unit(e.g. room) ->human concept(e.g. kitchen)

16

Overview

Conceptual Map is represented as a Description logic ontology

-conceptual taxonomy(hand-written commonsense ontololgy

representing various aspects of indoor envi-ronment)

-a storage of instances(T-Box and A-Box of a Description Logics rea-soning framework)


SLAM – Metric map

Videohttps://www.youtube.com/watch?v=mQQL8pmztb4

https://www.youtube.com/watch?v=mQQL8pmztb4

https://www.youtube.com/watch?v=mQQL8pmztb4

Talking about Places you can visit Referring to Elsewhere

1. “the location at position(X=35.56, Y=-3.92,

2. “the mug left of the plate right of the mug left of the plate”

3. “Peter’s office no.200 at the end of the corridor on the third floor of the Acme Corp. building 3 in the Acme Corp. Building complex, 47 Ever-green Terrace, Calisota, Planet Earth”

1. “the kitchen around the corner”2. “the red mug left of the china

plate”3. “Peter’s office”4. “the large hall on the first

floor”

Communicate goal issue.1) Robot are good at measuring exact distances, but humans are not2) Infinite recursion3) The robot might have a vast knowledge but have to separate uniquely the referent from all enti-ties.

Talking about Places you can visit

Issue1) Determining proper contrast Set

1) If contrast set is too much?

2) if contrast set is too little?

Issue2) Robot viewpoints?

1) Exact measure or Topological Ab-straction

-The context for a dialogue situated in large-scale space can be determined on the basis of a topological represen-tation like human

Determining the Appropriate Contrast set

8.6 Talking about things you can do

1) Action Planning: the goal of action planning is to choose actions and ordering rela-tions among these actions to achieve a suitable location and plans for executing the action

-Action planning and Dialogue processing should interact.

2) Event nucleus

-it models the action as an event with temporal and casual dimension

3) Indexicality, intentionality, and information structure

“First take the mug, and put it in near the plate”

Things you can do

Small Scale Guide Tour -Ambiguity, Semantic Grounding -Motivation working memory -Binding working memory

Large Scale Guide Tour -Multi-Layered Spatial mapping -topological map, conceptual map

Action -Action planning,

Summary

Debate issue

1. How to solve ambiguity problem in Situated Dia-logue

.

2. Drone and Robot cooperation


Q & A


Appendix 질문 대비

APPENDIX( 질문 대비 )


Ambiguity


Human Augmented Mapping

28

Ambiguity


Topological Partitioning

Solution> Topological Partitioning

-Large red stars indicate doorways and the different coloring of the nodes depicts the topological parti-tioning of the environment

30

Ambiguity

31

CCG grammar


Clarification


Event nucleus


Generation of Referring Expressions


Basic Incremental Algorithm for GRE


SLAM


Conceptual map


Multi-Layered Conceptual Spatial MAP


Ontolgoy


Semantic Ontology


Parsing


Simple Parse


Parsing


Parse


Referential Anchoring


Temporal-Causal Sequence


Temporal


Robotic Platform


Guide Learning


Guided Learning


Topology


Multi-Layered Conceptual Spatial Map


Perceptual Control System


Robotic Architecture


Spatial Interpretation


T-box


Delimitation


Human Augmented Mapping


Proxy- Reasoner


Wizard of Oz experiment, /nominal constructs


Indexicality


Intentionality

chapter 8. situated dialogue processing for human-robot interaction greert-jan m. kruijff, pierre...

Documents

see1 robot

humanrobot interactiongreert

contextthe objects

placesrepresenting places

set of objects function

spatial relations2 q1

new location

acoustic feature asr