chapter 8. situated dialogue processing for human-robot interaction greert-jan m. kruijff, pierre...

Chapter 8. Situated Dialogue Processing for Human-Robot Interaction

Greert-Jan M. Kruijff, Pierre Lison, Trevor Benjamin, Henril Jacobsson, Hendrik Zender, Ivana Kruijff-Korbayova, and Nick Hawes

Kim, Jong In

Oct,2, 2015

College of

Interdisciplinary Program in Cognitive Science

Seoul National University

Contents8.4. Talking about What You Can See(Small Scale)

What you see and what you mean What you All know, and What you are Saying Using What you see to Rank Alternative Interpretations Referring to What you see

8.5. Talking about places You can Visit(Large Scale) Talking about places Representing Places to Talk about Referring to Elsewhere Determining the Appropriate contrast set Understanding References to Elsewhere

8.6 Talking about things you can do

8.7. Conclusions

Overview

Talking about What you Can see Talking about What You Can See

1) A small-scale space or closed context

2) Perception – Symbol

3) Grounding – determines the meaning of the linguistic symbol 4) Old information(common ground) vs New information

(Guide Learning)

“The red mug is Big”

Guide Learning

Translation of Spatial expressions.

Talking about What you Can see

Ontology

Talking about What you Can see What you see and What you mean

1) Semantic – Ontologically Sorted

2) Delimitation , Quantification (Grounding)

3) Dealing with instances, update

4) Evaluation whether the new information is added

Motivation Working memory Binding Working memory

• Prompt to take action

• Intentional contents

• Operating modality

• Visuo-Spatial information

• Indexical content

Talking about What you Can see Using What you see to Rank Alternative Interpretations

<The discriminative model>:assign a score to each possible semantic interpretation of a given spoken input

1) Linguistic feature

-The acoustic feature (ASR score<auto speech recognition>)

-The syntactic Level (Parsing)

-The semantic Level (Logical form)1) Contextual Feature

-Situated context(the objects in the visual scene)

-Dialogue context(previously referred entities in the dialogue history)

2) Current data set -195 individual utterance -Check exact match, partial match, error match

-Depending on Grammar relaxation, Activated feature

Talking about What you Can see Using What you see to Rank Alternative Interpretations

1) Indexical Ambiguity

2) “Put the ball near the mug to the left of the box”

Using What you see to figure out what is meant

1) Robot could refer to objects and the spatial rela-tions

2) Q1. If the number of Object is too much?

3) Problem) Inter-object Spatial Relation

4) Define the set of objects –function as landmark –

Referring to What you see(small scale)

Talking about Places you can visit Talking about Places(Large Scale)

1) How people tend to employ many different strategies to introduce new location to Robot?

(“Space which cannot be perceived at once”)

2) How human presents a familiar indoor environment to a robot(Guided tour scenario)

-Human guided the robot around and names and objects

Talking about Places you can visit Talking about Places(Large Scale)

Strategies to introduce new locations.1) Naming whole rooms (“this it the kitchen” –referring to the room itself)2) Naming specific locations (“this is the kitchen” –referring to cooking area)3) Naming specific locations by the objects (“this is the coffee machine”)

->Personalizing the representation of the environment that robotConstructs)

Multi-Layer Spatial Map <Metric Map>1)The first layer of SLAMSLAM: Simultaneous localization and mapping

<Topological map>1) Abstract form.2) Topological segmentation is repre-sented by the coloring of the nodes3) In order to determine category of an area, we take a majority vote approach of the classification

<Conceptual map>1) On the highest level of Abstraction2) Spatial unit(e.g. room) ->human concept(e.g. kitchen)

Overview

Conceptual Map is represented as a Description logic ontology

-conceptual taxonomy(hand-written commonsense ontololgy

representing various aspects of indoor envi-ronment)

-a storage of instances(T-Box and A-Box of a Description Logics rea-soning framework)

SLAM – Metric map

Videohttps://www.youtube.com/watch?v=mQQL8pmztb4

Talking about Places you can visit Referring to Elsewhere

1. “the location at position(X=35.56, Y=-3.92,

2. “the mug left of the plate right of the mug left of the plate”

3. “Peter’s office no.200 at the end of the corridor on the third floor of the Acme Corp. building 3 in the Acme Corp. Building complex, 47 Ever-green Terrace, Calisota, Planet Earth”

1. “the kitchen around the corner”2. “the red mug left of the china

plate”3. “Peter’s office”4. “the large hall on the first

floor”

Communicate goal issue.1) Robot are good at measuring exact distances, but humans are not2) Infinite recursion3) The robot might have a vast knowledge but have to separate uniquely the referent from all enti-ties.

Talking about Places you can visit

Issue1) Determining proper contrast Set

1) If contrast set is too much?

2) if contrast set is too little?

Issue2) Robot viewpoints?

1) Exact measure or Topological Ab-straction

-The context for a dialogue situated in large-scale space can be determined on the basis of a topological represen-tation like human

Determining the Appropriate Contrast set

8.6 Talking about things you can do

1) Action Planning: the goal of action planning is to choose actions and ordering rela-tions among these actions to achieve a suitable location and plans for executing the action

-Action planning and Dialogue processing should interact.

2) Event nucleus

-it models the action as an event with temporal and casual dimension

3) Indexicality, intentionality, and information structure

“First take the mug, and put it in near the plate”

Things you can do

Small Scale Guide Tour -Ambiguity, Semantic Grounding -Motivation working memory -Binding working memory

Large Scale Guide Tour -Multi-Layered Spatial mapping -topological map, conceptual map

Action -Action planning,

Summary

Debate issue

1. How to solve ambiguity problem in Situated Dia-logue

2. Drone and Robot cooperation

Appendix 질문 대비

APPENDIX( 질문 대비 )

Ambiguity

Human Augmented Mapping

Ambiguity

Topological Partitioning

Solution> Topological Partitioning

-Large red stars indicate doorways and the different coloring of the nodes depicts the topological parti-tioning of the environment

Ambiguity

CCG grammar

Clarification

Event nucleus

Generation of Referring Expressions

Basic Incremental Algorithm for GRE

Conceptual map

Multi-Layered Conceptual Spatial MAP

Ontolgoy

Semantic Ontology

Parsing

Simple Parse

Parsing

Referential Anchoring

Temporal-Causal Sequence

Temporal

Robotic Platform

Guide Learning

Guided Learning

Topology

Multi-Layered Conceptual Spatial Map

Perceptual Control System

Robotic Architecture

Spatial Interpretation

Delimitation

Human Augmented Mapping

Proxy- Reasoner

Wizard of Oz experiment, /nominal constructs

Indexicality

Intentionality

chapter 8. situated dialogue processing for human-robot interaction greert-jan m. kruijff, pierre...

see1 robot

humanrobot interactiongreert

contextthe objects

placesrepresenting places

set of objects function

spatial relations2 q1

new location

acoustic feature asr

Documents

agile - open...

www.stadtbibliothek-bremen.de barbara lison 1 bremen public...

the development of victim- offender mediation in sweden...

asr, nlu, and chatbots · asr, nlu, and chatbots pierre...

ministry of training, colleges and universities marie-lison...

cern r. jacobsson between lhc and the grid - aspects of...

synthesis and characterisation of zno …306210/...t. jesper...

welcome to the master programme in economics!. 05/10/2015/...

720th railway operating battalion lison train station

tutorial dialog m od esfturing/typ chowdhury … ·...

osgi remote services - alexander broekhuis, bram de kruijff

o verview of seta r esources a lison s umpter & s ean e...

prof. d. lison

introduction to statistical machine...

jobm sfqpsu - european commission · funding scheme...

incremental processing of spoken dialogue for...

the interrelationship between the maritime liability...

the whole fromage by kathe lison - excerpt

dynamic hybrid cloud applications - bram de kruijff,...

volkmar lauber staffan jacobsson