Proceedings of SpLU-RoboNLP, pages 50–60Minneapolis, MN, June 6, 2019. c©2019 Association for Computational Linguistics

50

Semantic Spatial Representation: a unique representation of anenvironment based on an ontology for robotic applications

Guillaume SarthouLAAS-CNRS, Universite

de Toulouse, CNRS,Toulouse, France

gsarthou@laas.fr

Aurelie ClodicLAAS-CNRS, Universite

de Toulouse, CNRS,Toulouse, France

aclodic@laas.fr

Rachid AlamiLAAS-CNRS, Universite

de Toulouse, CNRS,Toulouse, Francealami@laas.fr

AbstractIt is important, for human-robot interaction, toendow the robot with the knowledge neces-sary to understand human needs and to be ableto respond to them. We present a formalizedand unified representation for indoor environ-ments using an ontology devised for a routedescription task in which a robot must provideexplanations to a person. We show that thisrepresentation can be used to choose a routeto explain to a human as well as to verbalizeit using a route perspective. Based on ontol-ogy, this representation has a strong possibil-ity of evolution to adapt to many other applica-tions. With it, we get the semantics of the envi-ronment elements while keeping a descriptionof the known connectivity of the environment.This representation and the illustration algo-rithms, to find and verbalize a route, have beentested in two environments of different scales.

1 Introduction

Asking one’s way, when one does not know ex-actly where one’s destination is, is something weall did. Just as we have all responded to such arequest from a lost person. This is the heart of theroad description task. This task, which seems sonatural to us in a human-human context, requiresin fact a set of knowledge (e.g. on the place, on thepossible points of reference in this particular envi-ronment) and ”good practices” (e.g. to give a patheasy to follow if possible) that need to be modeledif we want to implement it on a robot. This paperpresents a robotics application of our system butit would be possible to use it in other applicationssuch as virtual agent.

This route description task is an interesting ap-plication case through the variety of the needed in-formation (e.g. type of elements, place topology,names of the elements in natural language). It hasbeen well studied in the field of human-robot inter-action. Robot guides have already been deployed

in shopping centers (Okuno et al., 2009), museums(Burgard et al., 1998; Clodic et al., 2006; Sieg-wart et al., 2003) or airports (Triebel et al., 2016).However, using metrical (Thrun, 2008), topolog-ical (Morales Saiki et al., 2011), semantic repre-sentations, or trying to mix them together (Satakeet al., 2015b) (Chrastil and Warren, 2014) (Zen-der et al., 2008), it is difficult to have a uniformway to represent the environment. In addition, itis difficult to have a representation which allows tocalculate a route and at the same time to express itto the human with whom the robot interacts, be-cause this requires data of different types. Ouraim is to propose a single and standardized rep-resentation of an environment which can be usedto choose the appropriate route to be explained tothe human and at the same time to verbalize it us-ing a route perspective. The purpose of this pa-per is not to be applied to a guiding task in whicha mobile robot accompanies the human to his fi-nal destination but to explain to a human how toreach it. Consequently, we will not talk here abouta metrical representation like the one that can beused to navigate in the environment (Thrun, 2008)or to negotiate it use in (Skarzynski et al., 2017).

Route perspective means essentially to navigatementally in order to verbalize the path to followbut also to facilitate understanding and memoriz-ing instructions. The route perspective opposesthe survey perspective which is a top view withlandmarks and paths printed on a map. Moraleset al. (Morales et al., 2015) indicate that nam-ing parts of a geometric map does not leave theopportunity to compute such perspective. As in(Satake et al., 2015a), we have chosen to developour representation with an ontology as it allows toreason about the meaning of words and thus im-prove the understanding of human demands. Inaddition, we propose a way to merge the topologi-cal representation into the semantic representation

51

(the ontology) to get the meaning of the environ-ment elements while keeping a description of theconnectivity of the elements of the environment.We propose to name it semantic spatial represen-tation (SSR). With this, we are able to developthe two features presented in (Mallot and Basten,2009) for the route description task, which consistof selecting a sequence of places leading to the ob-jective and managing the declarative knowledge tochoose the right action to explain at each point ofthe sequence. Based on the principles of topologi-cal description, although represented semantically,we are able to compute multiple routes and newdetours for the same objective in contrast with aroute knowledge, which maps a predefined routeto a given request. Thanks to this capacity and tothe semantic knowledge of the environments avail-able in the representation, it is also possible to pro-vide the most relevant route to a user according tohis preferences and capabilities. A basic exam-ple would be that we will never recommend a pathwith stairs to a mobility impaired person. Morethan the extension of the spatial semantic hierar-chy (SSH) (Kuipers, 2000) allowing the represen-tation of the environment, we present here an al-gorithm to choose the routes and another one togenerate an explanation sentence. Both algorithmsare based solely on the knowledge provided by theSSR.

Regarding the representation of the environ-ment generally used in order to find an itinerary,we have first to analyse GNSS road navigationsystems. In (Liu, 1997) or (Cao and Krumm,2009), we find the same principle of a topologi-cal network representing the roads with semanticinformation attached to each of them. This typeof representation seems logical regarding the per-formance required for such systems operating invery large areas. However, GNSS road navigationsystems must respond only to this unique task offinding a path when a robot is expected to be ableto answer to various tasks. This is why we havedeveloped and implemented a representation thatcan be used more widely while still allowing thesearch for routes.

This paper focuses on the presentation of theSSR and on its usability for the route descriptiontask. For now, all the ontologies used to test theSSR have been made by hand. However, manyrecent research work leads to automatically gener-ate a topological representation of an environment

from geometric measurements (e.g. Region Adja-cency Graphs (Kuipers et al., 2004), Cell and Por-tal Graphs (Lefebvre and Hornus, 2003) or hierar-chical models (Lorenz et al., 2006), or from natu-ral language (Hemachandra et al., 2014)). We havenot done it yet, but our system could benefit fromthis work to generate a representation of an envi-ronment using SSR, which would solve the com-plexity of creating such a representation by hand.

In order to present our work, we will followthe three cognitive operations needed to gener-ate a spatial discourse (Denis, 1997): (section 2)an activation of an internal representation of theenvironment; (sections 3 and 4) the planning ofa route in the mental representation made previ-ously; (section 5) the formulation of the procedurethat the user must perform to achieve the objec-tive. The SSR and the algorithms demonstratingits usability have been tested in two environmentsof different scales: an emulated mall in our lab-oratory and a real mall. Results are presented insection 6 for the two environments.

2 Environment representation: SSR

In cognitive psychology, Semantic memory refersto the encyclopedic knowledge of words associ-ated to their meanings. Some authors have pro-posed a model of this semantic memory as be-ing a semantic network in which each concept islinked to others by properties and have designed acomputer-implemented model (Collins and Quil-lian, 1969). This initial model has since been for-malized as an ontology (Berners-Lee et al., 2001)and is already widely used in the semantic web.

This model is already used in robotics to ob-tain a detailed representation of the environmentin which robots operate. For example, (Satakeet al., 2015a) and (Beetz et al., 2018) use an on-tology to represent knowledge about the types ofitems such as the types of shops (restaurant, fash-ion store, for example) or the properties of itemssuch as the stores where they are sold.

(Kuipers, 2000) introduced the ’topologicallevel’ with SSH (spatial semantic hierarchy)which defines a place, a path and a re-gion and defined several relationships betweenthem. Ontologies are constructed using tripletswhere two concepts are linked by a property(e.g property(concept1, concept2)), however theKuipers SSH does not allow such representa-tion due to the use of some quadruplets (e.g

52

along(view, path, dir)) in addition to triplets. Toovercome this limitation, we propose a formalisa-tion, that we call Semantic Spatial Representation(SSR) to represent an environment with ontologies(i.e. using triplets).

In this section we present the minimal ontologythat constitutes the SSR but it can be extended torepresent the knowledge of the types and the prop-erties of the elements while preserving the first useof this model.

2.1 Classes

Figure 1: Classes for a representation of the topologyof an indoor environment in a semantic description.

Region: It represents a two-dimensional areathat is a subset of the overall environment. A de-scription of the environment must include at leastone region representing the entire environment.Regions are used to reduce the complexity of theroutes computation, so we recommend to use sev-eral region especially for large scale areas. A ba-sic use of the regions is for multi-storey buildingswhere each floor should be more naturally consid-ered as a region. Regions can be described as be-ing nested.Path: It is a one dimensional element along

which it is possible to move. A path must havea direction.

• Corridor: It represents a kind of path witha beginning and an end, for which beginningand end are distinct. The arbitrary directionchosen for a corridor defines the position ofits beginning and end. This defines also theright and left of the corridor.

• Openspace: It is a kind of path which doesnot have any begin or end. It can be viewedas a ”potato-shaped” describing the outline ofan open space. It materializes the possibil-ity of turning the gaze around the room andthe fact of not having to go through a definedpath to reach one of its points.

Place: It represents a point of zero dimensionthat can represent a physical or symbolic element.It can be extended to represent stores and land-marks in the example of a shopping center.

• Path intersection: It represents the con-nection between only two paths and thus awaypoint to go from one path to another. Inthe case of a crossing between three paths,three intersections would therefore be de-scribed.

• Interface: It represents the connection be-tween only two regions and thus a waypointto move from one region to another. It can bephysical, like a door or a staircase, or sym-bolic like a passage.

The distinction between paths and places is re-lated to the differences between the types of roomsmade by (Andresen et al., 2016) where some areused to circulate (corridors) while others have anexplicit use to the exclusion of traffic (place).

2.2 PropertiesProperties are used to express topological rela-tionships such as connections between paths andplaces or the order of places along a path. Allthe properties presented here can be extended withtheir inverse (e.g. isIn and hasIn) for a more ex-pressive model and thus easier handling.

Figure 2: Properties for a representation of the topol-ogy of an indoor environment in a semantic description.

isIn(path/place, region): path or place is inregion.isAlong(place, path): place is along path.

• isAlong(place, openspace): For openspaces, since there is no beginning or end,places are only defined as being along anopen space.

• isAlong(place, corridor): Forcorridors, the specific proper-ties isAtBeginEdgeOfPath,isAtEndEdgeOfPath,isAtLeftOfPath, isAtRightOfPathmust be used. The choice of these propertiesis made with the arbitrary direction definedby positioning itself at its beginning and bytraversing it towards its end.

53

isBeside(place1, place2): place1 is besideplace2. Specified properties isAtLeftOf andisAtRightOf must be used to express the orderof places. The choice of these properties is madeby positioning themselves at the place and facingthe path along the place.isInfrontOf(place1, place2): place1 is in

front of place2. This property does not need tobe applied to all the places described. The more itis used, the more the verbalization of the itinerarywill be easy. It is important to always define aplace in front of an intersection to be able to de-termine if the guided human will have to go leftor right in some cases. If there is no describedplace in front of an intersection, we can use aemptyP lace class that would inherit the placeclass.

The following axioms reduce the complexity ofthe description of the environment. The logicalrelations will therefore be solved by the ontologyreasoner.

• isAtLeftOf(place1, place2) ↔isAtRightOf(place2, place1)

• isInfrontOf(place1, place2) ↔isInfrontOf(place2, place1)

• isAlong(place, path) ∧isIn(path, region)→ isIn(place, region)

3 Computing routes

At this point, we have built an internal represen-tation of the environment using the Semantic Spa-tial Representation (SSR). We illustrate how thisrepresentation can be used to compute the pos-sible routes from one place to another. Even ifthe length of a route is taken into account in thechoice, the complexity of the description is an im-portant criterion (Morales et al., 2015). Whensomeone asks his way, the guide will not necessar-ily try to give him the shortest way. His main goalis to make sure the person reach her goal. In theexample of Figure 3, even if the red route is littlelonger than blue route, he will certainly propose itinstead. Every intersection or change of directionis a risk for the guided person to make a mistakeand thus get lost.

In this section, the goal is to provide multipleroutes so that we can allow to choose the best routebased on the person preferences. The possibility

Figure 3: Comparison of two routes in terms of com-plexity and length. The blue route (. . .) is the shortestbut is complex to explain while the red (- - -) is simpleralthough a bit longer.

of making this choice using the SSR will be pre-sented in section 4. In order to reduce the com-plexity of the search, especially for large scale en-vironments, we propose to work at two levels:

• First : Region level: considers only areas andpassages such as doors, stairs or elevators.

• Then : Place level: provides complete routesdescription including paths and intersectionswithin regions.

3.1 Region level

In large-scale environments such as multi-storeybuildings, routes computation can lead to combi-natorial explosion. Exploration at the region leveldecreases this effect by conducting a first high-level exploration. In Figure 4 we can see that theexploration of paths of regions 4 and 5 is uselessbecause these regions do not lead to the regioncontaining the goal. This exploration uses only theregions and interface elements described in sec-tion 2.

Figure 4: Representation of an environment at the re-gional level.

Each interface is connected to regions thanks tothe isIn property. With this property, a route find-ing algorithm, based on the breadth-first search,makes possible to find the shortest routes connect-ing two regions by using the semantic knowledge.

54

By including the knowledge base exploration di-rectly inside the search algorithms, it it not neces-sary to extract a topological graph with nodes andarcs. It is carried out within the search algorithmwithout preprocessing.

This algorithm applied to the example presentedin Figure 4 gives the tree of Figure 5. The finalroutes found by the algorithm are :

• region 1 − interface 1 − region 2 −interface 2− region 3

• region 1 − interface 1 − region 2 −interface 3− region 3

Region 5 has never been explored and region 4is not present in the final result. However, both so-lutions with interfaces 2 and 3 have been taken intoaccount. This type of results makes possible toquickly eliminate unnecessary solutions and thusreduces the complexity for a more detailed searchin a second time. This technique is similar to whatis done for GNSS road navigation systems wherethe main roads are studied upstream of secondaryroads with pyramidal (or hierarchical) route struc-ture (Bovy and Stern, 1990).

Figure 5: Exploration graph resulting from region-level search (sec.3.1) and the aggregation of start andend places (sec.3.2) .

3.2 Place level

Place-level search is based on the Region-levelsearch results with the aggregation of start andend places, so the format changes from region −place−region−...−region to a place−region−place− ...− place.

Place-level search works from one place to an-other through a single region. We have thereforedivided the previous solutions to meet this con-straint. This step aims to reduce complexity again.

Indeed, if several routes pass through the same re-gion with the same places of departure and arrival,the inner route can be calculated once and for all.In our example, the division gives five sub-routesinstead of six:

• start− region 1− interface 1

• interface 1− region 2− interface 2

• interface 2− region 3− end

• interface 1− region 2− interface 3

• interface 3− region 3− end

The place-level algorithm aims to replace eachsub-route region with a succession of paths andintersections. It works on the same principle asthe previous search algorithm using the isAlongproperty instead of the isIn property. To im-prove performance, we use moving forward forthe breadth-first search. It stops the explorationof the routes passing through a path already usedin previous route calculation steps. In addition, itprevents loops.

Figure 6: Representation of corridors and intersectionsin region 1 from the example 4

Taking the example of Figure 4 and focusingon region 1, we can solve the sub-route start −region 1 − interface 1. Region 1 is repre-sented with its corridors and intersections in Fig-ure 6. By applying the algorithm at the placelevel, we have the solution start − corridor 1 −intersection 1− corridor 5− interface 1. Bydoing the same for each sub-route, we can then re-compose the global routes and give the set detailedof routes from start to end.

4 Choosing the best route

Since the SRR is based on an ontology, we canhave the meaning of each element of the environ-ment and we can attach additional information to

55

them as features. Now that we have found sev-eral routes to the same goal, we want to select onebased on different criteria. This selection of routesis independent of the previous section and a vari-ety of cost functions can be implemented based onspecific application needs. In the following sub-section, we present an example of cost functionusing SSR and designed for robot guide to be de-ployed by the European project MuMMER (Fosteret al., 2016).

4.1 Example of cost functionAs mentioned in (Morales et al., 2015), the com-plexity of the route to explain to a human, which isthe number of steps in a route, is the most impor-tant criterion in choosing the route. A cost func-tion taking into account only the complexity of aroute R in the environmental context M would bef(R,M) with N being the number of steps of R.

f(R,M) = N (1)

However, to find a good itinerary, it is impor-tant to take into account the preferences and capa-bilities of the guided person. An easy example isthat we will never indicate a route with stairs toa person with reduced mobility. Using an ontol-ogy and therefore the possibility of describing theproperties of elements of the environment, we addthe property hasCostwhich associates an elementwith a criterion. Criteria rated σi are: saliency, ac-cessibility, comfort, security, ease of explanationand speed. Other criteria could easily be addedthrough the use of ontology according to the spe-cific needs of the environment. All these criteriaand their antonyms can be applied to each elementn. The preferences of a person P are costs relatedto the criterion σi noted ϕσi . This represents thesensitivity of P to the σi criterion. The cost func-tion becomes f(P,R,M) to take into account thepreference of person P .

f(P,R,M) = N ×N∏n=0

[∏i

(σin × ϕσi)] (2)

Because we focused only on the complexity ofthe route explanation and the characteristics of theelements of the environment, in the presented costfunction 2, the distances are not taken into ac-count. This information could be added by work-ing with a metric representation. Another possi-bility that can be explored is to add some of the

metric knowledge, such as the length of the cor-ridor, into the semantic representation of the en-vironment to preserve the working principle of aunique representation of the environment in thisroute description task.

5 Explanation generation

This section describes the third cognitive opera-tion of (Denis, 1997) to generate a spatial dis-course: the formulation of the procedure. As(Kopp et al., 2007), we define a route descriptionas a set of route segments, each connecting twoimportant points and explained in a chronologi-cal way. As (Tversky and Lee, 1999), we addto each route segment a triplet: orientation, ac-tion, and landmark to enable its explanation. Thedivision into segments corresponds to all paths,with their entry and exit points, provided by ourplanning part. However, the semantic representa-tion (SSR) used to plan the route is not directlyusable to generate the formulation of the proce-dure. With the current representation, the orien-tation and action are too complex to extract (giventhat they depend on the direction by which the per-son arrives). It is however possible to interpretthe semantic representation in relation to the esti-mated future position of the human. This interpre-tation is what we call the internal representation.This internal representation is composed of severalsub-representations each representing a path of theglobal environment. Each segment of the route isrepresented independently of the others. For openspace, we generate an ordered array of all loca-tions along it. For the corridors, we generate fourordered arrays to represent the left, the right,the beginedge and the endedge of the corridors.These information can be found in the ontologywith the properties isAlong, isAtLeftOfPath,and so on. To order the places in each array, we usethe properties isAtLeftOf and isAtRightOfalso present in the ontology. This internal repre-sentation can be displayed and gives Figure 7 forthe corridor 1 of region 1 from the example 4. TheisInfrontOf property is used to generate betterplacements.

Once we have an internal representation ofeach segment, we can determine the procedurethat the user must perform. (Kopp et al., 2007)mention that an action, a reorientation, a pro-gression or a positioning must be carried outat the end of each segment. The end of one

56

Figure 7: Internal representation of a the corridor 1of region 1 from the example 6, extracted from the se-mantic representation.

segment being the beginning of the next, wechoose to determine the actions at the beginningof each segment (which corresponds more to ourinternal representation). It allows to work onone path at a time. This rule is formalized as”choosing action Ai at place Pj will leadto place Pk” by (Mallot and Basten, 2009). Thisdetermination of actions can be made with our in-ternal representation, as shown in Figure 8 wherePj is the gray place and Pk can be one of theother. The red information at the top gives the”turn right” action with Pk being Place 9 orPlace 10, ”go in front of you” with Pk be-ing Place 3 and ”turn left” for the other places.The blue information on the sides gives the ori-entation of the sub-goal place Pk taking into ac-count the previous reorientation. With this ori-entation information we can give an explanationof the form ”take the corridor at your right”where the action is determined by the type ofPk and the side given by the orientation informa-tion. On the example of corridor 1, to go fromstart to intersection 1, the full sentence will beturn left then take the corridor at your right.Moreover, by taking into account the orientationof the guided person after an action we allow toprovide directions in the route perspective and sothe guided person to perform an imaginary tour ofthe environment (Kopp et al., 2007).

By working segment by segment in the ordergiven by the route search algorithm, we necessar-ily generate the explanations with a temporospa-tial ordering. This criterion is an important pointin Allen’s best practice in communicating routeknowledge (Allen, 2000).

The latest critical information presented by(Tversky and Lee, 1999) is landmark and wehave it in our representation. (Tversky and Lee,1998) noted that more than 90% of the guidingspots on maps and verbal directions contained ad-ditional information, which corresponds also to

Figure 8: Resolution of directions and directions withan entry in the hallway by the gray square ”start”.

the results of (Denis, 1997) and the Allen’s bestpractice (Allen, 2000). With our internal rep-resentation, we provide all the landmarks (cor-responding to places because being defined assuch) around which action must be taken andwe can therefore refer to it to help the guidedperson. On the previous example, the sentencemay be confusing because there are two cor-ridors on the left. We are able to refer toplace 8 which will be on the left or place 6which will be on the right by projecting the fu-ture position of the human at intersection 1.With this new information, the situation can bedisambiguated. The full sentence will becameturn left then take the corridor at your rightstraight after place 6.

The verbalization software based on the SSRand the principles described above was createdbased on a human-human study (Belhassein et al.,2017). Among the set of route description sen-tences, we have identified four types of explana-tory components: those corresponding to the be-ginning of route description, to the end of a route,of progress in the route and the particular case ofexplanations with one step. These four types areonly dependent of the position of the segment to beincluded in the global explanation procedure. Foreach type, we have identified various sub-types de-pending on the actions to be performed or the lo-cation of actions. For example, for the types ofend-of-procedure sentences, we distinguish thosewhere the end goal will be on the side, in front or atthe current future position. In total, we have iden-tified 15 sub-types. Each component of the ex-planation sentence has been classified into one ofthese sub-types. We want to be able to propose dif-ferent ways to express the same things so the sys-tem does not have only one way to express the verysame information. To represent similar sentencesand to be able to generate sentences with varia-tions, we have grouped sentences with close lex-ical structures. Each sentence is then representedwith its variations as follows: [”you will ”], [”see

57

”, ”find ”],[”it ”, ”/X ”], [”on ”], [”your ”, ”the ”],[”/D ”],[”side ”, ”when you walk ”, ””]. When us-ing a sentence, the variations are randomly chosenwith a uniform distribution. We can notice in theprevious example the use of variables such that Xwhich corresponds to the name of an element ofthe environment and D to a direction. We alsoused the Y variables for a reference points andDY for a reference point directions. If a sentencerequires a variable that we have not been able toextract from our internal representation, then an-other sentence with the same meaning or anothervariation of the sentence that does not require thevariable is chosen.

6 Applications

The SSR was first applied in an emulated mall todevelop the algorithms 1, but we also tested it ina real mall to study its applicability in a larger en-vironment. Table 1 indicates the number of ele-ments described in both environments. The num-ber of places does not correspond only to the sumof the shops, interfaces and intersections becausemuch more elements have been described, such asATMs, restrooms or carts location.

emulated realplace 83 249shop 19 135

interface 11 18path intersection 10 52

path 11 42region 5 4

Table 1: Number of elements described in the emu-lated and real environment.

Table 2 presents the CPU time needed for theroutes computation and cost function algorithmsfor several cases, applied to the real environmentrepresentation. Even though specialized algo-rithms that work with a specific representation ofthe environment may be faster than ours, we cansee here that they are acceptable in the contextof a human robot interaction and especially in aroute description task to both compute the pathand verbalize it. Indeed, by providing semantic,topological and spatial knowledge within a sin-gle representation it can be used by several algo-rithms usually requiring different representations.

1https://cloud.laas.fr/index.php/s/Mvfty2xN9qymR2T

We can also see the use of the regions to reduce thecomputation times with the two cases where threeroutes were found, one of the cases crossing oneregion and the other two.

Number Number Number Pathof of of finding

routes regions paths executionfound crossed used time (ms)

1 1 1 < 10

1 1 3 [20, 25]

3 1 9 [50, 55]

3 2 12 [30, 35]

16 2 75 [160, 170]

20 2 129 [180, 190]

Table 2: CPU time (min-max interval) time for com-puting routes in a big shopping mall description. Eachrow refers to a single request that can provide multipleroutes to the goal.

To show the usability of the internal representa-tion extracted from the SSR in the verbalization ofthe route, we have developed a software 2 that isable to verbalize the route found by our semanticplanner. In examples of the sentences synthesizedby this software (Table 3), we can see that for thesame goal, it is possible to use different points ofreference and to position them with respect to an-other element in the environment. All directionsshown in the A and B examples take into accountthe future position of the guided human and pro-vide indications from the perspective of the route.

Goal SentenceY You see there Y.Y It’s on the right side of Z.Y It’s on the left side of X.A Go through the door. Take the stairs at

your left and turn left. Go almost at thevery end of the corridor and, turn left atthe door. After that you will see A onyour right.

B Go straight down this aisle. Then, walkto the very end of the corridor andit’s on the left there.

Table 3: Sentences generated by a software using theinternal representation extracted from the SSR.

The applications presented previously have not2https://github.com/LAAS-HRI/route verbalization

58

only been tested as such, but have been integratedinto a global robotic architecture and deploy in amall center 3 as shown in figure 9. This integra-tion shows that the results obtained by algorithmsworking with a single semantic representation ofan environment are usable and are relevant in amore global task.

Figure 9: Robot describing a route to a human in amall using the SSR and the associated algorithms. Thesentence in green is the explanation of the route verbal-ized by the robot from the SSR representation: ”just godown the corridor and then go almost at the very end ofthis corridor and it’s on the left when you walk”.

7 Conclusions

We have proposed an environment model that suit-able to find routes, to choose one and ro be able toverbalize it using a single representation. The keycontribution of our work is the semantic spatialrepresentation (SSR), a formalization of how todescribe an environment such as a large and com-plex public space mall using an ontology. We havealso presented results about the use of our systemby a robot that provides route guiding to humansin a shopping mall.

To benefit from our system, it could be interest-ing to integrate this representation and the corre-sponding algorithms to a dialog system (Papaioan-nou et al., 2018) in order to exploit more deeplyits capacities. An interesting usage of this sys-tem already possible but not yet exploited becauseof the need of a dialog system, would be to usethe guided person previous knowledge to choosea route and/or to generate an explanation (”If youknow the place 2, from this one ...”). In the samevein, it would be possible to link it with a systemsuch as Shared Visual Perspective Planner (Wald-hart et al., 2018) to begin explaining the route froma visible point. This would reduce the length of the

3https://cloud.laas.fr/index.php/s/CJcPWmMU7TZGQJB

explanations and thus ensure a better understand-ing of the itinerary for the guided person. An-other improvement would be to use an ontologyto ground the interaction (Lemaignan et al., 2012)as part of the route description task.

At this stage, only the topological representa-tion has been integrated into the semantic repre-sentation. This is a good first step in working witha single representation that is easier to evolve andensure consistency of knowledge. Future workwould involve the integration of metric informa-tion, and thus geometric representation.

Acknowledgments

This work has been supported by the EuropeanUnions Horizon 2020 research and innovationprogramme under grant agreement No. 688147(MuMMER project).

ReferencesGary L. Allen. 2000. Principles and practices for com-

municating route knowledge. Applied cognitive psy-chology, 14(4):333359.

Erik Andresen, David Haensel, Mohcine Chraibi, andArmin Seyfried. 2016. Wayfinding and CognitiveMaps forPedestrian Models. In Traffic and Granu-lar Flow ’15, pages 249–256. Springer InternationalPublishing.

Michael Beetz, Daniel Beler, Andrei Haidu, Mihai Po-marlan, and Asil Kaan Bozcuog. 2018. KnowRob2.0 A 2nd Generation Knowledge ProcessingFramework for Cognition-enabled Robotic Agents.page 8.

Kathleen Belhassein, Aurelie Clodic, Helene Cochet,Marketta Niemela, Paivi Heikkila, Hanna Lammi,and Antti Tammela. 2017. Human-Human Guid-ance Study.

Tim Berners-Lee, James Hendler, and Ora Lassila.2001. The Semantic Web. Scientific American,284(5):34–43.

P. H. Bovy and E. Stern. 1990. Route Choice: Wayfind-ing in Transport Networks: Wayfinding in TransportNetworks. Studies in Operational Regional Science.Springer Netherlands.

Wolfram Burgard, Armin B. Cremers, Dieter Fox, DirkHahnel, Gerhard Lakemeyer, Dirk Schulz, WalterSteiner, and Sebastian Thrun. 1998. The MuseumTour-Guide Robot RHINO. In Autonome MobileSysteme 1998, 14. Fachgesprach, Karlsruhe, 30.November - 1. Dezember 1998, pages 245–254.

Lili Cao and John Krumm. 2009. From GPS Traces toa Routable Road Map. In Proceedings of the 17th

59

ACM SIGSPATIAL International Conference on Ad-vances in Geographic Information Systems, GIS ’09,pages 3–12, New York, NY, USA. ACM.

Elizabeth R. Chrastil and William H. Warren. 2014.From Cognitive Maps to Cognitive Graphs. PLOSONE, 9(11):e112544.

Aurelie Clodic, Sara Fleury, Rachid Alami, and al.2006. Rackham: An Interactive Robot-Guide. InThe 15th IEEE International Symposium on Robotand Human Interactive Communication, RO-MAN2006, Hatfield, Herthfordshire, UK, 6-8 September,2006, pages 502–509.

Allan M. Collins and M. Ross Quillian. 1969. Re-trieval time from semantic memory. Journal of Ver-bal Learning and Verbal Behavior, 8(2):240–247.

Michel Denis. 1997. The description of routes: Acognitive approach to the production of spatial dis-course. Cahiers de Psychologie Cognitive, 16:409–458.

Mary Ellen Foster, Rachid Alami, Olli Gestra-nius, Oliver Lemon, Marketta Niemela, Jean-MarcOdobez, and Amit Kumar Pandey. 2016. The MuM-MER Project: Engaging Human-Robot Interactionin Real-World Public Spaces. In Social Robotics.8th International Conference, ICSR 2016, KansasCity, MO, USA, November 1-3, 2016 Proceedings,Lecture Notes in Computer Science, pages 753–763.

Sachithra Hemachandra, Matthew R. Walter, StefanieTellex, and Seth Teller. 2014. Learning spatial-semantic representations from natural language de-scriptions and scene classifications. In 2014 IEEEInternational Conference on Robotics and Automa-tion (ICRA), pages 2623–2630, Hong Kong, China.IEEE.

Stefan Kopp, Paul A. Tepper, Kimberley Ferriman,Kristina Striegnitz, and Justine Cassell. 2007. Trad-ing Spaces: How Humans and Humanoids UseSpeech and Gesture to Give Directions. In ToyoakiNishida, editor, Wiley Series in Agent Technology,pages 133–160. John Wiley & Sons, Ltd, Chichester,UK.

B. Kuipers, J. Modayil, P. Beeson, M. MacMahon, andF. Savelli. 2004. Local metrical and global topolog-ical maps in the hybrid spatial semantic hierarchy.In IEEE ICRA, 2004. Proceedings. ICRA ’04. 2004,volume 5, pages 4845–4851 Vol.5.

Benjamin Kuipers. 2000. The Spatial Semantic Hier-archy. Artificial Intelligence, 119(1):191–233.

Sylvain Lefebvre and Samuel Hornus. 2003. Auto-matic Cell-and-portal Decomposition. report, IN-RIA.

Severin Lemaignan, Raquel Ros, Emrah Akin Sisbot,Rachid Alami, and Michael Beetz. 2012. Ground-ing the Interaction: Anchoring Situated Discourse inEveryday Human-Robot Interaction. International

Journal of Social Robotics, 4(2):pp.181–199. 20pages.

Bing Liu. 1997. Route finding by using knowledgeabout the road network. IEEE Transactions on Sys-tems, Man, and Cybernetics - Part A: Systems andHumans, 27(4):436–448.

Bernhard Lorenz, Hans Jrgen Ohlbach, and Edgar-Philipp Stoffel. 2006. A Hybrid Spatial Model forRepresenting Indoor Environments. In Web andWireless Geographical Information Systems, Lec-ture Notes in Computer Science, pages 102–112.Springer Berlin Heidelberg.

Hanspeter A. Mallot and Kai Basten. 2009. Embodiedspatial cognition: Biological and artificial systems.Image and Vision Computing, 27(11):1658–1670.

Yoichi Morales, Satoru Satake, Takayuki Kanda, andNorihiro Hagita. 2015. Building a Model of theEnvironment from a Route Perspective for Human-Robot Interaction. International Journal of SocialRobotics, 7(2):165–181.

Luis Yoichi Morales Saiki, Satoru Satake, TakayukiKanda, and Norihiro Hagita. 2011. Modeling Envi-ronments from a Route Perspective. In Proceedingsof the 6th International Conference on Human-robotInteraction, HRI ’11, pages 441–448, New York,NY, USA. ACM. Event-place: Lausanne, Switzer-land.

Yusuke Okuno, Takayuki Kanda, Michita Imai, Hi-roshi Ishiguro, and Norihiro Hagita. 2009. Provid-ing route directions: design of robot’s utterance, ges-ture, and timing. In HRI, 2009 4th ACM/IEEE Inter-national Conference on, pages 53–60. IEEE.

Ioannis Papaioannou, Christian Dondrup, and OliverLemon. 2018. Human-robot interaction requiresmore than slot filling - multi-threaded dialogue forcollaborative tasks and social conversation. InFAIM/ISCA Workshop on Artificial Intelligence forMultimodal Human Robot Interaction. ISCA.

Satoru Satake, Kotaro Hayashi, Keita Nakatani,and Takayuki Kanda. 2015a. Field trial of aninformation-providing robot in a shopping mall. In2015 IEEE/RSJ IROS, pages 1832–1839.

Satoru Satake, Keita Nakatani, Kotaro Hayashi,Takyuki Kanda, and Michita Imai. 2015b. Whatshould we know to develop an information robot?PeerJ Computer Science, 1:e8.

Roland Siegwart, Kai Oliver Arras, Samir Bouabdal-lah, Daniel Burnier, Gilles Froidevaux, Xavier Grep-pin, Bjorn Jensen, Antoine Lorotte, Laetitia Mayor,Mathieu Meisser, Roland Philippsen, Ralph Piguet,Guy Ramel, Gregoire Terrien, and Nicola Tomatis.2003. Robox at Expo.02: A large-scale installationof personal robots. Robotics and Autonomous Sys-tems, 42(3-4):203–222.

60

Kamil Skarzynski, Marcin Stepniak, Waldemar Bar-tyna, and Stanislaw Ambroszkiewicz. 2017. SO-MRS: a multi-robot system architecture based onthe SOA paradigm and ontology. arXiv:1709.03300[cs]. ArXiv: 1709.03300.

Sebastian Thrun. 2008. Simultaneous Localization andMapping. In Margaret E. Jefferies and Wai-KiangYeap, editors, Robotics and Cognitive Approachesto Spatial Mapping, Springer Tracts in AdvancedRobotics, pages 13–41. Springer Berlin Heidelberg,Berlin, Heidelberg.

Rudolph Triebel, Kai Arras, Rachid Alami, and al.2016. SPENCER: A Socially Aware Service Robotfor Passenger Guidance and Help in Busy Airports.In David S. Wettergreen and Timothy D. Barfoot,editors, Field and Service Robotics: Results of the10th International Conference, Springer Tracts inAdvanced Robotics, pages 607–622. Springer Inter-national Publishing, Cham.

Barbara Tversky and Paul U. Lee. 1998. How SpaceStructures Language. In Christian Freksa, Christo-pher Habel, and Karl F. Wender, editors, SpatialCognition: An Interdisciplinary Approach to Rep-resenting and Processing Spatial Knowledge, Lec-ture Notes in Computer Science, pages 157–175.Springer Berlin Heidelberg, Berlin, Heidelberg.

Barbara Tversky and Paul U. Lee. 1999. Pictorial andVerbal Tools for Conveying Routes. In Spatial Infor-mation Theory. Cognitive and Computational Foun-dations of Geographic Information Science, LectureNotes in Computer Science, pages 51–64. SpringerBerlin Heidelberg.

Jules Waldhart, Aurlie Clodic, and Rachid Alami.2018. Planning Human and Robot Placements forShared Visual Perspective. page 10.

H. Zender, O. Martnez Mozos, P. Jensfelt, G. J. M.Kruijff, and W. Burgard. 2008. Conceptual spatialrepresentations for indoor mobile robots. Roboticsand Autonomous Systems, 56(6):493–502.