Learning Spatial-Semantic Representations fromNatural Language Descriptions and Scene Classifications

Sachithra Hemachandra, Matthew R. Walter, Stefanie Tellex, and Seth Teller

Abstract— We describe a semantic mapping algorithm thatlearns human-centric environment models by interpreting nat-ural language utterances. Underlying the approach is a coupledmetric, topological, and semantic representation of the environ-ment that enables the method to fuse information from naturallanguage descriptions with low-level metric and appearancedata. We extend earlier work with a novel formulation thatincorporates spatial layout into a topological representation ofthe environment. We also describe a factor graph formulationof the semantic properties that encodes human-centric conceptssuch as type and colloquial name for each mapped region.The algorithm infers these properties by combining the user’snatural language descriptions with image- and laser-basedscene classification. We also propose a mechanism to moreeffectively ground natural language descriptions of distantregions using semantic cues from other modalities. We describehow the algorithm employs this learned semantic information topropose valid topological hypotheses, leading to more accuratetopological and metric maps. We demonstrate that integratinglanguage with other sensor data increases the accuracy of theachieved spatial-semantic representation of the environment.

I. INTRODUCTION

A challenge to realizing robots that work productivelyalongside human partners is the development of efficientcommand and control mechanisms. Researchers have re-cently sought to endow robots with the ability to inter-act more effectively with people through natural languagespeech [1, 2, 3, 4, 5] and gesture understanding [6]. Efficientinteraction is facilitated when robots reason over models thatencode high-level semantic properties of the environment.For example, such models could help a micro-aerial vehicleinterpret a first responder’s command to “fly up the stairwayon the right, go down the hall, and observe the kitchen.”

Semantic mapping algorithms [7, 8, 9, 10] extend the met-ric environment models traditionally employed in robotics toinclude higher-level concepts, including types and colloquialnames for regions, and the presence and use of objects inthe environment. These methods typically operate by aug-menting a standard SLAM metric map with a representationof the environment’s topology, and a distinct representationof its semantic properties, the latter of which is populatedby interpreting the robot’s sensor stream, typically throughscene classification. In this layered approach, the underlyingmetric map induces and embeds the topological and semantic

S. Hemachandra, M.R. Walter, and S. Teller are with the ComputerScience and Artificial Intelligence Laboratory at the Massachusetts Instituteof Technology, Cambridge, MA 02139 USA {sachih, mwalter,teller}@csail.mit.edu

S. Tellex is with the Computer Science Department at Brown University,Providence, RI 02912 USA stefie10@cs.brown.edu

Fig. 1. Maximum likelihood semantic map of the 6th floor of Stata building(pie charts denote the likelihood of different region categories).

attributes. However, while refinements to the metric mapimprove the topological and semantic maps, most techniquesdo not allow knowledge inferred at the semantic and topolog-ical levels to influence the other layers. Typical approachesinfer semantic information from LIDAR and camera datausing pre-trained scene appearance models. Some effortsare capable of integrating egocentric descriptions about therobot’s current location (e.g., “we are in the kitchen”), butcannot handle allocentric descriptions that involve spatialrelations between and labels for potentially distant regionsin the environment (e.g., “the exit is next to the cafeteria”).

We addressed some of these limitations in previouswork [11] with an algorithm that maintains a joint distribu-tion over a Semantic Graph, a coupled metric, topological,and semantic environment representation learned from userutterances and the robot’s low-level sensor data, during aguided tour. Our framework was able to to learn propertiesof the environment that could not be perceived with typicalsensors (e.g. colloquial names for regions, properties of areasoutside the robot’s sensor range) and use semantic knowledgeto influence the rest of the semantic graph, allowing robotsto efficiently learn environment models from users.

Our previous algorithm decomposed the environment intoa collection of fixed, uniformly-sized regions. This has thepotential to result in a topology that is inconsistent withhuman concepts of space. Consequently, the representationmay not model the spatial extent to which the user’s de-scriptions refer, resulting in incorrect language groundings.The semantic information in the framework was limitedto user-provided colloquial names and did not provide ameans to reason over properties such as region type thatcan be inferred from LIDARs, cameras, or other onboardsensors. Additionally, due to the absence of other semanticinformation, the framework required that a landmark location

be labeled by the user before the utterance can be grounded(e.g., processing the phrase “the kitchen is down the hallway”requires the “hallway” to have already been labeled).

This paper describes an extension of our earlier approachto learn richer and more meaningful semantic models of theenvironment. Whereas our earlier framework reasoned onlyabout the connectivity of deterministically created regions(at fixed intervals), the current approach reasons over theenvironment’s region segmentation as well as its inter-regionconnectivity. Additionally, we propose a factor graph repre-sentation for the semantic model that reasons not only overeach region’s labels, but also its canonical type. As before,we infer region labels from user-provided descriptions, butwe also incorporate scene classification using the robot’sonboard sensors, notably camera and laser range-finders, toestimate region types. By modeling the relation between anarea’s type and its colloquial name, the algorithm can reasonover both region type and region label, even in the absence ofspeech. This enables the method to more effectively groundallocentric user utterances (e.g., when grounding the phrase“the kitchen is down the hallway”, we no longer requirethe user to explicitly label the “hallway” beforehand). Wealso describe a mechanism by which the algorithm derives asemantically meaningful topology of the environment basedupon a factor graph model of the distribution, where edgesare proposed using a spatial-semantic prior distribution. Weshow that the improved topology model then allows themethod to better handle ambiguities common in naturallanguage descriptions.

II. RELATED WORK

A number of researchers have focused on the problem ofconstructing semantic environment models [7, 10, 8, 9]. Mostapproaches augment lower-level metric maps with higher-level topological and/or semantic information. However,these typically follow a bottom-up approach in which higher-level concepts are constructed from lower-level information,without any information flow back down to lower-levelrepresentations. In Walter et al. [11], we addressed this byintroducing a framework that uses semantic information de-rived from natural language descriptions uttered by humansto improve the topological and metric representations. Ourproposed approach uses additional semantic cues to evaluatesemantic similarity of regions to update the topology.

Several existing approaches [8, 10] have incorporatedhigher-level semantic concepts such as room type and thepresence of objects with the use of appearance models.Pronobis and Jensfelt [10] describe a multi-modal proba-bilistic framework incorporating semantic information from awide variety of modalities including object detections, placeappearance, and human-provided information. However, theirapproach is limited to handling egocentric descriptions (e.g.,“we are in the living room”). Additionally, they infer topol-ogy based on door detections, a heuristic that works wellonly in certain kinds of environments; they do not maintaina distribution over likely topologies. In [11], we maintaineda hypothesis over the distribution of topologies, but the

Fig. 2. Example of a semantic graph: Two regions R1 and R2 andtheir constituent nodes ni’s; distributions over node poses xi; and thecorresponding factor graph.

topology was constructed from segments created at fixedspatial intervals, which can be inconsistent with humanconcepts. In the present work, we address this limitation byproposing a method that maintains multiple hypotheses aboutregion segmentations and about connections among regions.

The problem mapping natural language utterances to theircorresponding referents in a robot’s world model has beenstudied by several researchers [3, 4, 2, 12, 13, 14, 15, 16] inthe context of command and control. However, these effortshave not focused on constructing semantic representationsand, instead, typically assume them to be known a priori.In [11], we augmented our semantic graph with complexlanguage descriptions. However, this could be accomplishedonly when the user had explicitly labeled a location beforedescribing another in reference to it (e.g., “the gym is downthe hallway” requires prior labeling of the hallway). The re-sulting representation contained only labels obtained throughlanguage descriptions; it did not integrate semantic cuesfrom other sources (such as appearance models). The presentmethod improves upon this by integrating information frommultiple semantic sources and maintaining a distributionover a larger set of semantic properties, rendering it capableof grounding language even in the absence of pre-labeledlandmark locations.

III. SEMANTIC GRAPH REPRESENTATION

This section presents our approach to maintaining a distri-bution over semantic graphs, an environment representationthat consists of metric, topological, and semantic maps.

A. Semantic Graphs

We define the semantic graph as a tuple containing topo-logical, metric and semantic representations of the environ-ment. Figure 2 shows an example semantic graph for a trivialenvironment.

The topology Gt is composed of nodes ni that denote therobot’s trajectory through the environment (sampled at 1 mdistances), node connectivity, and node region assignments.We associate with each node a set of observations thatinclude laser scans zi, semantic appearance observations

ai based on laser li and camera ii models, and avail-able language observations λi. We assign nodes to regionsRα = {n1, .., nm} that represent spatially coherent areasin the environment compatible with human concepts (e.g.,rooms and hallways). Undirected edges exist between nodepairs in this graph, denoting traversability. Edges betweenregions are inferred based on the edges between nodes inthe graph. A region edge exists between two regions if atleast one graph edge connects a node from one region to anode in the other. The topological layer consists of the nodes,edges, and the region assignments for the nodes.

The pose xi of each node ni is represented in a globalreference frame. The metric layer is induces by the topology,where edges in the topology also include metric constraintsbetween the corresponding node poses. Metric constraintsare calculated by scan-matching the corresponding laserobservations of each region. A pose graph representationis employed to maintain the distribution over the pose ofeach node, conditioned on these constraints. Occupancymaps can be constructed based on the node poses and theircorresponding laser observations.

Semantic information is also conditioned on the topologyas shown in the Fig. 2. The semantic layer consists of afactor graph with variables that represent the type Cr andlabels Λr for each region, properties that can be observedat each node (in each region), and factors that denote thejoint likelihood of these variables (e.g., the likelihood ofobserving a label given a particular room type). Observationsof these region properties are made using laser- and image-based scene classifiers and by grounding human descriptionsof the environment.

Algorithm 1: Semantic Mapping Algorithm

Input: Pt−1 ={P

(i)t−1

}, and (ut, zt, at, λt), where

P(i)t−1 =

{G

(i)t−1, X

(i)t−1, S

(i)t−1, w

(i)t−1

}Output: Pt =

{P

(i)t

}for i = 1 to n do

1) Employ proposal distribution to propagate thegraph sample based on ut, λt and at.

a) Sample region allocationb) Sample region edgesc) Merge newly connected regions

2) Update the Gaussian distribution over the nodeposes X(i)

t conditioned on topology.3) Update the factor graph representing semantic

properties for the topology based on appearanceobservations (lt and it) and language λt.

4) Compute the new particle weight w(i)t based

upon the previous weight w(i)t−1 and the metric

data zt.end

Normalize weights and resample if needed.

B. Distribution Over Semantic Graphs

We maintain the joint distribution over the topology Gt,the vector of locations Xt, and the set of semantic propertiesSt. Formally, we maintain this distribution over semanticgraphs {Gt, Xt, St} at time t conditioned upon the historyof metric exteroceptive sensor data zt = {z1, z2, . . . , zt},odometry ut = {u1, u2, . . . , ut}, scene appearance obser-vations at = {a1, a2, . . . , at} (where in our implemen-tation at = {lt, it}), and natural language descriptionsλt = {λ1, λ2, . . . , λt},

p(Gt, Xt, St|zt, ut, at, λt). (1)

Each variable λi denotes a (possibly null) utterance, suchas “This is the kitchen,” or “The gym is down the hall.” Wefactor the joint posterior into a distribution over the graphsand a conditional distribution over the node poses and labels,

p(Gt, Xt, St|zt, at, ut, λt) = p(St|Xt, Gt, zt, at, ut, λt)

× p(Xt|Gt, zt, at, ut, λt)× p(Gt|zt, at, ut, λt) (2)

As in Walter et al. [11], we maintain this factored distribu-tion using a Rao-Blackwellized particle filter, mitigating thehyper-exponential hypothesis space of the topology [11].

We represent the joint distribution over the topology, nodelocations, and labels as a set of particles

Pt = {P (1)t , P

(2)t , . . . , P

(n)t }. (3)

Each particle P (i)t ∈ Pt consists of the set

P(i)t =

{G

(i)t , X

(i)t , S

(i)t , w

(i)t

}, (4)

where G(i)t denotes a sample from the space of graphs,

X(i)t is the analytic distribution over locations, S(i)

t is thedistribution over semantic properties, and w(i)

t is the weightof particle i.

IV. SEMANTIC MAPPING ALGORITHM

Algorithm 1 outlines the process by which the method re-cursively updates the distribution over semantic graphs (2) toreflect the latest robot motion, metric sensor data, laser- andimage-based scene classifications, and the natural languageutterances. The following sections explain each step in detail.

A. The Proposal Distribution

We compute the prior distribution over the semantic graphGt, given the posterior from the last time step Gt−1, bysampling from a proposal distribution. This proposal distri-bution is a predictive prior over the current graph given theprevious graph, sensor data (excluding the current time step),appearance data, odometry, and language,

p(Gt|Gt−1, zt−1, at, ut, λt). (5)

We augment the topology Gt−1 to reflect the robot’smotion by adding a node nt to to the topology and anedge to the previous node nt−1, resulting in an intermediategraph G−t . This represents the robot’s current pose and theconnectivity to its previous pose. This yields an updated

vector of poses X−t and semantic properties S−t . The newnode is assigned to the current region.

1) Creation of New Regions: We then probabilisticallybisect the current region Rc using the spectral clusteringmethod proposed by Blanco et al. [17]. We construct thesimilarity matrix using the laser point overlap between eachpair of nodes in the region. Equation 6 defines the likelihoodof bisecting the region, which is based on the normalizedcut value Nc of the graph involving the proposed segments.The likelihood of accepting a proposed segmentation risesas the Nc value decreases, i.e., as the separation of the twosegments improves (minimizing the inter-region similarity),

P (s/Ncut) =1

(1 + αN3c ). (6)

This results in more spatially distinct areas in the worldhaving a higher likelihood of being distinct regions, leadingto more particles modeling these areas as separate regions.If a particle segments the current region, a new region Ri iscreated that does not include the newly added node.

2) Edge Proposals: When a new region Ri is created,the algorithm proposes edges between this region, and otherregions in the topology, excluding the current region Rc.

Fig. 3. Example of region edges being proposed (black lines representsrejected edge proposals; the red line represents an accepted edge).

The algorithm samples inter-region edges from a spatial-semantic proposal distribution that incorporates the semanticsimilarity of regions, as well as the spatial distributions of itsconstituent nodes. This reflects the notion that regions thatare nearby and semantically similar are more likely to beconnected. We measure semantic similarity based upon thelabel distribution associated with each region. The resultinglikelihood has the form

pa(Gt|G−t , zt−1, ut, at, λt) =∏

j:eij /∈E−

p(Gijt |G−t ) (7a)

∝∏

j:eij /∈E−

px(Gijt |G−t )ps(Gijt |G−t ), (7b)

where we have omitted the history of language observationsλt, metric measurements zt−1, appearance measurementsat, and odometry ut for brevity. Equation 7a reflects theassumption that additional edges that express constraintsinvolving the current node eij /∈ E− are conditionallyindependent. While px(Gijt |G−t ) encodes the likelihood ofthe edge based on the spatial properties of the two re-gions, ps(G

ijt |G−t ) describes the edge likelihood based on

the regions’ semantic similarity. Equation 7b reflects the

assumed conditional independence between the spatial- andthe semantic-based edges.

For the spatial distribution prior, we consider the distancedij between the mean nodes of the two regions, where themean node is that with its pose closest to the region’s averagepose

px(Gijt |G−t ) =

∫X−

t

p(Gijt |X−t , G−t , ut)p(X−t |G−t ) (8a)

=

∫dij

p(Gijt |dij , G−t )p(dij |G−t ). (8b)

The conditional distribution p(Gijt |dij , Gt−1, zt−1, ut) ex-

presses the likelihood of adding an edge between regionsRt and Rj based upon the location of their mean nodes.We represent the distribution for a particular edge betweenregions Ri and Rj with distance dij = |X̄Ri − X̄Rj |2 as

p(Gijt |dij , G−t , zt−1, ut) ∝ 1

1 + γd2ij

, (9)

where γ specifies a distance bias. For the evaluations in thispaper, we use γ = 0.3. We approximate the distance priorp(dij |G−t , zt−1, ut) with a folded Gaussian distribution.

The semantic prior expresses the increased likelihood thatedges exist between regions with similar distributions overlabels Λ. The label distributions for the regions are modeledin the semantic layer,

ps(Gijt |G−t ) =

∑S−t

p(Gijt |S−t , G−t )p(S−t |G−t ) (10a)

=∑

Λ−i ,Λ

−j

p(Gijt |Λ−i ,Λ−j , G

−t )p(Λ−i ,Λ

−j |G

−t ).

(10b)

Equation 11 expresses the likelihood of an edge existingbetween two regions, given the value of the regions’ respec-tive label values

p(Gijt |Λi,Λj) =

{θΛi if Λi = Λj

0 if Λi 6= Λj, (11)

where θΛidenotes the likelihood that edges exist between

nodes with the same label. In practice, we assume a uniformsaliency prior for each label. Equation 10b measures thecosine similarity between the label distributions.

After a region edge is sampled from the spatial-semanticprior, a scan-match procedure attempts to find the bestalignment between the two regions. Upon convergence ofthe scan-match routine, the edge is accepted and is used toupdate the topology.

3) Region Merges: After a new region Ri has beencreated and edges to other regions have been checked andadded, the algorithm determines whether it is possible tomerge with each region to which it is connected. The newly-created region is merged with an existing (connected) regionif the observations associated with the smaller of the tworegions can be adequately explained by the larger region.This results in regions being merged when the robot revisits

locations already represented in the graph. This merge pro-cess is designed to ensure that the complexity of the topologyincreases only when the robot explores new areas, leading tomore efficient region edge proposals as well as more compactlanguage groundings.

B. Updating the Metric Map Based on New Edges

The algorithm then updates the spatial distribution overthe node poses Xt conditioned on the proposed topology,

p(Xt|Gt, zt, ut, λt) = N−1(Xt; Σ−1t , ηt), (12)

where we parametrize the Gaussian in the canonical form interms of the information matrix Σ−1

t and information vectorηt. We use the iSAM algorithm [18], which iteratively solvesfor the QR factorization of the information matrix.

C. Updating the Semantic Layer

Compared with Walter et al. [11], our updated representa-tion maintains a distribution over a larger set of semanticproperties associated with the environment. The distribu-tion over the semantic layer is maintained using a factorgraph [19] that is conditioned on the topology for eachparticle.

Fig. 4. Semantic Layer (plate representation)

As Fig. 4 shows, the semantic layer maintains two vari-ables associated with each region r, namely the region cate-gory Cr (e.g., hallway, conference room, etc.) and the labelsassociated with the region Λr. For example, a conferenceroom can have multiple labels, such as “meeting room” and“conference room.” The factor that joins these two variablesrepresents the likelihood of each room category generatinga particular label.

At each node n within a region, the robot can observeone or more of these semantic properties. In our currentimplementation, these are the region appearance observedfrom laser scanners ln or cameras in, and the region labelsλk (and the associated correspondence variables Φk). We runbelief propagation for each particle at each time step as newvariables and factors are added. We subsequently update thecategory and label distributions for each region.

The following subsections outline how we integrate ob-servations of these node properties into the semantic layer.

1) Integrating Semantic Classification Results: Each nodehas an appearance variable an that is related to its regioncategory. We consider several general appearance classes(“room”, “hallway,” and “open area”) that are then observedusing robot sensors. The factor that connects a region cate-gory variable Cr to an appearance variable an encodes the

likelihood of a region category generating an appearanceclass (e.g., how often does a conference room appear as aroom). The category-to-appearance factor was trained usingannotated data from several other floors of the Stata building.

We make two appearance observations an using the laserand camera observations at node n. These are representedin the factor graph as the laser appearance ln and the imageappearance in. We use two pre-trained appearance models forlaser observations and camera images. The laser appearanceclassification model has been trained using laser featuressimilar to those outlined in Mozos et al. [20], while the imageappearance model has been trained using CRFH [21]. Laserand camera appearance variables ln and in are connected tothe node’s appearance an using factors built from the con-fusion matrix for the two trained models. The classificationresults for the two sensors provide a distribution representingthe likelihood of the observations being generated from eachappearance category. The classifier outputs are integrated tothe factor graph as factors attached to variables ln and in.

2) Integrating Language Observations: The robot canalso receive either ego-centric (e.g., “I am at the kitchen”) orallocentric (e.g., “The kitchen is down the hall”) descriptionsof the environment from the user. We use these observationsto update the likelihood of observing labels in each region.We maintain the label for each region as another variableΛr in our factor graph. The region label is related to theregion category, as each category is more likely to generatesome labels than others. For example, while a person mightdescribe a “cafeteria” as a “dinning hall,” it is unlikely thatthey will describe it as an “office.” For our current exper-iments, we identified a limited subset of labels associatedwith each region category in our representation (e.g., thehallway category can generate “hall,” “hallway,” “corridor,”or “walkway”). When building these factors between labelsand room categories, we assign higher likelihoods to labelsassociated with each category and smaller likelihoods to theother labels (capturing the likelihood of generating theselabels given a particular room category).

A node can have zero, one, or multiple label observationsdepending on the way the person describes a location. Werepresent each label observation with a variable λk and acorrespondence variable Φk, which denotes the likelihoodthat the label was intended to describe the current location.The correspondence variable Φk is a binary-valued variablespecifying whether or not the label describes the region.If the label doesn’t correspond to that region (Φk = 0),the observation λk is uninformative about the region’s la-bel, and will have equal likelihood for each label value.However, when the correspondence holds (Φk = 1), thefactor encodes the likely co-occurrences between differentlabels. For example, if the robot heard the label “conferenceroom” with a high likelihood of Φk = 1, it will resultin other labels that often co-occur with “conference room”(e.g., “meeting room”) as having high likelihoods as well.Currently, high co-occurrence is added for words that aresynonyms (e.g., “hallway” and “corridor”). In this way, weuse the correspondence variable to handle the ambiguity

Fig. 5. Maximum likelihood semantic map of a multi-building environment on the MIT campus.

inherent in grounding natural language descriptions. When alabel is grounded to a region, we create a label observationλk and correspondence variable Φk, and connect it to theassociated region’s label variable Λr using a co-occurrencefactor. We integrate the correspondence observation Φk byattaching a factor encoding this likelihood. We treat theobserved label as having no uncertainty, as our current modeldoes not model errors arising from speech recognition.

We derive the factor denoting the observation of Φk basedon the type of description given by the user. If the user de-scribes the current location (e.g., “I am at the living room”),we have higher correspondence with spatially local nodes.For such descriptions, we allocate a high likelihood of cor-respondence with the current region, i.e., p(Φk = 1) = 0.8.For allocentric descriptions (e.g., “the kitchen is down thehallway”, where the user describes the location of the referent“kitchen” with relation to the landmark “hallway”), weuse the G3 framework [4] to calculate the correspondencelikelihood given the potential landmarks. We marginalizethe landmarks to arrive at correspondence likelihood of eachreferent region in a manner similar to our previous approach.

In handling allocentric descriptions, the current methodimproves upon our earlier approach in two ways. Firstly,we no longer require common landmarks to be explicitlydescribed before being able to ground the language correctly.We do this by leveraging the richer semantic representationmade possible by integrating additional semantic informationto arrive at likely landmarks. Secondly, while some expres-sions can be ambiguous (e.g., there could be multiple regionsdown the hall), the presence of other semantic cues increasesthe accuracy of the final label distribution since incorrectgroundings will have less impact if the region’s appearanceis different from the label observation.

D. Updating the Particle Weights and Resampling

Particle weights are updated and resampling in the samefashion as Walter et al. [11].

V. RESULTS

We evaluate our algorithm through four experiments inwhich a human gives a robotic wheelchair [9] a narratedguided tour of the Stata Center (S3, S4, S6) as well as amulti-building indoor tour (MF) on the MIT campus. The

robot was equipped with a forward-facing LIDAR, a camera,wheel encoders, and a microphone. In these experiments wedrove the robot using a joystick, and provided it with textualnatural language descriptions at specific salient locations.

We evaluate the resulting semantic maps with regards totheir topological accuracy, compactness, segmentation accu-racy, and semantic accuracy. All experiments were run with10 particles. The results show that our framework producesmore compact and more accurate semantic graphs than ourprevious approach. They also demonstrate the improvementin semantic accuracy due to language descriptions. We alsoshow the ability of our framework to ground complex lan-guage even in the absence of previous labels for the referent(e.g., it handles the expression “the lobby is down the hall”even when the hall has not been labeled).

A. Topological accuracy

We compare the topological accuracy, conditioned uponthe resulting segmentation, by comparing the maximumlikelihood map with the ground truth topology. We definea topology as matching ground truth if node pairs that arespatially close (1 m) in a metrically accurate alignment are atmost one region hop away. This avoids penalizing occasionalregions that do not contain valid edges as long as a nearbyregion was accurately connected (and possibly merged withthe nodes from a prior visit). This can happen when an edgewas not sampled or when scan-matching failed to converge.

The percentage of close node pairs that were more thanone region hop away from each other for the third, fourth andsixth floor were 2.8%, 3.7%, and 3.8%, respectively. Mostregion-matching errors occurred in areas with significantclutter, causing scan-matching failures. Metric maps derivedfrom the maximum likelihood particles were accurate for allthree floors.

B. Topological Compactness

We compare the allocation of nodes to regions in thecurrent framework to the previous method. In the previousapproach, the topology update did not merge regions evenwhen the robot revisited a region; it simply created an edgebetween the regions. The redundancy of the regions has sev-eral negative implications. Firstly, it unnecessarily increasesthe hypothesis space of possible region edges, reducing

Fig. 6. Maximum likelihood semantic map of the 3rd floor of the Statabuilding (pie charts denote the likelihood of different region categories).

TABLE IREGION ALLOCATION EFFICIENCY (Sc)

Floor Proposed OldFramework Framework

Stata Floor 3 (S3) .67 .29Stata Floor 4 (S4) .77 .37Stata Floor 6 (S6) .69 .52

the likelihood of a sample proposing valid region edges.Secondly, it increases the hypothesis space for groundinglanguage, forcing the framework to consider more regionpairs as possible groundings for user descriptions.

We measure the duplicity of the region allocation as

Sc = Ns/Nt, (13)

where Ns is the number of close node pairs (< 1m) assignedto the same region and Nt is the total number of closenode pairs. If the topology is efficient at allocating regions,this ratio should be high, as only nodes near region bound-aries should belong to different regions. Table I comparesthese scores three different floors. The new method scoressignificantly higher in all three experiments. The differencebecomes more pronounced as the robot revisits more regions.Since the sixth floor dataset did not have too many revisitedregions, the scores for the two approaches are closer.

C. Segmentation Accuracy

Table II outlines the segmentation accuracy for the maxi-mum likelihood particle for two datasets, outlined accordingto region type. We picked the best matches based on theJaccard index (number of intersecting nodes divided by thenumber of union nodes) for each ground truth annotatedregion and the resulting segmented region. Since our segmen-tation method depends on the similarity of laser observations,large cluttered region types such as lab spaces and loungestend to be over-segmented. Additionally long hallways tendto be over-segmented by our method, which is reflected inthe lower scores for hallways.

D. Inference of Semantic Properties

Table II also outlines the semantic accuracy for the maxi-mum likelihood particle for two datasets. Semantic accuracywas calculated for each ground truth region by assigning eachconstituent node with its parent region’s category distribution

(a) Appearance only (b) With language

Fig. 7. Region category distribution (a) for a region with only appearanceinformation and (b) and with both appearance and language “the lounge isbehind us”. (category “lounge”: yellow).

TABLE IIREGION SEGMENTATION AND SEMANTIC ACCURACY

Region Segmentation Semantic AccuracyType Accuracy Without Lang With Lang

S3 MF S3 MF S3 MFConference room 80.0 81.7 8.8 15.1 48.5 58.7Elevator lobby 59.7 72.8 18.8 12.8 64.1 46.4Hallway 49.4 55.7 44.5 58.5 44.4 58.0Lab 52.8 30.1 11.8 27.2 14.2 30.6Lounge 42.9 39.4 28.6 36.6 62.0 40.5Office 62.5 76.1 78.1 45.6 98.6 60.2

and taking the cosine similarity. We observe that the semanticaccuracy with language is higher for most region types, withthe exception of hallways that show minimal improvementsince they were rarely described by users. Some regions suchas labs, which were labeled with egocentric descriptions,have low scores because the regions are over-segmentedand the language is attributed only to the current region.Figure 7 compares the region category properties with andwithout language. In the absence of language (Fig. 7(a)),the appearance of the region gives equal likelihood for both“elevator lobby” and “lounge.” In Fig. 7(b), the region wasgrounded with the label “lounge” and the framework inferreda higher likelihood of the region category being a lounge.

E. Grounding Allocentric Language Descriptions

We also tested our framework with allocentric languagedescriptions. When handling phrases that include a landmarkand a referent (e.g., “the gym is down the hall”), ourearlier framework required the landmark to have alreadybeen labeled before describing the referent location. Withour new framework, the robot is able to ground languagewhen the landmark corresponds to locations that may nothave been labeled, but can be inferred from other semanticcues (e.g., appearance classification). We tested this situationusing several instances in our dataset.

Figure 8 shows instances in which allocentric languageutterances were grounded into the semantic graph. As thelabel distributions for the surrounding regions demonstrate,the framework is able to ground the referent with a highdegree of accuracy, even though the landmark was neverexplicitly labeled. However, since there is more uncertaintyabout the landmark region, the information derived fromthe allocentric language has less influence on the semanticproperties on the region (since we marginalize the landmarklikelihood when calculating the grounding likelihood Φ).

(a) (b)

Fig. 8. Resulting category distribution for complex language phrases (a)“the elevator lobby is down the hall” and (b) “the lounge is down the hall”.

VI. CONCLUSION

We described a framework that enables robots to constructspatial-semantic representations of environments from natu-ral language descriptions and scene classifications. Comparedto earlier methods, our approach handles natural languagedescriptions more efficiently, and produces semantic repre-sentations that are richer and more compact.

Currently, our method grounds descriptive language underthe assumption that it refers to concepts that exist in therobot’s representation, either due to previously interpreteddescriptions or to appearance classification. We are exploringa mechanism that reasons about the presence of the referredentities and grounds the utterance only when it is confidentabout the validity of the grounding. We also plan to inte-grate additional semantic cues that inform region labels andcategories, such as the presence of common objects foundindoors. In our current implementation, the factors betweenregion category and labels were constructed using a set ofpredefined labels and relationships identified by us (usingsynonyms for region types). In the future we plan to learnthese relationships from data.

Our approach maintains multiple hypotheses regarding theregion assignments of the topology by using particles tosample region assignments based on a prior that considersthe spatial similarity of nodes. However, particle weightsare calculated based only on laser data. This can lead tovalid segmentation hypotheses being discarded during re-sampling if the laser observation likelihood is low. We planto incorporate region hypothesis scores based on appearance.In the current approach, the robot passively accepts humanassertions about the environment. We also plan to exploremore interactive scenarios where the robot reasons over itshypothesis space and asks questions of the human to resolveambiguities.

VII. ACKNOWLEDGMENTS

We would like to thank S. Shemet and F. Paerhati fortheir help collecting and annotating datasets as well as intraining the appearance models. This work was supported inpart by the Robotics Consortium of the U.S Army ResearchLaboratory under the Collaborative Technology AllianceProgram, Cooperative Agreement W911NF-10-2-0016.

REFERENCES[1] G. Bugmann, E. Klein, S. Lauria, and T. Kyriacou, “Corpus-based

robotics: A route instruction example,” Proc. Intelligent AutonomousSystems, pp. 96–103, 2004.

[2] J. Dzifcak, M. Scheutz, C. Baral, and P. Schermerhorn, “What to doand how to do it: Translating natural language directives into temporaland dynamic logic representation for goal management and actionexecution,” in Proc. IEEE Int’l Conf. on Robotics and Automation(ICRA), 2009, pp. 4163–4168.

[3] C. Matuszek, D. Fox, and K. Koscher, “Following directions usingstatistical machine translation,” in Proc. ACM/IEEE Int’l. Conf. onHuman-Robot Interaction (HRI), 2010, pp. 251–258.

[4] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee,S. Teller, and N. Roy, “Understanding natural language commandsfor robotic navigation and mobile manipulation,” in Proc. Nat’l Conf.on Artificial Intelligence (AAAI), 2011, pp. 1507–1514.

[5] D. L. Chen and R. J. Mooney, “Learning to interpret natural languagenavigation instructions from observations,” in Proc. Nat’l Conf. onArtificial Intelligence (AAAI), 2011, pp. 859–865.

[6] K. Nickel and R. Stiefelhagen, “Visual recognition of pointing gesturesfor human-robot interaction,” Image and Vision Computing, vol. 25,no. 12, pp. 1875–1884, Dec. 2007.

[7] B. Kuipers, “The spatial semantic hierarchy,” Artificial Intelligence,vol. 119, no. 1, pp. 191–233, 2000.

[8] H. Zender, O. Martı́nez Mozos, P. Jensfelt, G. Kruijff, and W. Bur-gard, “Conceptual spatial representations for indoor mobile robots,”Robotics and Autonomous Systems, vol. 56, no. 6, pp. 493–502, 2008.

[9] S. Hemachandra, T. Kollar, N. Roy, and S. Teller, “Following andinterpreting narrated guided tours,” in Proc. IEEE Int’l Conf. onRobotics and Automation (ICRA), 2011, pp. 2574–2579.

[10] A. Pronobis and P. Jensfelt, “Large-scale semantic mapping andreasoning with heterogeneous modalities,” in Proc. IEEE Int’l Conf.on Robotics and Automation (ICRA), 2012, pp. 3515–3522.

[11] M. R. Walter, S. Hemachandra, B. Homberg, S. Tellex, and S. Teller,“Learning semantic maps from natural language descriptions,” in Proc.Robotics: Science and Systems (RSS), 2013.

[12] M. MacMahon, B. Stankiewicz, and B. Kuipers, “Walk the talk:Connecting language, knowledge, and action in route instructions,”in Proc. Nat’l Conf. on Artificial Intelligence (AAAI), 2006, pp. 1475–1482.

[13] M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams,M. Bugajska, and D. Brock, “Spatial language for human-robotdialogs,” IEEE Trans. on Systems, Man, and Cybernetics, Part C:Applications and Reviews, vol. 34, no. 2, pp. 154–167, 2004.

[14] T. Kollar, S. Tellex, D. Roy, and N. Roy, “Toward understandingnatural language directions,” in Proc. ACM/IEEE Int’l. Conf. onHuman-Robot Interaction (HRI), 2010, pp. 259–266.

[15] S. Tellex, P. Thaker, R. Deits, T. Kollar, and N. Roy, “Towardinformation theoretic human-robot dialog,” in Proc. Robotics: Scienceand Systems (RSS), 2012.

[16] C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox,“A joint model of language and perception for grounded attributelearning,” in Proc. Int’l Conf. on Machine Learning (ICML), 2012.

[17] J.-L. Blanco, J. Gonzalez, and J. Fernandez-Madrigal, “Consistentobservation grouping for generating metric-topological maps thatimproves robot localization,” in Proc. IEEE Int’l Conf. on Roboticsand Automation (ICRA), 2006.

[18] M. Kaess, A. Ranganathan, and F. Dellaert, “iSAM: Incrementalsmoothing and mapping,” Trans. on Robotics, vol. 24, no. 6, pp. 1365–1378, 2008.

[19] J. M. Mooij, “libDAI: A free and open source C++ library for discreteapproximate inference in graphical models,” J. Machine LearningResearch, vol. 11, pp. 2169–2173, Aug. 2010.

[20] O. Mozos, C. Stachniss, and W. Burgard, “Supervised learning ofplaces from range data using adaboost,” in Proc. IEEE Int’l Conf. onRobotics and Automation (ICRA), 2005.

[21] A. Pronobis, O. M. Mozos, B. Caputo, and P. Jensfelt, “Multi-modal semantic place classification,” Int’l J. of Robotics Research,vol. 29, no. 2–3, pp. 298–320, Feb. 2010. [Online]. Available:http://www.pronobis.pro/publications/pronobis2010ijrr