Efficient Generation of Motion Plans from Attribute-Based NaturalLanguage Instructions Using Dynamic Constraint Mapping

Jae Sung Park, Biao Jia, Mohit Bansal, and Dinesh Manochahttp://gamma.cs.unc.edu/SafeMP/NLP/ (full video)

Abstract— We present an algorithm for combining naturallanguage processing (NLP) and fast robot motion planningto automatically generate robot movements. Our formulationuses a novel concept called Dynamic Constraint Mapping totransform complex, attribute-based natural language instruc-tions into appropriate cost functions and parametric constraintsfor optimization-based motion planning. We generate a factorgraph from natural language instructions called the DynamicGrounding Graph (DGG), which takes latent parameters intoaccount. The coefficients of this factor graph are learnedbased on conditional random fields (CRFs) and are used todynamically generate the constraints for motion planning. Wemap the cost function directly to the motion parameters of theplanner and compute smooth trajectories in dynamic scenes.We highlight the performance of our approach in a simulatedenvironment and via a human interacting with a 7-DOF Fetchrobot using intricate language commands including negation,orientation specification, and distance constraints.

I. INTRODUCTION

In the field of human-robot interaction (HRI), naturallanguage has been used as an interface to communicate ahuman’s intent to a robot [1], [2], [3], [4]. Much of the workin this area is related to specifying simple tasks or commandsfor robot manipulation, such as picking up and placingobjects. As robots are increasingly used in complex scenariosand applications, it is important to develop a new generationof motion planning and robot movement techniques thatcan respond appropriately to diverse, attribute-based NLPinstructions for HRI, e.g., instructions containing negationbased phrases or references to position, velocity, and distanceconstraints. Furthermore, we need efficient techniques toautomatically map the NLP instructions to such motionplanners.

Humans frequently issue commands that include sentenceswith orientation-based or negation constraints such as “put abottle on the table and keep it upright” or “move the knifebut don’t point it towards people,” or sentences with velocity-based constraints such as “move slowly when you get closeto a human.” To generate robot actions and movements inresponse to such complex natural language instructions, weneed to address two kinds of challenges:1. The accurate interpretation of attribute-based natural lan-guage instructions and their grounded linguistic semantics,especially considering the environment and the context.2. The motion planner needs to generate appropriate trajec-tories based on these complex natural language instructions.This includes appropriately setting up the motion planningproblem based on different motion constraints (e.g., orien-

(a) (b) (c)Fig. 1. The Fetch robot is moving a soda can on a table based onNLP instructions. Initially the user gives the “pick and place” command.However, when the robot gets closer to the book, the person says “don’t putit there” (i.e. negation) and the robot uses our dynamic constraint mappingfunctions and optimization-based planning to avoid the book. Our approachcan generate appropriate motion plans for such attributes.

tation, velocity, smoothness, and avoidance) and computingsmooth and collision-free paths.

At a high level, natural language instructions can bedecomposed into task description and attributes. Task de-scriptions are usually verb or noun phrases that describe theunderlying task performed by a robot. The attributes includevarious adjectives, adverbs, or prepositional phrases that areused to specify additional conditions the robot must (or mustnot) satisfy. For example, these conditions may specify someinformation related to the movement speed, the orientation,the physical space characteristics, or the distances. Therefore,it is important to design motion planners that consider theserobotic task descriptions and robot motion constraints.Main Results: We present an algorithm for generating pa-rameterized constraints for optimization-based motion plan-ning from complex, attribute-based natural language instruc-tions. We use Dynamic Grounding Graphs (DGG) to parseand interpret the commands and to generate the constraints.Our formulation includes the latent parameters in the ground-ing process, allowing us to model many continuous variablesin our grounding graph. Furthermore, we present a newdynamic constraint mapping that takes DGG as the input andcomputes different constraints and parameters for the motionplanner. The appropriate motion parameters are speed, orien-tation, position, smoothness, repulsion, and avoidance. Thefinal trajectory of the robot is computed using a constraintoptimization solver. Compared to prior techniques, our over-all approach offers the following benefits:

• The inclusion of latent parameters in the groundinggraph allows us to model continuous variables that areused by our mapping algorithm. Our formulation com-putes the dynamic grounding graph based on conditionalrandom fields.

• We present a novel dynamic constraint mappingused to compute different parametric constraints for

optimization-based motion planning.• Our grounding graphs can handle more complex,

attribute-based natural language instructions and ourmapping algorithm uses appropriate cost functions asparameters over the continuous space. Compared toprior methods, our approach is much faster and betterable to handle more complex and attribute-based naturallanguage instructions.

We highlight the performance of our algorithms in a sim-ulated environment and on a 7-DOF Fetch robot operatingnext to a human. Our approach can handle a rich set ofnatural language commands and can generate appropriatepaths. These include complex commands such as picking(e.g., “pick up a red object near you”), correcting the motion(e.g., “don’t pick up that one”), and negation (e.g., “don’tput it on the book”).

II. RELATED WORK

Most algorithms used to map natural language instructionto robot actions tend to separate the problem into two parts:parsing and motion planning computation. In this section,we give a brief overview of prior work in these areas.

A. Natural Language Processing

Nyga et al. [6], [7], [8], [9] have developed Probabilis-tic Robot Action Cores through which robots understandnatural language instructions from a knowledge base. Thenatural language understanding system creates a proba-bilistic model for converting taxonomies to semantics andfind the combination of semantics that gives the highestprobability. Duvallet et al. [10] use a probabilistic graph-ical learning model called Generalized Grounding Graphs(G3) on a ground vehicle for a navigation problem givennatural language commands. Branavan et al. [3], [11] usereinforcement learning to learn a mapping from naturallanguage instructions and then apply it to sequences ofexecutable actions. Matuszek et al. [4] use a statisticalmachine translation model to map natural language instruc-tions to path description language, which allows a robot tonavigate while following directions. Duvallet et al. [12] useimitation learning to train the model through demonstrationsof humans following directions. Paul et al. [13] propose theAdaptive Distributed Correspondence Graph (ADCG). Arkinet al. [14] further extend DCG, proposing the HierarchicalDistributed Correspondence Graph (HDCG), which definesconstraints as discrete inequalities and grounds word phrasesto corresponding inequalities. Chung et al. [15] use HDCGon ground vehicles to implement navigation commands anddemonstrate performance improvements over G3 in terms ofrunning time, factor evaluations, and correctness. We chosethe DCG model as our base framework, because it has aflexibility for many applications and extensions based on theparse tree from the input natural language instructions.

B. Robot Motion Planning in Dynamic Environments

Many replanning algorithms have been suggested to gener-ate collision-free motion plans in dynamic environments. Fox

Fig. 2. Overall pipeline of our approach highlighting the NLP parsingmodule and the motion planner. Above the dashed line (from left toright): Dynamic Grounding Graphs (DGG) with latent parameters that areused to parse and interpret the natural language commands, generationof optimization-based planning formulation with appropriate constraintsand parameters using our mapping algorithm. We highlight the high-levelinterface below the dashed line.

et al. [16] propose the dynamic window approach to computeoptimal velocity in a short time window. Optimization-based motion planners [17], [18], [19] solve a constrainedoptimization problem to generate smooth and collision-freerobot paths. We present an automatic scheme that generatesthe motion planning problem from NLP instructions.

There is some work on integrating optimization-based mo-tion planning with NLP in 2D workspaces. Silver et al. [20]develop an algorithm for learning navigation cost functionsfrom demonstrations. Howard et al. [2] use a probabilisticgraphical model to generate motion planning constraints fora 2D navigation problem. Compared to these methods, ourapproach can handle 3D workspaces and high-dimensionalconfiguration spaces to generate robot motions correspond-ing to complex NLP instructions. Other techniques focuson efficiency in human-robot collaborative tasks. MarkovDecision Processes (MDP) are widely used to compute thebest robot action policies [21], [22]. These techniques arecomplementary to our approach.

III. DYNAMIC GROUNDING GRAPHS

Fig. 2 shows the basic pipeline of our approach. Whennatural language commands are given as input, the NLPmodule (upper left) creates an optimization problem for amotion planning module (upper middle). The robot motiontrajectory is then computed from the motion planning module(upper right). As the planned trajectory is executed (bottomright), the result is fed back to the NLP module. In thissection, we present the algorithms used in the NLP module.

We extend the ideas of the Generalized Grounding Graphs(G3) model and the Distributed Correspondence Graph(DCG) model [2] by including the latent variables in thegrounding graph and using them to compute the constraintsfor motion planning. The input to our algorithm is the naturallanguage instruction. We do not account for any errors due tovoice recognition. From a natural language command input,we construct a factor graph, as shown in Fig. 3(a), which isbased on the parsing of the command. For each node of theparse tree, we generate three types of nodes: word phrasenode λ , grounding node γ , and correspondence node φ .

The input sentence Λ is parsed using the NLTK li-brary [23]. The word phrase of each node in the parse treeis denoted as λi for i = 1,2, · · · . Children of λi are λi1, · · · ,λim. The root node of the parse tree is λ1. For example, inFig. 3(a), the input sentence is “Put the cup on the table.”The parse tree has the root word phrase λ1 =“Put”. Its nounλ2 =“the cup” and the preposition λ3 =“on,” which are thechildren nodes of the root node. The noun phrase λ4 =“thetable” is the child node of λ3.

Our goal is to compute a mapping from a natural languagesentence Λ to the cost function parameters H given therobotic environment E where the robot is operating. E isa representation of the environment, which is composed ofobstacle positions, orientations, and the robot’s configuration.Feature vectors are constructed in the factor graph from thedescription of the environment. H is a real-valued vectorthat contains all cost function parameters and weights usedin the optimization-based motion planner. It also includesthe weights of different types of cost functions used in theoptimization formulation.

We first compute the groundings γi of each word phraseλi. The grounding of each word phrase is the mapping fromthe word phrase to its meaning in the real world. Groundingscan be objects, locations, motions, tasks, or constraints. Inour model, the grounding γi depends on its word phraseλi and its children grounding nodes γi1, · · · , γim, wherethe tree structure of the grounding nodes follows the parsetree. Correspondence node φi indicates the correct matchingbetween the word phrase λi and the grounding γi. It isa binary variable; φi is true if the word phrase and thegrounding match and f alse if they do not.

A. Latent Parameters

A key novel component of our approach is the inclusionof latent variables in the grounding graph. Our primary goalis to compute the best cost function parameters H to be useddirectly for optimization-based motion planning. We denoteH ∈Rh, a real vector of size h, as a collection of cost functionparameters. In this case, the size h and the number of costfunction parameters depend on the types of cost functionsthat are used. 1 From the predicted groundings γi, thecost function parameters in the motion planning formulation(Fig. 3(b)) are inferred through the latent variable H. Hcontains all the cost function parameters (e.g., weights ofcost functions, locations, and orientations). Details about thecost functions are described in Section IV.

In Fig. 3(b), the resulting constraint-based motion planningproblems are shown. We use the collision avoidance costfunction as the default smoothness cost function and thetarget location cost function, though weights can vary. Thetarget location, whose 3D coordinates are the cost functionparameters, is set on the surface of the table. The costfunction parameter node H contains the weights of the

1In this paper, we set maximum h = 22 to fully specify the smoothness,the end-effector position, the end-effector orientation, the end-effector speed,and the repulsion cost functions. It is a sum of three terms: 5 for weights,16 for positions and orientations, and 1 for an exponential constant.

parameters and the 3D coordinates of the target location.In the bottom of Fig. 3(b), where a new “Don’t” commandis given, a repulsion cost function is added. Thus, the costfunction weight and the location of the repulsion source(below the robots end-effector position) are added to H.

B. Probabilistic Model

We present a new probabilistic model to compute H,Λ and E. We assume the conditional independence of theprobabilities so that we can construct a factor graph (seeFig. 3(a)). With the independence assumptions, a singlefactor is connected to a word phrase node and its childrengrounding nodes, which contain information about the sub-components. This graphical representation corresponds to thefollowing equation:

p(H|λ1, · · · ,λn,E) = p(H|γ1,E)∏i

p(γi|λi,φi,γi1, · · · ,γim,E).

(1)

For the root factor connecting H, γ1 and E, we formulate thecontinuous domain of H. We compute the Gaussian MixtureModel (GMM) on the probability distribution p(H|γ1,E) andmodel our probability with non-root factors as follows:

p(γi|λi,φi,γi1, · · · ,γim,E) (2)

=1Z

exp(−θTi f (γi,λi,φi,γi1, · · · ,γim,E)), (3)

where Z is the normalization factor, and θi and f are thelog-linearizations of the feature function. The function fgenerates a feature vector given a grounding γi, a wordphrase λi, a correspondence φi, children groundings γi j,and the environment E. The information from the roboticenvironment is used in the feature function f and in thelog-linearized feature function f . The attributes of objects inthe robotic world such as shapes and colors are encoded asmultidimensional binary vectors, which indicate whether theobject has a given attribute.Word phrases. The feature vector includes binary-valuedvectors for the word and phrase occurrences, and Part ofSpeech (PoS) tags. There is a list of words that could beencountered in the training dataset such as {put, pick, cup,up, there, · · · }. The word list is constructed from H2SLdataset [13] that contains 1672 word phrases. If the wordphrase contains the word “put,” then the occurrence vectorat the first index is set to 1 and the others are set to 0. Thislist also includes real-valued word similarities between theword and the pre-defined seed words. The seed words are thepre-defined words that the users expect to encounter in thenatural language instructions. We used Glove word2vec [24]to measure cosine-similarity (i.e. the inner product of twovectors divided by the lengths of the vectors) between thewords.Robot states. From the robot state, we collect the robotjoint angles, the velocities, the end-effector position, the end-effector velocity, etc. This information can affect the costfunction parameters even while processing the same naturallanguage commands. We also store information about theobjects that are close to the robot.

(a) (b) (c)

Fig. 3. Factor graphs for different commands: In the environment inthe right-hand column, there is a table with a thin rectangular object on it.A robot arm is moving a cup onto the table, but we want it to avoid movingover the book when given NLP instructions. (a) The command “Put the cupon the table” is given and the factor graph is constructed (left). Appropriatecost functions for the task are assigned to the motion planning algorithm(middle) and used to compute the robot motion (right). (b) As the robotgets close to the book, another command “Don’t put it there” is given witha new factor graph and cost functions.

C. Factor Graph using Conditional Random Fields

We represent our dynamic grounding graph as a factorgraph. We build a factor graph based on the probabilisticmodel described in Section III-B and use it to train andto infer the meaning of given commands. In particular, weuse Conditional Random Fields (CRF) [25] as a learningmodel for factor graphs because CRFs are a good fit forapplying machine learning to our probabilistic graph modelwith conditional probabilities.

At the inference step, we used the trained CRF factorgraph models to find the best groundings Γ and the costfunction parameters H by solving the CRF maximizationproblem

maximizeH,γ1,··· ,γn

p(H|γ1,E)∏i

1Z

exp(θ Ti f (γi,λi,φi,γi1, · · · ,γim,E)).

(4)

When the nodes H, γ1, · · · ,γn are optimized, they createa tree structure in the factor graph, meaning that we cansolve the optimization problem efficiently using dynamicprogramming. Each factor depends on its parent and childrenvarying variables and other fixed variables connected to it.This implies that we can solve the sub-problems in a bottom-up manner and combine the results to solve the biggerproblem corresponding to the root node.

During the training step of CRF, we solve the optimizationproblem of maximizing the probability of the samples in thetraining dataset over the feature coefficients θi for every parsetree structure. The optimization problem is the same as Eq.(4), except that the problem is solved over θ instead of Hand Γ. More details are given in [26].

IV. DYNAMIC CONSTRAINT MAPPING WITH NLP INPUT

We use an optimization-based algorithm [27] to solve thecost minimization problem. The function and constraintsof this cost minimization problem come from DGG, asexplained in Sec. III. In this section, we present our mappingalgorithm, Dynamic Constraint Mapping, which maps the

word phrase groundings to proper cost function parameterscorresponding to natural language instructions [26].

A. Robot Configurations and Motion Plans

We denote a single configuration of the robot as a vector q,which consists of joint-angles or other degrees-of-freedom.A configuration at time t, where t ∈ R, is denoted as q(t).We assume q(t) is twice differentiable, and its derivativesare denoted as q′(t) and q′′(t). The n-dimensional space ofconfiguration q is the configuration space C . We representbounding boxes of each link of the robot as Bi. The boundingboxes at a configuration q are denoted as Bi(q).

For a planning task with a given start configuration q0and derivative q′0, the robot’s trajectory is represented by amatrix Q, whose elements correspond to the waypoints [17],[18], [27]:

Q =

q0 q1 qn−1 qnq′0 q′1 · · · q′n−1 q′n

t0 = 0 t1 tn−1 tn = T

. (5)

The robot trajectory passes through n + 1 waypointsq0, · · · ,qn, which will be optimized by an objective functionunder constraints in the motion planning formulation. Robotconfiguration at time t is interpolated from two waypoints.Formally, for j such that t j ≤ t ≤ t j+1, the configuration q(t)and derivative q′(t) are cubically interpolated using q j, q′j,q j+1, and q′j+1.

The i-th cost functions of the motion planner are Ci(Q).Our motion planner solves an optimization problem withnon-linear cost functions and linear joint limit constraintsto generate robot trajectories for time interval [0,T ],

minimizeQ ∑

iwiCi(Q)

subject toqmin ≤ q(t)≤ qmax,q′min ≤ q′(t)≤ q′max

0≤ ∀t ≤ T.(6)

In the optimization formulation, Ci is the i-th cost functionand wi is the weight of the cost function.

B. Parameterized Constraints

The overall optimization formulation is given in Eqn. 6.To formulate the constraints, we use the following cost func-tions, which are designed to account for various attributesin the NLP instructions, including collision avoidance. Inour formulation, we use many types of cost functions suchas collision avoidance, robot smoothness, robot end-effectorspeed, target positions, and target orientations. These costfunctions are used to handle many attributes of naturallanguage instructions. Each cost function has its weight andmay also have other cost function parameters, if necessary.

To handle various attributes, we use the following param-eterized constraints in our optimization formulation.Collision avoidance:

Ccollision(Q) =∫ T

0∑

i∑

jdist(Bi(t),O j)

2dt, (7)

where dist(Bi(t),O j) is the penetration depth between a robotbounding box Bi(t) and an obstacle O j.

Smoothness: This cost function corresponds to the integralof the first derivative of joint angles over the trajectoryduration.End-effector position:

Cposition(Q) =∫ T

0||pee(t)−ptarget ||2dt, , (8)

where pee(t) is the robot end-effector position at time t andptarget is the target position. The target position ptarget isconsidered as a cost function parameter.End-effector orientation:

Corientation(Q) =∫ T

0angledist(qee(t),qtarget)

2dt, (9)

Cupvector(Q) =∫ T

0angledist(nup(t),ntarget)

2dt, (10)

where qee(t) is the quaternion representation of the robotend-effector’s orientation at time t, qtarget is the end-effectororientation that we want the robot to maintain, nup is thenormal up-vector of the robot’s end-effector, and ntarget isthe target up-vector. As with the end-effector position cost,the target orientation qtarget is the cost function parameter.End-effector speed:

Cspeed(Q) =∫ T

0||vee(t)−vtarget ||2dt, (11)

where vee(t) is the robot’s end-effector speed at time t andvtarget is the target speed. The parameters of this cost functioncorrespond to vtarget .Repulsion:

Crepulsion(Q) =∫ T

0exp(−c||pee(t)−pr||)dt, (12)

where pr is the position to which we don’t want the robotto move. The coefficient c > 0 suggests how much the costis affected by ||pee(t)−prepulsive||, the distance between theend-effector position and the repulsion source.

V. IMPLEMENTATION AND RESULTS

We have implemented our algorithm and evaluated itsperformance in a simulated environment and on a 7-DOFFetch robot. All the timings are generated on a multi-core PC with Intel i7-4790 8-core 3.60GHz CPU and a16GB RAM. We use multiple cores for fast evaluation andparallel trajectory search to compute a good solution to theconstrained optimization problem [27].

A. Training DGGs for Demonstrations

We describe how the training dataset for our DGG modelwere generated. The training dataset for DGGs requires threecomponents: a natural language sentence, a robotic envi-ronment, and the cost function parameters for optimization-based motion planners.

For each demonstration, we write tens of different sen-tences that specify the take goals the constraints for the mo-tion plans with different nouns, pronouns, adjectives, verbs,adverbs, preposition, etc. For each sentence, we generate arandom robotic environment and an initial state for the robot.

(a) (b) (c)Fig. 4. The simulated Fetch robot arm reaches towards one of the two redobjects. (a) When a command “pick up one of the red objects” is issued, therobot moves to the red object on the right because of the DGG algorithm. (b)If the user doesn’t want the robot to pick up the object on the right, he/sheuses a command “don’t pick up that one.” Our DGG algorithm dynamicallychanges the cost function parameters. (c) The robot approaches the objecton the right and stops.

(a) (b) (c)Fig. 5. In this simulated environment, the human instructs the robot to“put the cube on table” (a). As it approaches the laptop (b), the humanuses a negation NLP command “don’t put it there,” so the robot places itat a different location (c).

In addition, the robot joint values and joint velocities arerandomly set as initial states. We collect tens or hundredsof random robotic environments. For a natural languagesentence, a random robotic environment, and a random initialstate for the robot, the cost function parameters are assignedmanually or synthesized from other examples. Hundreds ofmultiple data samples are generated from generated datasamples by switching the correspondence variable in theDGG model from 1 (true) to 0 (false) and changing thegrounding variables to the wrong ones to match the falsecorrespondence variable. The training dataset is created withup to 100,000 samples in our experiments. When the costfunction parameters are determined, the optimization-basedmotion planner is used to compute a feasible robot trajectory.In the optimization-based motion planning algorithm, thereare some waypoints through which the robot trajectoryshould pass. For the robot’s safety, we check if the robottrajectory with given cost function parameters is in- collisionand appropriately set a higher value of the coefficient of thecollision cost and compute a new trajectory. This process isrepeated until the trajectory is collision-free.

We use different training data for each scenario. For thescenarios shown in Fig. 4, the initial pose of the robot in frontof the table and the positions of the blue and red objectson the table are randomly set. For “Pick up” commands,appropriate cost function parameters are computed so that therobot picks up a blue or red object depending on the givencommand. Similarly, in Fig. 5, the position and orientation ofthe laptop is initialized randomly. Given the “Put” command,we create an end-effector position cost function so that therobot places the object on the table; and a repulsive costfunction to avoid the laptop position.

TABLE IPLANNING PERFORMANCES WITH VARYING SIZES OF TRAINING DATA

FOR THE SCENARIO IN FIG. 4 WITH 21 DIFFERENT NLP INSTRUCTIONS.# Training Data Success Rate Duration Smoothness Cost

1,000 5/10 23.46s (5.86s) 8.72 (5.56)3,000 9/10 16.02s (3.28s) 2.56 (0.64)10,000 10/10 13.16s (1.24s) 1.21 (0.32)30,000 10/10 12.81s (0.99s) 0.78 (0.12)

100,000 10/10 12.57s (0.97s) 0.72 (0.10)

TABLE IIRUNNING TIME OF OUR DGG AND MOTION PLANNING MODULES.

Scenarios Instructions |H| DGGTime

PlanningTime

Pick up an object (Fig. 4) 10 12 32ms 93msDon’t put on laptop (Fig. 5) 20 13 16ms 98ms

Move around obstacle 45 9 16ms 95msStatic Instructions (video) 20 18 73ms 482ms

Dynamic Instructions (Fig. 1) 21 18 58ms 427ms

B. Simulations and Real Robot Demonstrations

We evaluate the performance on optimization problemsthat occur in complex environments composed of multipleobjects. Based on the NLP commands, the robot decides topick an appropriate object or is steered towards the goalposition in a complex scene. In particular, the user givesNLP commands such as “move right,” “move up,” “moveleft,” or “move down” to guide the robot. For each suchcommand, we compute the appropriate cost functions.

In terms of the 7-DOF Fetch robot, we test the perfor-mance of our approach on different tasks corresponding to:(1) moving a soda can on the table from one position to another; (2) not moving the soda can over the book [26].

C. Analysis

We evaluate the performance based on the following:Success Rate: The ratio of successful task completion amongall trials. Failure includes colliding with the obstacles due toan incorrect mapping of cost function parameters, violatingconstraints specified by natural language commands, and notcompleting the task due to some other reason.Trajectory Duration: The time between the giving of thefirst NLP command and the robot’s successful completionof the task after trajectory computation. A shorter durationimplies a higher performance.Trajectory Smoothness Cost: A cost based on evaluatingthe trajectory smoothness according to standard metrics anddividing it by the trajectory duration. A lower cost impliesa smoother and more stable robot trajectory.

Table I shows the results on our benchmarks with varyingnumbers of training data samples in the simulation envi-ronment shown in Fig. 4. When the number of trainingdata samples increases, the success rate also increases whilethe trajectory duration and the trajectory smoothness costdecrease. Table II shows the running time of our algorithmand the distances from the obstacle on the table in the real-world scenarios.

VI. BENEFITS AND COMPARISONS

Most prior methods that combine NLP and motion plan-ning have focused on understanding natural language instruc-tions to compute robot motion for simple environments and

constraints. Most of these methods are limited to navigationapplications [28], [15], [10] or simple settings [11], or theyare not evaluated on real robots [14]. Nyga et al. [6], [7],[8], [9] use probabilistic relation models based on knowledgebases to understand natural language commands that describevisual attributes of objects. This is complimentary to ourwork. Broad et al. [29] extend DCG for a robot manipulatorso that it will handle natural language correction for robotmotion in realtime. In our approach, the goal is to generateappropriate high-DOF motion trajectories in response toattribute-based natural language instructions like negation,distance or orientation constraints, etc. Unlike prior methods,the output of our NLP parsing algorithm is directly coupledwith the specification of the motion planning problem as aconstrained optimization method.

It may be possible to extend prior methods [1], [2] tohandle attribute-based NLP instructions. For example, dis-tance attributes require a number of constraints in the motionplanning formulation. In natural language instructions suchas “Pick up the blue block and put it 20cm to the leftof the red block” or “Pick up one of the two blocks onthe rightmost, and place it 10 inches away from the blockon the leftmost,” the exact distance specifications are thedistance attributes. Prior methods that use G3, DCG, or theHybrid G3-DCG models have only been evaluated with asmall number of attributes (distance, orientation and contact)to solve constrained motion planning problems. These priortechniques use discretized constraints [2], each of which canbe active (i.e. f (x) > 0)), inverted ( f (x) < 0), or ignored(i.e. not included). Therefore, it is not possible to representan explicit constraint corresponding to the value of thecontinuous variable distance in their formulation.

VII. LIMITATIONS, CONCLUSIONS AND FUTURE WORK

We present an motion planning algorithm that computesappropriate motion trajectories for a robot based on complexNLP instructions. Our formulation is based on two novelconcepts: dynamic grounding graphs and dynamic constraintmapping. We highlight the performance in simulated andreal-world scenes with a 7-DOF manipulator operating nextto humans. We use a high dimensional optimization algo-rithm and the solver may get stuck in local minima, thoughwe use multiple initializations to solve this problem.

As future work, we would like to overcome these limita-tions and evaluate the approach in challenging scenarios withmoving obstacles while performing complex robot tasks.More work is needed to handle the full diversity of a naturallanguage, especially for rare words, complicated grammarstyles, and hidden intentions or emotions in human speech.We plan to incorporate stronger natural language processingand machine learning methods such as those based onsemantic parsing, neural sequence-to-sequence models, etc.

ACKNOWLEDGMENT

This research is supported in part by ARO grantW911NF19-1-0069, ARO-YIP Award W911NF-18-1-0336,and Intel.

REFERENCES

[1] T. Kollar, S. Tellex, M. R. Walter, A. Huang, A. Bachrach,S. Hemachandra, E. Brunskill, A. Banerjee, D. Roy, S. Teller, et al.,“Generalized grounding graphs: A probabilistic framework for under-standing grounded language,” JAIR, 2013.

[2] T. M. Howard, S. Tellex, and N. Roy, “A natural language planner in-terface for mobile manipulators,” in Robotics and Automation (ICRA),2014 IEEE International Conference on. IEEE, 2014, pp. 6652–6659.

[3] S. R. Branavan, H. Chen, L. S. Zettlemoyer, and R. Barzilay, “Re-inforcement learning for mapping instructions to actions,” in ACL-IJCNLP. Association for Computational Linguistics, 2009.

[4] C. Matuszek, D. Fox, and K. Koscher, “Following directions usingstatistical machine translation,” in HRI. IEEE, 2010.

[5] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility andpredictability of robot motion,” in HRI. IEEE, 2013.

[6] D. Nyga and M. Beetz, “Everything robots always wanted to knowabout housework (but were afraid to ask),” in 2012 IEEE/RSJ Inter-national Conference on Intelligent Robots and Systems. IEEE, 2012,pp. 243–250.

[7] ——, “Cloud-based probabilistic knowledge services for instructioninterpretation,” in Robotics Research. Springer, 2018, pp. 649–664.

[8] D. Nyga, M. Picklum, S. Koralewski, and M. Beetz, “Instructioncompletion through instance-based learning and semantic analogicalreasoning,” in 2017 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 2017, pp. 4270–4277.

[9] D. Nyga, M. Picklum, and M. Beetz, “What no robot has seenbeforeprobabilistic interpretation of natural-language object descrip-tions,” in 2017 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 2017, pp. 4278–4285.

[10] F. Duvallet, M. R. Walter, T. Howard, S. Hemachandra, J. Oh, S. Teller,N. Roy, and A. Stentz, “Inferring maps and behaviors from naturallanguage instructions,” in Experimental Robotics. Springer, 2016,pp. 373–388.

[11] S. Branavan, N. Kushman, T. Lei, and R. Barzilay, “Learning high-level planning from text,” in Proceedings of the 50th Annual Meetingof the Association for Computational Linguistics: Long Papers-Volume1. Association for Computational Linguistics, 2012, pp. 126–135.

[12] F. Duvallet, T. Kollar, and A. Stentz, “Imitation learning for naturallanguage direction following through unknown environments,” inICRA. IEEE, 2013, pp. 1047–1053.

[13] R. Paul, J. Arkin, N. Roy, and T. M. Howard, “Efficient grounding ofabstract spatial concepts for natural language interaction with robotmanipulators.” in Robotics: Science and Systems, 2016.

[14] J. Arkin and T. M. Howard, “Towards learning efficient models fornatural language understanding of quantifiable spatial relationships,”in RSS 2015 Workshop on Model Learning for Human-Robot Com-munication, 2015.

[15] I. Chung, O. Propp, M. R. Walter, and T. M. Howard, “On theperformance of hierarchical distributed correspondence graphs forefficient symbol grounding of robot instructions,” in Intelligent Robotsand Systems (IROS), 2015 IEEE/RSJ International Conference on.IEEE, 2015, pp. 5247–5252.

[16] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach tocollision avoidance,” IEEE Robotics & Automation Magazine, vol. 4,no. 1, pp. 23–33, 1997.

[17] M. Kalakrishnan, S. Chitta, E. Theodorou, P. Pastor, and S. Schaal,“STOMP: Stochastic trajectory optimization for motion planning,”in Proceedings of IEEE International Conference on Robotics andAutomation, 2011, pp. 4569–4574.

[18] M. Zucker, N. Ratliff, A. D. Dragan, M. Pivtoraiko, M. Klingensmith,C. M. Dellin, J. A. Bagnell, and S. S. Srinivasa, “Chomp: Covarianthamiltonian optimization for motion planning,” The InternationalJournal of Robotics Research, vol. 32, no. 9-10, pp. 1164–1193, 2013.

[19] C. Park, J. Pan, and D. Manocha, “Real-time optimization-basedplanning in dynamic environments using GPUs,” in Proceedings ofIEEE International Conference on Robotics and Automation, 2013.

[20] D. Silver, J. A. Bagnell, and A. Stentz, “Learning autonomous drivingstyles and maneuvers from expert demonstration,” in ExperimentalRobotics. Springer, 2013, pp. 371–386.

[21] S. Nikolaidis, P. Lasota, G. Rossano, C. Martinez, T. Fuhlbrigge, andJ. Shah, “Human-robot collaboration in manufacturing: Quantitativeevaluation of predictable, convergent joint action,” in Robotics (ISR),2013 44th International Symposium on. IEEE, 2013, pp. 1–6.

[22] H. S. Koppula, A. Jain, and A. Saxena, “Anticipatory planning for

human-robot teams,” in Experimental Robotics. Springer, 2016, pp.453–470.

[23] S. Bird, “Nltk: the natural language toolkit,” in Proceedings of theCOLING/ACL on Interactive presentation sessions. Association forComputational Linguistics, 2006, pp. 69–72.

[24] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectorsfor word representation,” in EMNLP, 2014, pp. 1532–1543.

[25] C. Sutton, A. McCallum, et al., “An introduction to conditional randomfields,” Foundations and Trends R© in Machine Learning, vol. 4, no. 4,pp. 267–373, 2012.

[26] J. S. Park, B. Jia, M. Bansal, and D. Manocha, “Generatingrealtime motion plans from complex natural language commandsusing dynamic grounding graphs,” CoRR, vol. abs/1707.02387, 2017.[Online]. Available: http://arxiv.org/abs/1707.02387

[27] C. Park, J. Pan, and D. Manocha, “ITOMP: Incremental trajectoryoptimization for real-time replanning in dynamic environments,” inProceedings of International Conference on Automated Planning andScheduling, 2012.

[28] J. Oh, T. M. Howard, M. R. Walter, D. Barber, M. Zhu, S. Park,A. Suppe, L. Navarro-Serment, F. Duvallet, A. Boularias, et al.,“Integrated intelligence for human-robot teams,” in International Sym-posium on Experimental Robotics. Springer, 2016, pp. 309–322.

[29] A. Broad, J. Arkin, N. Ratliff, T. Howard, and B. Argall, “Real-timenatural language corrections for assistive robotic manipulators,” TheInternational Journal of Robotics Research, vol. 36, no. 5-7, pp. 684–698, 2017.