Densely Connected Graph Convolutional Networks forGraph-to-Sequence Learning

Zhijiang Guo1∗ , Yan Zhang1∗, Zhiyang Teng1,2, Wei Lu1

1Singapore University of Technology and Design8 Somapah Road, Singapore, 487372

2School of Engineering, Westlake University, China{zhijiang_guo,yan_zhang,zhiyang_teng}@mymail.sutd.edu.sg

tengzhiyang@westlake.edu.cn, luwei@sutd.edu.sg

Abstract

We focus on graph-to-sequence learning,which can be framed as transducing graphstructures to sequences for text generation.To capture structural information associatedwith graphs, we investigate the problemof encoding graphs using graph convolu-tional networks (GCNs).Unlike various ex-isting approaches where shallow architec-tures were used for capturing local structuralinformation only, we introduce a dense con-nection strategy, proposing a novel DenselyConnected Graph Convolutional Networks(DCGCNs). Such a deep architecture is ableto integrate both local and non-local featuresto learn a better structural representation of agraph. Our model outperforms the state-of-the-art neural models significantly on AMR-to-text generation and syntax-based neuralmachine translation.

1 Introduction

Graphs play an important role in natural languageprocessing (NLP) as they are able to capture richerstructural information than sequences and trees.Generally, semantics of sentences can be encodedas graphs. For example, the Abstract MeaningRepresentation (AMR) (Banarescu et al., 2013) isa directed, labeled graph as shown in Figure 1,where nodes in the graph denote semantic con-cepts and edges denote relations between con-cepts. Such graph representations can capture richsemantic-level structural information, and are at-tractive representations useful for semantics re-lated tasks such as semantic parsing (Guo andLu, 2018) and natural language generation (Becket al., 2018). In this paper, we focus on the graph-to-sequence learning tasks, where we aim at learn-ing representations for graphs which are useful fortext generation.

Graph convolutional networks (GCNs) (Kipfand Welling, 2017) are variants of convolutional

∗∗ Equally Contributed.

L=2L=1 L=3

t hi ng

ARG1

guy

you

know- 01

mean- 01

i

t hi ng

guy

you

know- 01

mean- 01

i

t hi ng

guy

you

know- 01

mean- 01

i

mod

ARG0

ARG0

ARG0

ARG1ARG0

mod

ARG0

ARG0 ARG0

ARG0

ARG1ARG0

mod

Figure 1: A 3-layer densely connected graph con-volutional network. The example AMR graph herecorresponds to the sentence “You guys know what Imean.” Every layer encodes information about im-mediate neighbors and 3 layers are needed to capturethird-order neighborhood information (nodes that are 3hops away from the current node). Each layer concate-nates all preceding outputs as the input.

neural networks (CNNs) that operate directly ongraphs, where the representation of each node isiteratively updated based on those of its adjacentnodes in the graph through an information prop-agation scheme. For example, the first layer ofGCNs can only capture the graph’s adjacency in-formation between immediate neighbors, whilewith the second layer one will be able to cap-ture second-order proximity information (neigh-borhood information two hops away from onenode) as shown in Figure 1. Formally, L layerswill be needed in order to capture neighborhoodinformation that is L hops away.

GCNs have been successfully applied to manyNLP tasks (Bastings et al., 2017; Zhang et al.,2018b). Interestingly, while deeper GCNs withmore layers will be able to capture richer neigh-borhood information of a graph, empirically ithas been observed that the best performance isachieved with a 2-layer model (Li et al., 2018).Therefore, recent efforts that leverage recurrence-based graph neural networks have been exploredas the alternatives to encode the structural infor-mation of graphs. Examples include graph-state

long short-term memory (LSTM) networks (Songet al., 2018) and Gated Graph Neural Networks(GGNNs) (Beck et al., 2018). Deep architecturesbased on such recurrence-based models have beensuccessfully built for tasks such as language gen-eration, where rich neighborhood information cap-tured was shown useful.

Compared to recurrent neural networks, con-volutional architectures are highly parallelizableand are more amenable to hardware acceleration(Gehring et al., 2017). It is therefore worthwhileto explore the possibility of applying deeper GCNsthat are able to capture more non-local informationassociated with the graph for graph-to-sequencelearning. Prior efforts try to train deep GCNs byincorporating residual connections (Bastings et al.,2017). Xu et al. (2018) show that vanilla resid-ual connections proposed by He et al. (2016) arenot effective for graph neural networks. Theynext attempt to resolve this issue by adding ad-ditional recurrent layers on top of graph convo-lutional layers. However, they are still confinedto relatively shallow GCNs architectures (at most6 layers in their experiments), which may not beable to capture the rich non-local interactions forlarger graphs.

In this paper, to better address the issueof learning deeper GCNs, we introduce denseconnectivity to GCNs and propose the novelDensely Connected Graph Convolutional Net-works (DCGCNs), inspired by DenseNets (Huanget al., 2017) that distill insights from residual con-nections. The dense connectivity strategy is il-lustrated in Figure 1 schematically. Direct con-nections are introduced from any layer to all itspreceding layers. For example, the third layer re-ceives the outputs of the first layer and the sec-ond layer, capturing the first-order, the second-order and the third-order neighborhood informa-tion. With the help of dense connections, we areable to train multi-layer GCN models with a largedepth, allowing rich local and non-local informa-tion to be captured for learning a better graph rep-resentation than those learned from the shallowerGCN models.

Experiments show that our model is able toachieve better performance for graph-to-sequencelearning tasks. For the AMR-to-text generationtask, our model surpasses the current state-of-the-art neural models trained on LDC2015E86and LDC2017T10 by 2 and 4.3 BLEU pointsrespectively. For the syntax-based neural ma-chine translation task, our model is also consis-tently better than others, showing the effective-

ness of the model on a large training set. Ourcode is available at https://github.com/Cartus/DCGCN.1

2 Densely Connected GCNs

In this section, we will present the basic com-ponents used for constructing our Densely Con-nected GCN model.

2.1 GCNs

GCNs are neural networks that operate directlyon graph structures (Kipf and Welling, 2017).Here we mathematically illustrate how multi-layerGCNs work on an undirected graph G = (V, E),where V and E are the set of nodes and edges, re-spectively. The convolution computation for nodev at the l-th layer, which takes the input featurerepresentation h(l−1) as input and outputs the in-duced representation h

(l)v , can be defined as

h(l)v = ρ

( ∑u∈N (v)

W (l)h(l−1)u + b(l)

)(1)

where W (l) is the weight matrix, b(l) is the biasvector, N (v) is the set of one-hop neighbors ofnode v, and ρ is an activation function (e.g., RELU(Nair and Hinton, 2010)). h

(0)v is the initial input

xv, where xv ∈ Rd and d is the input feature di-mension.

GCNs with Residual Connections. Bastingset al. (2017) integrate residual connections (Heet al., 2016) into GCNs to help information propa-gation. Specifically, each node is updated accord-ing to Eqn.(1) first and then the resulting represen-tation is combined with the node’s representationfrom the last iteration:

h(l)v = ρ

( ∑u∈N (v)

W (l)h(l−1)u +b(l)

)+h(l−1)

v (2)

GCNs with Layer Aggregations. Xu et al.(2018) propose layer aggregations for GCNs, inwhich the final representation of each node is com-puted by combining the node’s representationsfrom all GCN layers:

hfinalv = LA(h(l)

v ,h(l−1)v , ....,h(1)

v ) (3)

where theLA function can be concatenation, max-pooling or LSTM-attention operations as definedin (Xu et al., 2018).

1Our implementation is based on MXNET (Chen et al.,2015) and the Sockeye (Felix et al., 2017) toolkit.

m-layer sub-block

n-layer sub-block

Add & Linear Combination

Input

Add & Linear Combination

Figure 2: Each DCGCN block has two sub-blocks.Both of them are densely connected graph convolu-tional layers with different numbers of layers. A lineartransformation is employed between two sub-blocks,followed by a residual connection.

2.2 Dense Connectivity

Dense connectivity is the core component of theproposed DCGCN. With dense connectivity, nodev in the l-th layer not only takes inputs fromh(l−1), but also receives information from all thepreceding layers, as shown in Figure 2. Mathe-matically, we first define g

(l)u as the concatenation

of the initial node representation and the node rep-resentations produced in layers 1, · · · , l − 1:

g(l)u = [xu;h

(1)u ; ...;h(l−1)

u ]. (4)

Such a mechanism allows deeper layers to captureall previous information to alleviate the problemdiscussed in Section 1 in graph neural networks.Similar strategies are also proposed in previousworks (He et al., 2016; Huang et al., 2017).

While dense connectivity allows training deeperneural networks, every intermediate layer is des-ignated to be of very small size, allowing addingonly a small set of features-maps at each layer.The final classifier makes predictions based on allfeature-maps, which is called “collective knowl-edge” (Huang et al., 2017). Such a strategy im-proves the parameter efficiency. In practice, thedimensions of these small hidden layers dhiddenare decided by the number of layers L and theinput feature dimension d. In DCGCN, we usedhidden = d/L.

For example, if we have a 3-layer (L=3)DCGCN model and input dimension is 300 (d =300), the hidden dimension of each layer will bedhidden = d/L = 300/3 = 100. Then weconcatenate the output of each layer to form the

DCGCN  BlockLSTM  Layer

LSTM  Layer

Attention  Mechanism

DCGCN  Block

DCGCN  Block

Linear  Combination

Prediction

Input  Embedding Output Embedding

PositionalEncoding

Graph  Encoder Decoder

Figure 3: The model concatenates node embeddingsand positional embeddings as inputs. The encoder con-tains a stack of N identical blocks. The linear transfor-mation layer combines output of all blocks into hiddenrepresentations. These are fed into an attention mech-anism, generating the context vector. The decoder, a2-layer LSTM (Hochreiter and Schmidhuber, 1997),makes predictions based on hidden representations andthe context vector.

new representation. We have 3 layers so the out-put dimension is 300 (3 × 100). Different fromthe GCN model whose hidden dimension is largerthan or equal to the input dimension, DCGCNmodel shrinks the hidden dimension as the numberof layers increases in order to improve the param-eter efficiency similar to DenseNets (Huang et al.,2017).

Accordingly, we modify the convolution com-putation of each layer as:

h(l)v = ρ

( ∑u∈N (v)

W (l)g(l)u + b(l)

)(5)

The column dimension of the weight matrixincreases by dhidden per layer, i.e., W (l) ∈Rdhidden×d(l) , where d(l) = d+ dhidden × (l − 1).

2.3 Graph Attention

Attention mechanisms have become almost ade facto standard in many sequence-based tasks(Vaswani et al., 2017). In DCGCNs, we also in-corporate the self-attention strategy by implicitlyspecifying different weights to different nodes in aneighborhood similar to graph attention networks(Velickovic et al., 2018).

In order to perform self-attention on nodes,attention coefficients are required. The inputfor the calculation is a set of vectors, g̃(l) =

{g̃(l)1 , g̃

(l)2 , ..., g̃

(l)n }, after node-wise feature trans-

formation g̃(l)u = W (l)g

(l)u . As an initial step, a

shared linear projection parameterized by a weight

matrix,Wa ∈ Rdhidden×dhidden , is applied to nodesin the graph. Attention coefficients can be com-puted as:

α(l)ij =

exp(φ(a>[Wag̃

(l)i ;Wag̃

(l)j ]))∑

k∈Niexp

(φ(a>[Wag̃

(l)i ;Wag̃

(l)k ]))

(6)where a ∈ R2dhidden is a weight vector, φ is theactivation function (here we use LeakyReLU (Gir-shick et al., 2014)). These coefficients are used tocompute a linear combination of the node repre-sentations. Modifying the convolution computa-tion for attention, we arrive at:

h(l)v = ρ

( ∑u∈N (v)

α(l)vuW

(l)g(l)u + b(l)

)(7)

where α(l)vu are normalized attention coefficients

computed by the attention mechanism at l-th layer.Note that, these coefficients will not change the di-mension of the output representations.

3 Graph-to-Sequence Model

In the following we will explain the model archi-tecture of the graph-to-sequence model. We lever-age DCGCNs as the graph encoder, which directlymodels the graph structure without linearization.

3.1 Graph Encoder

The graph encoder is composed of DCGCNblocks, as shown in Figure 3. Within eachDCGCN block, we design two types of multi-layer DCGCNs as two sub-blocks to capture graphstructure at different abstract levels. As Figure 2shows, in each block, the first sub-block has n-layers and the second sub-block has m-layers.This prototype shares the same spirit with theusage of two different-sized filters in DenseNets(Huang et al., 2017).

Linear Combination Layer. In addition todensely connected layers, we include a linear com-bination layer between multi-layer DCGCNs tofilter the representations from different DCGCNslayers, reaching a more expressive representation.This strategy is inspired by ELMo (Peters et al.,2018), which combines the hidden states from dif-ferent LSTM layers. We also employ a residualconnection (He et al., 2016) to incorporate theinitial inputs of multi-layer GCNs into the linearcombination layer, see Figure 3. Formally, the out-put of the linear combination layer is defined as:

hcomb =Wcomb

(hout + xv

)+ bcomb (8)

op2op1

pur pose

and

come- 01 pur pose

op1

st udy- 01 op2

gnode

st udy- 01

come- 01

l ear n- 01

l ear n- 01

and

default reverse self global

Figure 4: An AMR graph (top) and its correspond-ing extended Levi graph (bottom). The extended Levigraph contains an additional global node and four dif-ferent type of edges.

where hout is the output of the densely connectedlayers by concatenating outputs from all previousL layers hout = [h(1); ...;h(L)] and hout ∈ Rd.xv is the input of the DCGCN layer. hout andxv share the same dimension d. Wcomb ∈ Rd×d

is a weight matrix and bcomb is a bias vector forthe linear transformation. Both Wcomb and bcomb

are different according to different DCGCN lay-ers. In addition, another linear combination layeris added to get the final representations as shownin Figure 3.

3.2 Extended Levi Graph

In order to improve the information propagationprocess in graph structures such as AMR graphsand dependency trees, previous researchers enrichthe original input graphs with additional transfor-mations. Marcheggiani and Titov (2017) add re-verse edges as well as self-loop edges for eachnode to the original graph. This strategy is sim-ilar to the bidirectional recurrent neural networks(RNNs) (Elman, 1990) which can enjoy the in-formation propagation from two directions. Becket al. (2018) adapt this approach and addition-ally transform the directed input graphs into Levigraphs (Gross et al., 2013). Basically, edges in theoriginal graphs are turned into additional nodesin Levi graphs. With this approach, we can en-code the original edge labels and node inputs inthe same way. Specifically, Beck et al. (2018) de-fine three types of edge labels on the Levi graph:

Book me the morning flight

ROOT

dobj

iobj

nmod

det

me

book ROOT

i obj

t he dobj

gnode

mor ni ng

f l i ght

det

nmod

default backward forward self global

Figure 5: A dependency tree and its extended Levigraph.

default, reverse and self, which refer to the origi-nal edges, the new virtual edges which are reverseto the original edges and the self-loop edges.

Scarselli et al. (2009) add another node that isconnected to all other nodes. Zhang et al. (2018a)uses a global sentence-level node to assemble andback-distribute information. Motivated by theseworks, we propose extended Levi graph, whichadds a global node in Levi graph. For every nodex in the original Levi graph, there is a new edge(global) from the global node to x. Figure 4 showsan example AMR graph and its corresponding ex-tended Levi graph. The edge type vocabulary forthe extended Levi graph of the AMR graph nowbecomes T = {default, reverse, self, global}.Our motivations are three-folds. First, the globalnode gives each node a global view of the inputgraph, which can make each node more awareof the non-local information. Second, the globalnode can serve as a hub to help node commu-nications, which can facilitate the node informa-tion propagation process. Third, the output vec-tors of the global node in the encoder can be usedas the initial states of the decoder, which are cru-cial for sequence-to-sequence learning tasks. Priorefforts average representations of all nodes as thegraph embedding to initialize the decoder. Instead,we directly use the learned representation of theglobal nodes, which captures the information fromall nodes in the whole graph.

The input to the syntax-based neural machinetranslation task is the dependency tree. Unlike theAMR graph, the sentence contains significant se-quential information. Beck et al. (2018) inject this

information by adding sequential connections toeach token. In our model, we also add forwardand backward sequential connections as illustratedin Figure 5. Therefore, the edge type vocabularyfor the extended Levi graph of the dependency treebecomes T = {default, reverse, self, global, for-ward, backward}.

Positional encodings about the relative or abso-lute position of the tokens have been proved bene-ficial for sequence learning (Gehring et al., 2017).We also include positional encodings by concate-nating them with the learned word embeddings.The positional encodings are indexed by integervalues representing the minimum distance fromthe root node. For example, come-01 in Figure 4is the root node of the AMR graph, so its indexshould be 0, where and is the child node of come-01, its index is 1. Notice that we denote the indexof the global node as -1.

3.3 Direction AggregationDirectionality and edge labels play an importantrole in linguistic structures. Information fromincoming edges, outgoing edges and self edgesshould be treated differently by using separateweight matrices. Moreover, information from in-coming edges that have different labels shouldhave different weight matrices too. Following thismotivation, we incorporate the directionality of anedge directly in its label. For example, node learn-01 in Figure 4 has three incoming edges, theseedges have three different types: default (fromnode op2), self (from node learn-01) and global(from node gnode). For AMR graph we have fourtypes of edges while for dependency trees we havesix as mentioned in Section 3.2. Thus, consider-ing different type of edges, we modify the convo-lution computation as:

v(l)t = ρ

( ∑u∈N (v)

dir(u,v)=t

α(l)vuW

(l)t g(l)

u + b(l)t

)(9)

where dir(u, v) selects the weight matrix and biasterm associated with the edge type t. For example,in the AMR generation task, there are four edgetypes: default, reverse, self and global. Each typecorresponds to a separate weight matrix and a sep-arate bias term.

Now we need to aggregate representationslearned from different types of edges. A simpleway to do this is averaging them to get the finalrepresentations. However, Hamilton et al. (2017)show that using a mean-based function to aggre-gate feature information from different nodes may

Dataset Train Dev TestAMR15 (LDC2015E86) 16,833 1,368 1,371AMR17 (LDC2017T10) 36,521 1,368 1,371

English-Czech 181,112 2,656 2,999English-German 226,822 2,169 2,999

Table 1: The number of sentences in four datasets.

not be satisfactory, since information from differ-ent sources should not be treated equally. Thus weassign different weights to information from dif-ferent types of edges to integrate such information.Specifically, we concatenate the learned represen-tations from all types of edges and perform a lineartransformation, mathematically illustrated as:

f([v(l)1 ; · · · ;v(l)

T ]) =Wf [v(l)1 ; · · · ;v(l)

T ] + bf

(10)where Wf ∈ Rd

′×dhidden is the weight matrix andd′= T × dhidden. T is the size of the edge type

vocabulary and dhidden is the hidden dimension inDCGCN layers as described in Section 2.2. bf ∈Rdhidden is a bias vector. Finally, the convolutioncomputation becomes:

h(l)v = ρ

(f([v

(l)1 ; · · · ;v(l)

T ]))

(11)

3.4 DecoderWe use an attention-based LSTM decoder (Bah-danau et al., 2015). The initial state of the decoderis the representation of the global node describedin Section 3.2. The decoder yields the natural lan-guage sequence by calculating a sequence of hid-den states sequentially. Here we also include thecoverage mechanism (Tu et al., 2016). Therefore,when generating the t-th token, the decoder con-siders five factors: the attention memory, the wordembedding of the (t − 1)-th token, the previoushidden state of LSTM, the previous context vectorand the previous coverage vector.

4 Experiments

4.1 Experimental SetupWe assess the effectiveness of our models ontwo typical graph-to-sequence learning tasks, in-cluding AMR-to-text generation and syntax-basedneural machine translation (NMT). For the AMR-to-text generation task, we use two benchmarks— the LDC2015E86 dataset (AMR15) and theLDC2017T10 dataset (AMR17). In these datasets,each instance contains a sentence and an AMRgraph. We follow Konstas et al. (2017) to ap-ply entity simplification in the preprocessing steps.We then transform each preprocessed AMR graph

into its extended Levi graph as described in Sec-tion 3.2. For the syntax-based NMT task, we eval-uate our model on both the En-De and the En-CsNews Commentary v11 dataset from the WMT16translation task2. We parse English sentences af-ter tokenization to generate the dependency treeson the source side using SyntaxNet (Alberti et al.,2017)3. We tokenzie Czech and German using theMoses tokenizer4. On the target side, we use byte-pair encodings (BPE) (Sennrich et al., 2016) with8,000 merge operations to obtain subwords. Wetransform the labelled dependency trees into theircorresponding extended Levi graphs as describedin Section 3.2. Table 1 shows the statistics of thesefour datasets. The AMR-to-text datasets containabout 16K ∼ 36K training instances. The NMTdatasets are relatively large, consisting of around200K training instances.

We tune model hyper-parameters using randomlayouts based on the results of the developmentset. We choose the number of DCGCN blocks(Block) from {1, 2, 3, 4}. We select the featuredimension d from {180, 240, 300, 360, 420}. Wedo not use pretrained embeddings. The encoderand the decoder share the training vocabulary. Weadopt Adam (Kingma and Ba, 2015) with an initiallearning rate 0.0003 as the optimizer. The batchsize (Batch) candidates are {16, 20, 24}. We de-termine when to stop training based on the per-plexity change in the development set. For de-coding, we use beam search with beam size 10.Through preliminary experiments, we find that thecombinations (Block=4, d=360, Batch=16) and(Block=2, d=360, Batch=24) give best results onAMR and NMT tasks, respectively. Followingprevious works, we evaluate the results in termsof both BLEU (B) scores (Papineni et al., 2002)and sentence-level CHRF++ (C) scores (Popovic,2017; Beck et al., 2018). Particularly, we use caseinsensitive BLEU scores for AMR and case sensi-tive BLEU scores for NMT. For ensemble models,we train five models with different random seedsand then use Sockeye (Felix et al., 2017) to per-form default ensemble decoding.

4.2 Main Results on AMR-to-text GenerationWe compare the performance of DCGCNs withthe other three kinds of models: (1) sequence-to-sequence (Seq2Seq) models which use linearizedgraphs as inputs; (2) recurrent graph encoders

2http://www.statmt.org/wmt16/translation-task.html

3https://github.com/tensorflow/models/tree/master/research/syntaxnet

4https://github.com/moses-smt/mosesdecoder

Model T #P B CSeq2SeqB (Beck et al., 2018) S 28,4M 21.7 49.1GGNN2Seq (Beck et al., 2018) S 28.3M 23.3 50.4Seq2SeqB (Beck et al., 2018) E 142M 26.6 52.5GGNN2Seq (Beck et al., 2018) E 141M 27.5 53.5

DCGCN (ours) S 18.5M 27.6 57.3E 92.5M 30.4 59.6

Table 2: Main results on AMR17. #P shows the modelsize in terms of parameters; “S” and “E” denote singleand ensemble models, respectively.

(GGNN2Seq, GraphLSTM); (3) models trainedwith external resources. For convenience, we de-note the LSTM-based Seq2Seq models of Konstaset al. (2017) and Beck et al. (2018) as Seq2SeqKand Seq2SeqB, respectively. GGNN2Seq (Becket al., 2018) is the model that leverages GGNNs asgraph encoders.

Table 2 shows the results on AMR17. Our sin-gle model achieves 27.6 BLEU points, which isthe new state-of-the-art result for single models. Inparticular, our single DCGCN model consistentlyoutperforms Seq2Seq models by a significant mar-gin when trained without external resources. Forexample, the single DCGCN model gains 5.9 moreBLEU points than the single models of Seq2SeqBon AMR17. These results demonstrate the impor-tance of explicitly capturing the graph structure inthe encoder.

In addition, our single DCGCN model obtainsbetter results than previous ensemble models. Forexample, on AMR17, the single DCGCN modelis 1 BLEU point higher than the ensemble modelof Seq2SeqB. Our model requires substantiallyfewer parameters, e.g., the parameter size is only3/5 and 1/9 of those in GGNN2Seq and Seq2SeqB,respectively. The ensemble approach based oncombining five DCGCN models initialized withdifferent random seeds achieves a BLEU score of30.4 and a CHRF++ score of 59.6.

Under the same setting, our model also consis-tently outperforms graph encoders based on re-current neural networks or gating mechanisms.For GGNN2Seq, our single model is 3.3 and 0.1BLEU points higher than their single and ensem-ble models, respectively. We also have simi-lar observations in term of CHRF++ scores forsentence-level evaluations. DCGCN also outper-forms GraphLSTM by 2.0 BLEU points in thefully supervised setting as shown in Table 3. Notethat GraphLSTM uses char-level neural represen-tations and pretrained word embeddings, whileour model solely relies on word-level represen-tations with random initializations. This empiri-

Model External BSeq2SeqK (Konstas et al., 2017) - 22.0GraphLSTM (Song et al., 2018) - 23.3DCGCN(single) - 25.7DCGCN(ensemble) - 28.2TSP (Song et al., 2016) ALL 22.4PBMT (Pourdamghani et al., 2016) ALL 26.9Tree2Str (Flanigan et al., 2016) ALL 23.0SNRG (Song et al., 2017) ALL 25.6Seq2SeqK (Konstas et al., 2017) 0.2M 27.4GraphLSTM (Song et al., 2018) 0.2M 28.2DCGCN(single) 0.1M 29.0DCGCN(single) 0.2M 31.6Seq2SeqK (Konstas et al., 2017) 2M 32.3GraphLSTM (Song et al., 2018) 2M 33.6Seq2SeqK (Konstas et al., 2017) 20M 33.8DCGCN(single) 0.3M 33.2DCGCN(ensemble) 0.3M 35.3

Table 3: Main results on AMR15 with/without externalGigaword sentences as auto-parsed data are used.

cally shows that compared to recurrent graph en-coders, DCGCNs can learn better representationsfor graphs.

Moreover, we compare our results with thestate-of-the-art semi-supervised models on theAMR15 test set (Table 3), including non-neuralmethods such as TSP (Song et al., 2016), PBMT(Pourdamghani et al., 2016), Tree2Str (Flaniganet al., 2016) and SNRG (Song et al., 2017). Allthese non-neural models train language modelson the whole Gigaword corpus. Our ensemblemodel gives 28.2 BLEU points without externaldata, which is better than them.

Following Konstas et al. (2017); Song et al.(2018), we also evaluate our model using externalGigaword sentences as training data. We first usethe additional data to pretrain the model, then fine-tune it on the gold data. Using additional 0.1Mdata, the single DCGCN model achieves a BLEUscore of 29.0, which is higher than Seq2SeqK(Konstas et al., 2017) and GraphLSTM (Songet al., 2018) trained with 0.2M additional data.When using the same amount of 0.2M data, theperformance of DCGCN is 4.2 and 3.4 BLEUpoints higher than Seq2SeqK and GraphLSTM.DCGCN model is able to achieve a competitiveBLEU points (33.2) by using 0.3M external data,while GraphLSTM achieves a score of 33.6 by us-ing 2M data and Seq2SeqK achieves a score of33.8 by using 20M data. These results show thatour model is more effective in terms of using au-tomatically generated AMR graphs. Using 0.3Madditional data, our ensemble model achieves thenew state-of-the-art result of 35.3 BLEU points.

Model Type English-German English-Czech#P B C #P B C

BoW+GCN (Bastings et al., 2017) Single - 12.2 - - 7.5 -CNN+GCN (Bastings et al., 2017) Single - 13.7 - - 8.7 -BiRNN+GCN (Bastings et al., 2017) Single - 16.1 - - 9.6 -PB-SMT (Beck et al., 2018) Single - 12.8 43.2 - 8.6 36.4Seq2SeqB (Beck et al., 2018) Single 41.4M 15.5 40.8 39.1M 8.9 33.8GGNN2Seq (Beck et al., 2018) Single 41.2M 16.7 42.4 38.8M 9.8 33.3DCGCN (ours) Single 29.7M 19.0 44.1 28.3M 12.1 37.1Seq2SeqB (Beck et al., 2018) Ensemble 207M 19.0 44.1 195M 11.3 36.4GGNN2Seq (Beck et al., 2018) Ensemble 206M 19.6 45.1 194M 11.7 35.9DCGCN (ours) Ensemble 149M 20.5 45.8 142M 13.1 37.8

Table 4: Main results on English-German and English-Czech datasets.

4.3 Main Results on Syntax-based NMT

Table 4 shows the results for the English-German (En-De) and English-Czech (En-Cs)translation tasks. BoW+GCN, CNN+GCN andBiRNN+GCN refer to employing the followingencoders with a GCN layer on top respectively: 1)a bag-of-words encoder, 2) a one-layer CNN, 3)a bidirectional RNN. PB-SMT is the phrase-basedstatistical machine translation model using Moses(Koehn et al., 2007). Our single model achieves19.0 and 12.1 BLEU points on the En-De and En-Cs tasks, respectively, significantly outperformingall the single models. For example, compared tothe best GCN-based model (BiRNN+GCN), oursingle DCGCN model surpasses it by 2.7 and 2.5BLEU points on the En-De and En-Cs tasks, re-spectively. Our models consist of full GCN lay-ers, removing the burden of employing a recurrentencoder to extract non-local contextual informa-tion in the bottom layers. Compared to non-GCNmodels, our single DCGCN model is 2.2 and 1.9BLEU points higher than the current state-of-the-art single model (GGNN2Seq) on the En-De andEn-Cs translation tasks, respectively. In addition,our single model is comparable to the ensembleresults of Seq2SeqB and GGNN2Seq, while thenumber of parameters of our models is only about1/6 of theirs. Additionally, the ensemble DCGCNmodels achieve 20.5 and 13.1 BLEU points on theEn-De and En-Cs tasks, respectively. Our ensem-ble results are significantly higher than those ofthe state-of-the-art syntax-based ensemble modelsreported by GGNN2Seq (En-De: 20.5 v.s. 19.6;En-Cs: 13.1 v.s. 11.7 in terms of BLEU).

4.4 Additional Experiments

Layers in the sub-block. Table 5 shows the ef-fect of the number of layers of each sub-block onthe AMR15 development set. DenseNets (Huanget al., 2017) use two kinds of convolution filters:

Block n m B C

1

1 1 17.6 48.31 2 19.2 50.32 1 18.4 49.11 3 19.6 49.43 1 20.0 50.53 3 21.4 51.03 6 21.8 51.76 3 21.7 51.56 6 22.0 52.1

23 6 23.5 53.36 3 23.3 53.46 6 22.0 52.1

Table 5: The effect of the number of layers insideDCGCN sub-blocks on the AMR15 development set.

1× 1 and 3× 3. Similar to DenseNets, we choosethe values of n and m for layers from [1, 2, 3, 6].We choose this value range by considering thescale of non-local nodes, the abstract informationat different level and the calculation efficiency.For brevity, we only show representative config-urations. We first investigate DCGCN with oneblock. In general, the performance increases whenwe gradually enlarge n andm. For example, whenn=1 and m=1, the BLEU score is 17.6; when n=6and m=6, the BLEU score becomes 22.0. We ob-serve that the three settings (n=6, m=3), (n=3,m=6) and (n=6,m=6) give similar results for both1 DCGCN block and 2 DCGCN blocks. Since thefirst two settings contain less parameters than thethird setting, it is reasonable to choose either (n=6,m=3) or (n=3, m=6). For later experiments, weuse (n=6, m=3).

Comparisons with Baselines. The first block inTable 6 shows the performance of our two base-line models: multi-layer GCNs with residual con-nections (GCN+RC) and multi-layer GCNs withboth residual connections and layer aggregations(GCN+RC+LA). In general, increasing the num-ber of GCN layers from 2 to 9 boosts the model

GCN B C GCN B C+RC (2) 16.8 48.1 +RC+LA (2) 18.3 47.9+RC (4) 18.4 49.6 +RC+LA (4) 18.0 51.1+RC (6) 19.9 49.7 +RC+LA (6) 21.3 50.8+RC (9) 21.1 50.5 +RC+LA (9) 22.0 52.6+RC (10) 20.7 50.7 +RC+LA (10) 21.2 52.9DCGCN1 (9) 22.9 53.0 DCGCN3 (27) 24.8 54.7DCGCN2 (18) 24.2 54.4 DCGCN4 (36) 25.5 55.4

Table 6: Comparisons with baselines. +RC denotesGCNs with residual connections. +RC+LA refers toGCNs with both residual connections and layer aggre-gations. DCGCNi represents our model with i blocks,containing i × (n + m) layers. The number of layersfor each model is shown in parenthesis.

performance. However, when the layer num-ber exceeds 10, the performance of both baselinemodels start to drop. For example, GCN+RC+LA(10) achieves a BLEU score of 21.2, which isworse than GCN+RC+LA (9). In preliminary ex-periments, we cannot manage to train very deepGCN+RC and GCN+RC+LA models. In con-trast, our DCGCN models can be trained us-ing a large number of layers. For example,DCGCN4 contains 36 layers. When we increasethe DCGCN blocks from 1 to 4, the model per-formance continues increasing on AMR15 devel-opment set. We therefore choose DCGCN4 forthe AMR experiments. Using a similar method,DCGCN2 is selected for the NMT tasks. Whenthe layer numbers are 9, DCGCN1 is better thanGCN+RC in term of B/C scores (21.7/51.5 v.s.21.1/50.5). GCN+RC+LA (9) is sightly betterthan DCGCN1. However, when we set the numberto 18, GCN+RC+LA achieves a BLEU score of19.4, which is significantly worse than the BLEUscore obtained by DCGCN2 (23.3). We also tryGCN+RC+LA (27), but it does not converge. Inconclusion, these results above can show the ro-bustness and effectiveness of our DCGCN models.

Performance v.s. Parameter Budget. Wealso evaluate the performance of DCGCN modelagainst different number of parameters on theAMR generation task. Results are shown in Fig-ure 6. Specifically, we try four parameter bud-gets, including 11.8M, 14.0M, 16.2M and 18.4M.These numbers correspond to the model size (interms of number of parameters) of DCGCN1,DCGCN2, DCGCN3 and DCGCN4, respectively.For each budget, we vary both the depth of GCNmodels and the hidden vector dimensions of eachnode in GCNs in order to exhaust the entire bud-get. For example,GCN(2)−512,GCN(3)−426,GCN(4)−372 andGCN(5)−336 contain about

Model D #P B CDCGCN(1) 300 10.9M 20.9 52.0DCGCN(2) 180 22.2 52.3DCGCN(2) 240 11.3M 22.8 52.8DCGCN(4) 180 11.4M 23.4 53.4DCGCN(1) 420 12.6M 22.2 52.4DCGCN(2) 300 12.5M 23.8 53.8DCGCN(3) 240 12.3M 23.9 54.1DCGCN(2) 360 14.0M 24.2 54.4DCGCN(3) 300 24.4 54.2DCGCN(2) 420 15.6M 24.1 53.7DCGCN(4) 300 24.6 54.8DCGCN(3) 420 18.6M 24.5 54.6DCGCN(4) 360 18.4M 25.5 55.4

Table 7: Comparisons of different DCGCN models un-der almost the same parameter budget.

11.8M parameters, where GCN(i) − d indicatesa GCN model with i layers and the hidden size foreach node is d. We compare DCGCN1 with thesefour models. DCGCN1 gives 22.9 BLEU points.For the GCN models, the best result is obtainedby GCN(5) − 336, which falls behind DCGCN1by 2.0 BLEU points. We compare DCGCN2,DCGCN3 and DCGCN4 with their equal-sizedGCN models in a similar way. The results showthat DCGCN consistently outperforms GCN un-der the same parameter budget. When the param-eter budget becomes larger, we can observe thatthe performance difference becomes more promi-nent. In particular, the BLEU margins betweenDCGCN models and their best GCN models are2.0, 2.7, 2.7 and 3.4, respectively.

Performance v.s. Layers. We compareDCGCN models with different layers under thesame parameter budget. Table 7 shows the results.For example, when both DCGCN1 and DCGCN2are limited to 10.9M parameters, DCGCN2obtains 22.2 BLEU points, which is higher thanDCGCN1 (20.9). Similarly, when DCGCN3 andDCGCN4 contain 18.6M and 18.4M parameters.DCGCN4 outperforms DCGCN3 by 1 BLEUpoint with a slightly smaller model. In general,we found when the parameter budget is the same,deeper DCGCN models can obtain better resultsthan the shallower ones.

Level of Density. Table 8 shows the ablationstudy of the level of density of our model. Weuse DCGCNs with 4 dense blocks as the fullmodel. Then we remove dense connections grad-ually from the last block to the first block. In gen-eral, the performance of the model drops substan-tially as we remove more dense connections untilit cannot converge without dense connections. The

11.8 14.0 16.2 18.4number of parameters (M)

18

20

22

24

26B

LEU

2-512 3-4264-372

5-3364-512

6-4268-372

10-336 6-512

9-42612-372

15-336

8-51212-426

16-37220-336

1-360

2-3603-360

4-360

GCNDCGCN

Figure 6: Comparison of DCGCN and GCN over different number of parameters. a-b means the model has alayers (a blocks for DCGCN) and the hidden size is b (e.g., 5-336 means a 5-layers GCN with the hidden size336).

Model B CDCGCN4 25.5 55.4-{4} dense block 24.8 54.9-{3, 4} dense blocks 23.8 54.1-{2, 3, 4} dense blocks 23.2 53.1

Table 8: Ablation study for density of connections onthe dev set of AMR15. -{i} dense block denotes re-moving the dense connections in the i-th block.

Model B CDCGCN4 25.5 55.4Encoder Modules-Linear Combination 23.7 53.2-Global Node 24.2 54.6-Direction Aggregation 24.6 54.6-Graph Attention 24.9 54.7-Global Node&Linear Combination 22.9 52.4Decoder Modules-Coverage Mechanism 23.8 53.0

Table 9: Ablation study for modules used in the graphencoder and the LSTM decoder

full model gives 25.5 BLEU points on the AMR15dev set. After removing the dense connectionsin the last block, the BLEU score becomes 24.8.Without using the dense connections in the lasttwo blocks, the score drops to 23.8. Furthermore,excluding the dense connections in the last threeblocks only gives 23.2 BLEU points. Althoughthese four models have the same number of lay-ers, dense connections allow the model to achievemuch better performance. If all the dense connec-tions are not considered, the model does not cov-erage at all. These results indicate dense connec-tions do play a significant role in our model.

Ablation Study for Encoder and Decoder. Fol-lowing Song et al. (2018), we conduct a furtherablation study for modules used in the graph en-coder and LSTM decoder on the AMR15 dev set,

including linear combination, global node, direc-tion aggregation, graph attention mechanism andcoverage mechanism using the 4-block models byalways keeping the dense connections.

Table 9 shows the results. For the encoder, wefind that the linear combination and the globalnode have more contributions in terms of B/Cscores. The results drop by 2/2.2 and 1.3/1.2points respectively after removing them. With-out these two components, our model gives aBLEU score of 22.6, which is still better thanthe best GCN+RC model (21.1) and the bestGCN+RC+LA model (22.1). Adding either theglobal node or the linear combination improvesthe baseline models with only dense connections.This suggests that enriching input graphs with theglobal node and including the linear combinationcan facilitate GCNs to learn better information ag-gregations, producing more expressive graph rep-resentations. Results also show the linear combi-nation is more effective than the global node. Con-sidering them together further enhances the modelperformance. After removing the graph attentionmodule, our model gives 24.9 BLEU points. Sim-ilarly, excluding the direction aggregation moduleleads to a performance drop to 24.6 BLEU points.The coverage mechanism is also effective in ourmodels. Without the coverage mechanism, the re-sult drops by 1.7/2.4 points for B/C scores.

4.5 Analysis and Discussion

Graph size. Following Bastings et al. (2017),we show in Figure 7 the CHRF++ score vari-ations according to the graph size |G| on theAMR2015 development set, where |G| refersto the number of nodes in the extended Levigraph. We bin the graph size into five classes(≤ 30, (30, 40], (40, 50], (50, 60], > 60). We av-erage the sentence-level CHRF++ scores of the

<=30 30-40 40-50 50-60 >60Graph size

42

46

50

54

58CH

RF++

GCN+RCGCN+RC+LADCGCN

Figure 7: CHRF++ scores with respect to the inputgraph size for three models.

sentences in the same bin to plot Figure 7. Forsmall graphs (i.e., |G| ≤ 30), DCGCN obtainssimilar results as the baselines. For large graphs,DCGCN significantly outperforms the two base-lines. In general, as the graph size increases, thegap between DCGCN and the two baselines be-comes larger. In addition, we can also noticethat the margin between GCN and GCN+LA isquite stable, while the margin between DCGCNand GCN+LA varies according to the graph size.The trend for BLEU scores is similar to CHRF++scores. This suggests that DCGCN can performbetter for larger graphs as its deeper architec-ture can capture the long-distance dependencies.Dense connections facilitate information propaga-tion in large graphs, while shallow GCNs mightstruggle to capture such dependencies.

Example output. Table 10 shows example out-puts from three models for the AMR-to-text task,together with the corresponding AMR graph aswell as the text reference. The word “technology”in the reference acts as a link between “globaltrade” and “weapons of mass destruction”, offer-ing the background knowledge to help understandthe context. The word “instructions” also plays acrucial role in the generated sentence – without theword the sentence will have a significantly differ-ent meaning. Both GCN+RC and GCN+RC+LAfail to successfully generate these two importantwords. The output from GCN+RC does not evenappear to be grammatically correct. In contrast,DCGCN manages to generate both words. Webelieve this is because DCGCN is able to learnricher semantic information by capturing complexlong dependencies. GCN+RC+LA does generatean output which looks similar to the reference atthe token level. However, the conveyed semanticinformation in the generated sentence largely dif-fers from that of the reference. DCGCNs do nothave this problem.

(s / state-0100 :ARG0 (p / person0000 :ARG0-of (h / have-org-role-91000000 :ARG1 (i / intelligence00000000 :mod (c / country :wiki "united_states"0000000000 :name (n / name :op1 "u.s.")))000000 :ARG2 (o / official)))00 :ARG1 (c2 / continue-010000 :ARG0 (p2 / person000000 :ARG0-of (h2 / have-org-role-9100000000 :ARG2 (o2 / official0000000000 :mod (c3 / country :wiki "north_korea"000000000000 :name (n2 / name :op1 "north" :op2000000000000 "korea")))))0000 :ARG1 (t / trade-01000000 :ARG1 (t2 / technology00000000 :purpose (w / weapon0000000000 :ARG2-of (d / destroy-01000000000000 :degree (m / mass))))000000 :mod (g / globe))0000 :ARG2-of (i2 / include-01000000 :ARG1 (i3 / instruct-0100000000 :ARG3 (m2 / make-0100000000000 :ARG1 (m3 / missile0000000000000 :ARG1-of (a / advanced-02)))))))Reference: u.s. intelligence officials stated that north koreanofficials are continuing global trade in technology for weaponsof mass destruction including instructions for making advancedmissiles.GCN+RC: a u.s. intelligence official stated that north koreaofficials continued the global trade for weapons of massdestruction by making advanced missiles to make advancedmissiles.GCN+RC+LA: a u.s. intelligence official stated that northkorea officials continued global trade with weapons of massdestruction including making advanced missiles.DCGCN: a u.s. intelligence official stated that north koreaofficials continue global trade on technology for weapons ofmass destruction including instructions to make advancedmissiles.

Table 10: Example outputs.

5 Related Work

Our work builds on a rich line of recent ef-forts on graph-to-sequence models, graph convo-lutional networks and densely connected convolu-tional networks.

Graph-to-sequence learning. Early researchefforts for graph-to-sequence learning are basedon statistical methods. Lu et al. (2009) presenta language generation model using the tree-structured meaning representation based on treeconditional random fields. Lu and Ng (2011) pro-pose a model for language generation from lambdacalculus expressions which can be represented asforest structures. Konstas and Lapata (2012, 2013)leverage hypergraphs for concept-to-text genera-tion. Flanigan et al. (2016) transform a givenAMR graph into a spanning tree, before translat-ing it into a sentence using a tree-to-string trans-ducer. Pourdamghani et al. (2016) adopt a phrase-based model for machine translation (Koehn et al.,2003) based on a linearized AMR graph. Songet al. (2017) leverage a synchronous node replace-ment grammar. Konstas et al. (2017) also linearizethe input graph and feed it to the Seq2Seq model

(Sutskever et al., 2014).Sequence based neural networks may lose

structural information from the original graphsince they require linearization of the input graph.Recent research efforts consider developing en-coders with graph neural networks. Beck et al.(2018) employ GGNNs (Li et al., 2016) as theencoder and introduce the Levi graph that allowsnodes and edges to have their own hidden repre-sentations. Song et al. (2018) propose the graph-state LSTM to directly encode graph-level seman-tics. In order to capture non-local information, theencoder performs graph state transition by infor-mation exchange between connected nodes. Theirwork belongs to the family of recurrent neural net-works (RNN). Our graph encoder is built basedon the graph convolutional networks (GCNs). Re-current graph neural networks (Li et al., 2016;Song et al., 2018) use gated operations to updatenode states while graph convolutional networksuse linear transformation. The contrast betweenour model and theirs is reminiscent of the contrastbetween CNN and RNN.

Closest to our work, Bastings et al. (2017) stackGCNs upon a RNN or CNN encoder since 2-layerGCNs may not be able to capture non-local infor-mation, especially when the graph is large. Ourgraph encoder solely relies on the DCGCN model,whose deep network structure encodes richer lo-cal and non-local information for learning bettergraph representations.

Densely connected convolutional networks.Intuitively, neural networks should be able to learnrich representations by stacking a large number oflayers. However, empirical results often do notsupport such an intuition – useful information cap-tured in earlier layers may get lost after passingthrough subsequent layers. Many recent effortsfocus on resolving such an issue. Highway Net-works (Srivastava et al., 2015) use bypassing pathsalong with gating units to train networks. ResNets(He et al., 2016), in which identity mappings areused as bypassing paths, have achieved impressiveperformance on various tasks. DenseNets (Huanget al., 2017) refine this insight and propose a denseconnectivity strategy, which connects all layers di-rectly with each other to ensure maximum infor-mation flow between layers.

Graph convolutional networks. Early effortsthat attempt to extend neural networks to deal witharbitrary structured graphs are introduced by Goriet al. (2005) and Scarselli et al. (2009), where thestates of nodes are updated based on the states

of their neighbors. Bruna (2014) then appliesthe convolution operation on graph Laplacians toconstruct efficient architectures in the spectral do-main. Subsequent efforts improve its computa-tional efficiency with local spectral convolutiontechniques (Henaff et al., 2015; Defferrard et al.,2016; Kipf and Welling, 2017).

Our approach is closely related to GCNs (Kipfand Welling, 2017), which restrict the filters to op-erate on a first-order neighborhood around eachnode. Recent improvements and extensions ofGCNs include using additional aggregation meth-ods such as vertex attention (Velickovic et al.,2018) or pooling mechanism (Hamilton et al.,2017) to better summarize neighborhood states.

However, the best performance of GCNs isachieved with a 2-layer model while deeper mod-els perform worse though they can potentiallyhave access to more non-local information. Liet al. (2018) shows that this issue is due to theover-smoothed output representations that impededistinguishing nodes from different clusters. Re-cent attempts that try to address this issue includesthe use of layer-aggregation functions (Xu et al.,2018), which combine learned features from alllayers, and the use of co-training and self-trainingmechanisms that encourage exploration on the en-tire graph (Li et al., 2018).

6 Conclusion

We introduce the novel densely connected graphconvolutional networks (DCGCNs) to learn struc-tural graph representations. Experimental resultsshow that DCGCNs can outperform state-of-the-art models in two tasks: AMR-to-text gener-ation and syntax-based neural machine transla-tion. Unlike previous designs of GCNs, DCGCNsscale naturally to significantly more layers with-out suffering from performance degradation andoptimization difficulties, thanks to the introduceddense connectivity mechanism. Such a deep ar-chitecture allows the encoder to better capture therich structural information of a graph, especiallywhen it is large.

There are multiple venues for future work. Onenatural question we would like to ask is how tomake use of the proposed framework to performimproved graph representation learning for vari-ous graph related tasks (Xu et al., 2018). On theother hand, we would also like to investigate howother NLP applications such as relation extraction(Zhang et al., 2018b) and semantic role labeling(Marcheggiani and Titov, 2017) can potentiallybenefit from our proposed approach.

Acknowledgements

We would like to thank the anonymous reviewersand our Action Editor Stefan Riezler for their com-ments and suggestions on this work. We wouldalso like to thank Daniel Beck, Linfeng Song,Joost Bastings, Zuozhu Liu and Yiluan Guo fortheir helpful suggestions. This work is supportedby Singapore Ministry of Education AcademicResearch Fund (AcRF) Tier 2 Project MOE2017-T2-1-156. This work is also partially supported bySUTD project PIE-SGP-AI-2018-01.

References

Chris Alberti, Daniel Andor, Ivan Bogatyy,Michael Collins, Daniel Gillick, LingpengKong, Terry Koo, Ji Ma, Mark Omernick, SlavPetrov, Chayut Thanapirom, Zora Tung, andDavid Weiss. 2017. Syntaxnet models for theconll 2017 shared task. arXiv preprint.

Dzmitry Bahdanau, Kyunghyun Cho, and YoshuaBengio. 2015. Neural machine translation byjointly learning to align and translate. In Proc.of ICLR.

Laura Banarescu, Claire Bonial, Shu Cai,Madalina Georgescu, Kira Griffitt, Ulf Herm-jakob, Kevin Knight, Philipp Koehn, MarthaPalmer, and Nathan Schneider. 2013. Abstractmeaning representation for sembanking. InProc. of LAW@ACL.

Joost Bastings, Ivan Titov, Wilker Aziz, DiegoMarcheggiani, and Khalil Sima’an. 2017.Graph convolutional encoders for syntax-awareneural machine translation. In Proc. of EMNLP.

Daniel Beck, Gholamreza Haffari, and TrevorCohn. 2018. Graph-to-sequence learning usinggated graph neural networks. In Proc. of ACL.

Joan Bruna. 2014. Spectral networks and deep lo-cally connected networks on graphs. In Proc. ofICLR.

Tianqi Chen, Mu Li, Yutian Li, Min Lin, NaiyanWang, Minjie Wang, Tianjun Xiao, Bing Xu,Chiyuan Zhang, and Zheng Zhang. 2015.Mxnet: A flexible and efficient machine learn-ing library for heterogeneous distributed sys-tems. arXiv preprint.

Michaël Defferrard, Xavier Bresson, and PierreVandergheynst. 2016. Convolutional neuralnetworks on graphs with fast localized spectralfiltering. In Proc. of NIPS.

Jeffrey L. Elman. 1990. Finding structure in time.Cognitive science, 14(2):179–211.

Hieber Felix, Domhan Tobias, DenkowskiMichael, Vilar David, Sokolov Artem, CliftonAnn, and Post Matt. 2017. Sockeye: A toolkitfor neural machine translation. arXiv preprint.

Jeffrey Flanigan, Chris Dyer, Noah A. Smith, andJaime G. Carbonell. 2016. Generation from ab-stract meaning representation using tree trans-ducers. In Proc. of NAACL-HLT.

Jonas Gehring, Michael Auli, David Grangier, De-nis Yarats, and Yann Dauphin. 2017. Convolu-tional sequence to sequence learning. In Proc.of ICML.

Ross Girshick, Jeff Donahue, Trevor Darrell, andJitendra Malik. 2014. Rich feature hierarchiesfor accurate object detection and semantic seg-mentation. In Proc. of CVPR.

Michele Gori, Gabriele Monfardini, and FrancoScarselli. 2005. A new model for learning ingraph domains. In Proc. of IJCNN.

Jonathan L. Gross, Jay Yellen, and Ping Zhang.2013. Handbook of Graph Theory, Second Edi-tion. Chapman & Hall/CRC.

Zhijiang Guo and Wei Lu. 2018. Better transition-based amr parsing with a refined search space.In Proc. of EMNLP.

William L. Hamilton, Rex Ying, and JureLeskovec. 2017. Inductive representation learn-ing on large graphs. In Proc. of NIPS.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, andJian Sun. 2016. Deep residual learning for im-age recognition. In Proc. of CVPR.

Mikael Henaff, Joan Bruna, and Yann LeCun.2015. Deep convolutional networks on graph-structured data. arXiv preprint.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.Long short-term memory. Neural computation,9(8):1735–1780.

Gao Huang, Zhuang Liu, Laurens van der Maaten,and Kilian Q. Weinberger. 2017. Densely con-nected convolutional networks. In Proc. ofCVPR.

Diederik P. Kingma and Jimmy Ba. 2015. Adam:A method for stochastic optimization. In Proc.of ICLR.

Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolu-tional networks. In Proc. of ICLR.

Philipp Koehn, Hieu Hoang, Alexandra Birch,Chris Callison-Burch, Marcello Federico,Nicola Bertoldi, Brooke Cowan, Wade Shen,Christine Moran, Richard Zens, Chris Dyer,Ondrej Bojar, Alexandra Constantin, and EvanHerbst. 2007. Moses: Open source toolkitfor statistical machine translation. In Proc. ofACL(Demo).

Philipp Koehn, Franz Josef Och, and DanielMarcu. 2003. Statistical phrase-based transla-tion. In Proc. of NAACL-HLT.

Ioannis Konstas, Srinivasan Iyer, Mark Yatskar,Yejin Choi, and Luke S. Zettlemoyer. 2017.Neural amr: Sequence-to-sequence models forparsing and generation. In Proc. of ACL.

Ioannis Konstas and Mirella Lapata. 2012. Unsu-pervised concept-to-text generation with hyper-graphs. In Proc. of NAACL-HLT.

Ioannis Konstas and Mirella Lapata. 2013. Induc-ing document plans for concept-to-text genera-tion. In Proc. of EMNLP.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018.Deeper insights into graph convolutional net-works for semi-supervised learning. In Proc.of AAAI.

Yujia Li, Daniel Tarlow, Marc Brockschmidt, andRichard S. Zemel. 2016. Gated graph sequenceneural networks. In Proc. of ICLR.

Wei Lu and Hwee Tou Ng. 2011. A probabilis-tic forest-to-string model for language genera-tion from typed lambda calculus expressions. InProc. of EMNLP.

Wei Lu, Hwee Tou Ng, and Wee Sun Lee. 2009.Natural language generation with tree condi-tional random fields. In Proc. of EMNLP.

Diego Marcheggiani and Ivan Titov. 2017. En-coding sentences with graph convolutional net-works for semantic role labeling. In Proc. ofEMNLP.

Vinod Nair and Geoffrey E. Hinton. 2010. Rec-tified linear units improve restricted boltzmannmachines. In Proc. of ICML.

Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for au-tomatic evaluation of machine translation. InProc. of ACL.

Matthew E. Peters, Mark Neumann, Mohit Iyyer,Matt Gardner, Christopher Clark, Kenton Lee,and Luke S. Zettlemoyer. 2018. Deep con-textualized word representations. In Proc. ofNAACL-HLT.

Maja Popovic. 2017. chrf++: words helping char-acter n-grams. In Proc. of WMT@ACL.

Nima Pourdamghani, Kevin Knight, and Ulf Her-mjakob. 2016. Generating english from abstractmeaning representations. In Proc. of INLG.

Franco Scarselli, Marco Gori, Ah Chung Tsoi,Markus Hagenbuchner, and Gabriele Monfar-dini. 2009. The graph neural network model.IEEE Transactions on Neural Networks, 20(1).

Rico Sennrich, Barry Haddow, and AlexandraBirch. 2016. Neural machine translation of rarewords with subword units. In Proc. of ACL.

Linfeng Song, Xiaochang Peng, Yue Zhang,Zhiguo Wang, and Daniel Gildea. 2017. Amr-to-text generation with synchronous node re-placement grammar. In Proc. of ACL.

Linfeng Song, Yue Zhang, Xiaochang Peng,Zhiguo Wang, and Daniel Gildea. 2016. Amr-to-text generation as a traveling salesman prob-lem. In Proc. of EMNLP.

Linfeng Song, Yue Zhang, Zhiguo Wang, andDaniel Gildea. 2018. A graph-to-sequencemodel for amr-to-text generation. In Proc. ofACL.

Rupesh Kumar Srivastava, Klaus Greff, and Jür-gen Schmidhuber. 2015. Training very deepnetworks. In Proc. of NIPS.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.2014. Sequence to sequence learning with neu-ral networks. In Proc. of NIPS.

Zhaopeng Tu, Zhengdong Lu, Yang Liu, XiaohuaLiu, and Hang Li. 2016. Modeling coverage forneural machine translation. In Proc. of ACL.

Ashish Vaswani, Noam Shazeer, Niki Parmar,Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,Lukasz Kaiser, and Illia Polosukhin. 2017. At-tention is all you need. In Proc. of NIPS.

Petar Velickovic, Guillem Cucurull, ArantxaCasanova, Adriana Romero, Pietro Liò, andYoshua Bengio. 2018. Graph attention net-works. In Proc. of ICLR.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomo-hiro Sonobe, Ken ichi Kawarabayashi, and Ste-fanie Jegelka. 2018. Representation learning ongraphs with jumping knowledge networks. InProc. of ICML.

Yue Zhang, Qi Liu, and Linfeng Song. 2018a.Sentence-state lstm for text representation. InProc. of ACL.

Yuhao Zhang, Peng Qi, and Christopher D. Man-ning. 2018b. Graph convolution over pruneddependency trees improves relation extraction.In Proc. of EMNLP.