Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 171–182July 5 - 10, 2020. c©2020 Association for Computational Linguistics

171

Fact-based Text Editing

Hayate Iso†∗ Chao Qiao‡ Hang Li‡†Nara Institute of Science and Technology ‡ByteDance AI Lab

hyate.iso@gmail.com, {qiaochao, lihang.lh}@bytedance.com

Abstract

We propose a novel text editing task, referredto as fact-based text editing, in which the goalis to revise a given document to better de-scribe the facts in a knowledge base (e.g., sev-eral triples). The task is important in practicebecause reflecting the truth is a common re-quirement in text editing. First, we propose amethod for automatically generating a datasetfor research on fact-based text editing, whereeach instance consists of a draft text, a revisedtext, and several facts represented in triples.We apply the method into two public table-to-text datasets, obtaining two new datasetsconsisting of 233k and 37k instances, respec-tively. Next, we propose a new neural networkarchitecture for fact-based text editing, calledFACTEDITOR, which edits a draft text by re-ferring to given facts using a buffer, a stream,and a memory. A straightforward approach toaddress the problem would be to employ anencoder-decoder model. Our experimental re-sults on the two datasets show that FACTE-DITOR outperforms the encoder-decoder ap-proach in terms of fidelity and fluency. Theresults also show that FACTEDITOR conductsinference faster than the encoder-decoder ap-proach.

1 Introduction

Automatic editing of text by computer is an impor-tant application, which can help human writers towrite better documents in terms of accuracy, flu-ency, etc. The task is easier and more practical thanthe automatic generation of texts from scratch andis attracting attention recently (Yang et al., 2017;Yin et al., 2019). In this paper, we consider a newand specific setting of it, referred to as fact-basedtext editing, in which a draft text and several facts(represented in triples) are given, and the system

∗ The work was done when Hayate Iso was a researchintern at ByteDance AI Lab.

Set of triples{(Baymax, creator, Douncan Rouleau),

(Douncan Rouleau, nationality, American),(Baymax, creator, Steven T. Seagle),(Steven T. Seagle, nationality, American),(Baymax, series, Big Hero 6),(Big Hero 6, starring, Scott Adsit)}

Draft textBaymax was created by Duncan Rouleau, a winner ofEagle Award. Baymax is a character in Big Hero 6 .

Revised textBaymax was created by American creatorsDuncan Rouleau and Steven T. Seagle . Baymax isa character in Big Hero 6 which stars Scott Adsit .

Table 1: Example of fact-based text editing. Facts arerepresented in triples. The facts in green appear inboth draft text and triples. The facts in orange arepresent in the draft text, but absent from the triples.The facts in blue do not appear in the draft text, butin the triples. The task of fact-based text editing is toedit the draft text on the basis of the triples, by deletingunsupported facts and inserting missing facts whileretaining supported facts.

aims to revise the text by adding missing facts anddeleting unsupported facts. Table 1 gives an exam-ple of the task.

As far as we know, no previous work did addressthe problem. In a text-to-text generation, given atext, the system automatically creates another text,where the new text can be a text in another language(machine translation), a summary of the originaltext (summarization), or a text in better form (textediting). In a table-to-text generation, given a tablecontaining facts in triples, the system automaticallycomposes a text, which describes the facts. Theformer is a text-to-text problem, and the latter atable-to-text problem. In comparison, fact-basedtext editing can be viewed as a ‘text&table-to-text’problem.

172

First, we devise a method for automatically cre-ating a dataset for fact-based text editing. Recently,several table-to-text datasets have been created andreleased, consisting of pairs of facts and corre-sponding descriptions. We leverage such kind ofdata in our method. We first retrieve facts andtheir descriptions. Next, we take the descriptionsas revised texts and automatically generate drafttexts based on the facts using several rules. Webuild two datasets for fact-based text editing onthe basis of WEBNLG (Gardent et al., 2017) andROTOWIRE, consisting of 233k and 37k instancesrespectively (Wiseman et al., 2017) 1.

Second, we propose a model for fact-based textediting called FACTEDITOR. One could employan encoder-decoder model, such as an encoder-decoder model, to perform the task. The encoder-decoder model implicitly represents the actions fortransforming the draft text into a revised text. Incontrast, FACTEDITOR explicitly represents theactions for text editing, including Keep, Drop,and Gen, which means retention, deletion, andgeneration of word respectively. The model utilizesa buffer for storing the draft text, a stream to storethe revised text, and a memory for storing the facts.It also employs a neural network to control theentire editing process. FACTEDITOR has a lowertime complexity than the encoder-decoder model,and thus it can edit a text more efficiently.

Experimental results show that FACTEDITORoutperforms the baseline model of using encoder-decoder for text editing in terms of fidelity andfluency, and also show that FACTEDITOR can per-form text editing faster than the encoder-decodermodel.

2 Related Work

2.1 Text Editing

Text editing has been studied in different settingssuch as automatic post-editing (Knight and Chan-der, 1994; Simard et al., 2007; Yang et al., 2017),paraphrasing (Dolan and Brockett, 2005), sentencesimplification (Inui et al., 2003; Wubben et al.,2012), grammar error correction (Ng et al., 2014),and text style transfer (Shen et al., 2017; Hu et al.,2017).

The rise of encoder-decoder models (Cho et al.,2014; Sutskever et al., 2014) as well as the atten-tion (Bahdanau et al., 2015; Vaswani et al., 2017)

1The datasets are publicly available at https://github.com/isomap/factedit

and copy mechanisms (Gu et al., 2016; Gulcehreet al., 2016) has dramatically changed the land-scape, and now one can perform the task rela-tively easily with an encoder-decoder model suchas Transformer provided that a sufficient amountof data is available. For example, Li et al. (2018)introduce a deep reinforcement learning frameworkfor paraphrasing, consisting of a generator and anevaluator. Yin et al. (2019) formalize the prob-lem of text edit as learning and utilization of editrepresentations and propose an encoder-decodermodel for the task. Zhao et al. (2018) integrateparaphrasing rules with the Transformer model fortext simplification. Zhao et al. (2019) proposes amethod for English grammar correction using aTransformer and copy mechanism.

Another approach to text editing is to view theproblem as sequential tagging instead of encoder-decoder. In this way, the efficiency of learningand prediction can be significantly enhanced. Vuand Haffari (2018) and Dong et al. (2019) con-duct automatic post-editing and text simplificationon the basis of edit operations and employ Neu-ral Programmer-Interpreter (Reed and De Freitas,2016) to predict the sequence of edits given a se-quence of words, where the edits include KEEP,DROP, and ADD. Malmi et al. (2019) propose a se-quential tagging model that assigns a tag (KEEPor DELETE) to each word in the input sequenceand also decides whether to add a phrase beforethe word. Our proposed approach is also basedon sequential tagging of actions. It is designed forfact-based text editing, not text-to-text generation,however.

2.2 Table-to-Text Generation

Table-to-text generation is the task which aims togenerate a text from structured data (Reiter andDale, 2000; Gatt and Krahmer, 2018), for exam-ple, a text from an infobox about a term in biol-ogy in wikipedia (Lebret et al., 2016) and a de-scription of restaurant from a structured represen-tation (Novikova et al., 2017). Encoder-decodermodels can also be employed in table-to-text gen-eration with structured data as input and gener-ated text as output, for example, as in (Lebretet al., 2016). Puduppully et al. (2019) and Iso et al.(2019) propose utilizing an entity tracking modulefor document-level table-to-text generation.

One issue with table-to-text is that the style ofgenerated texts can be diverse (Iso et al., 2019). Re-

https://github.com/isomap/factedithttps://github.com/isomap/factedit

173

y′ AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission that was operated by PATIENT-2 .x̂′ AGENT-1 served as PATIENT-3 was a crew member of the BRIDGE-1 mission .x′ AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission .

(a) Example for insertion. The revised template y′ and the reference template x̂′ share subsequences. The set of triple templatesT \T̂ is {(BRIDGE-1, operator, PATIENT-2)}. Our method removes “that was operated by PATIENT-2” from the revisedtemplate y′ to create the draft template x′.

y′ AGENT-1 was created by BRIDGE-1 and PATIENT-2 .x̂′ The character of AGENT-1 , whose full name is PATIENT-1 , was created by BRIDGE-1 and PATIENT-2 .x′ AGENT-1 , whose full name is PATIENT-1 , was created by BRIDGE-1 and PATIENT-2 .

(b) Example for deletion. The revised template y′ and the reference template x̂′ share subsequences. The set of triple templatesT̂ \T is {(AGENT-1, fullName, PATIENT-1)}. Our method copies “whose full name is PATIENT-1” from the reference templatex′ to create the draft template x′.

Table 2: Examples for insertion and deletion, where words in green are matched, words in gray are not matched,words in blue are copied, and words in orange are removed. Best viewed in color.

searchers have developed methods to deal with theproblem using other texts as templates (Hashimotoet al., 2018; Guu et al., 2018; Peng et al., 2019).The difference between the approach and fact-based text editing is that the former is about table-to-text generation based on other texts, while thelatter is about text-to-text generation based on struc-tured data.

3 Data Creation

In this section, we describe our method of datacreation for fact-based text editing. The methodautomatically constructs a dataset from an existingtable-to-text dataset.

3.1 Data Sources

There are two benchmark datasets of table-to-text, WEBNLG (Gardent et al., 2017)2 and RO-TOWIRE(Wiseman et al., 2017)3. We create twodatasets on the basis of them, referred to as WEBE-DIT and ROTOEDIT respectively. In the datasets,each instance consists of a table (structured data)and an associated text (unstructured data) describ-ing almost the same content.4.

For each instance, we take the table as triplesof facts and the associated text as a revised text,and we automatically create a draft text. The setof triples is represented as T = {t}. Each triple tconsists of subject, predicate, and object, denoted

2The data is available at https://github.com/ThiagoCF05/webnlg. We utilize version 1.5.

3We utilize the ROTOWIRE-MODIFIED data providedby Iso et al. (2019) available at https://github.com/aistairc/rotowire-modified. The authors also pro-vide an information extractor for processing the data.

4In ROTOWIRE, we discard redundant box-scores and un-related sentences using the information extractor and heuristicrules.

as t = (subj, pred, obj). For simplicity, we referto the nouns or noun phrases of subject and objectsimply as entities. The revised text is a sequenceof words denoted as y. The draft text is a sequenceof words denoted as x.

Given the set of triples T and the revised text y,we aim to create a draft text x, such that x is not inaccordance with T , in contrast to y, and thereforetext editing from x to y is needed.

3.2 Procedure

Our method first creates templates for all the sets oftriples and revised texts and then constructs a drafttext for each set of triples and revised text based ontheir related templates.

Creation of templatesFor each instance, our method first delexical-izes the entity words in the set of triples T andthe revised text y to obtain a set of triple tem-plates T ′ and a revised template y′. For exam-ple, given T ={(Baymax, voice, Scott Adsit)} andy =“Scott Adsit does the voice for Baymax”, itproduces the set of triple templates T ′ ={(AGENT-1, voice, PATIENT-1)} and the revised templatey′ =“AGENT-1 does the voice for PATIENT-1”.Our method then collects all the sets of triple tem-plates T ′ and revised templates y′ and retains themin a key-value store with y′ being a key and T ′being a value.

Creation of draft textNext, our method constructs a draft text x using aset of triple templates T ′ and a revised template y′.For simplicity, it only considers the use of eitherinsertion or deletion in the text editing, and one caneasily make an extension of it to a more complex

https://github.com/ThiagoCF05/webnlghttps://github.com/ThiagoCF05/webnlghttps://github.com/aistairc/rotowire-modifiedhttps://github.com/aistairc/rotowire-modified

174

setting. Note that the process of data creation isreverse to that of text editing.

Given a pair of T ′ and y′, our method retrievesanother pair denoted as T̂ ′ and x̂′, such that y′ andx̂′ have the longest common subsequences. Werefer to x̂′ as a reference template. There are twopossibilities; T̂ ′ is a subset or a superset of T ′.(We ignore the case in which they are identical.)Our method then manages to change y′ to a drafttemplate denoted as x′ on the basis of the relationbetween T ′ and T̂ ′. If T̂ ′ ( T ′, then the drafttemplate x′ created is for insertion, and if T̂ ′ ) T ′,then the draft template x′ created is for deletion.

For insertion, the revised template y′ and thereference template x̂′ share subsequences, and theset of triples T \T̂ appear in y′ but not in x̂′. Ourmethod keeps the shared subsequences in y′, re-moves the subsequences in y′ about T \T̂ , andcopies the rest of words in y′, to create the drafttemplate x′. Table 2a gives an example. The sharedsubsequences “AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission” are kept. The set oftriple templates T \T̂ is {(BRIDGE-1, operator,PATIENT-2)}. The subsequence “that was oper-ated by PATIENT-2” is removed. Note that thesubsequence “served” is not copied because it isnot shared by y′ and x̂′.

For deletion, the revised template y′ and thereference template x̂′ share subsequences. Theset of triples T̂ \T appear in x̂′ but not in y′.Our method retains the shared subsequences iny′, copies the subsequences in x̂′ about T̂ \T ,and copies the rest of words in y′, to createthe draft template x′. Table 2b gives an exam-ple. The subsequences “AGENT-1 was created byBRIDGE-1 and PATIENT-2” are retained. Theset of triple templates T̂ \T is {(AGENT-1, full-Name, PATIENT-1)}. The subsequence “whosefull name is PATIENT-1” is copied. Note that thesubsequence “the character of” is not copied be-cause it is not shared by y′ and x̂′.

After getting the draft template x′, our methodlexicalizes it to obtain a draft text x, where thelexicons (entity words) are collected from the cor-responding revised text y.

We obtain two datasets with our method, referredto as WEBEDIT and ROTOEDIT, respectively. Ta-ble 3 gives the statistics of the datasets.

In the WEBEDIT data, sometimes entities onlyappear in the subj’s of triples. In such cases, wealso make them appear in the obj’s. To do so, we

WEBEDIT ROTOEDITTRAIN VALID TEST TRAIN VALID TEST

#D 181k 23k 29k 27k 5.3k 4.9k#Wd 4.1M 495k 624k 4.7M 904k 839k#Wr 4.2M 525k 649k 5.6M 1.1M 1.0M#S 403k 49k 62k 209k 40k 36k

Table 3: Statistics of WEBEDIT and ROTOEDIT, where#D is the number of instances, #Wd and #Wr are the to-tal numbers of words in the draft texts and the revisedtexts, respectively, and #S is total the number of sen-tences.

introduce an additional triple (ROOT, IsOf, subj)for each subj, where ROOT is a dummy entity.

4 FACTEDITOR

In this section, we describe our proposed model forfact-based text editing referred to as FACTEDITOR.

4.1 Model Architecture

FACTEDITOR transforms a draft text into a revisedtext based on given triples. The model consistsof three components, a buffer for storing the drafttext and its representations, a stream for storing therevised text and its representations, and a memoryfor storing the triples and their representations, asshown in Figure 1.

FACTEDITOR scans the text in the buffer, copiesthe parts of text from the buffer into the streamif they are described in the triples in the mem-ory, deletes the parts of the text if they are notmentioned in the triples, and inserts new parts ofnext into the stream which is only presented in thetriples.

The architecture of FACTEDITOR is inspired bythose in sentence parsing Dyer et al. (2015); Watan-abe and Sumita (2015). The actual processing ofFACTEDITOR is to generate a sequence of wordsinto the stream from the given sequence of wordsin the buffer and the set of triples in the memory.A neural network is employed to control the entireediting process.

4.2 Neural Network

InitializationFACTEDITOR first initializes the representations ofcontent in the buffer, stream, and memory.

There is a feed-forward network associated withthe memory, utilized to create the embeddings oftriples. Let M denote the number of triples. The

175

embedding of triple tj , j = 1, · · · ,M is calculatedas

tj = tanh(W t · [esubjj ; epredj ; e

objj ] + bt),

where W t and bt denote parameters,esubjj , e

predj , e

objj denote the embeddings

of subject, predicate, and object of triple tj , and[ ; ] denotes vector concatenation.

There is a bi-directional LSTM associated withthe buffer, utilized to create the embeddings ofwords of draft text. The embeddings are obtained asb = BILSTM(x), where x = (x1, . . . ,xN ) is thelist of embeddings of words and b = (b1, . . . , bN )is the list of representations of words, where Ndenotes the number of words.

There is an LSTM associated with the stream forrepresenting the hidden states of the stream. Thefirst hidden state is initialized as

s1 = tanh

(W s ·

[∑Ni=1 biN

;

∑Mj=1 tj

M

]+ bs

)

whereW s and bs denotes parameters.

Action predictionFACTEDITOR predicts an action at each time t us-ing the LSTM. There are three types of action,namely Keep, Drop, and Gen. First, it composesa context vector t̃t of triples at time t using atten-tion

t̃t =

M∑j=1

αt,jtj

where αt,j is a weight calculated as

αt,j ∝ exp(v>α · tanh (W α · [st; bt; tj ])

)where vα andW α are parameters. Then, it createsthe hidden state zt for action prediction at time t

zt = tanh(W z · [st; bt; t̃t] + bz

)where W z and bz denote parameters. Next, itcalculates the probability of action at

P (at | zt) = softmax(W a · zt)

where W a denotes parameters, and chooses theaction having the largest probability.

Stream Bufferst bt

poppush

tt~

(a) The Keep action, where the top embedding of the bufferbt is popped and the concatenated vector [̃tt; bt] is pushedinto the stream LSTM.

Stream Bufferst bt

pop

(b) The Drop action, where the top embedding of the bufferbt is popped and the state in the stream is reused at the nexttime step t+ 1.

Stream Buffer

tt

st bt

Wp yt~

push

(c) The Gen action, where the concatenated vector[t̃t;W pyt] is pushed into the stream, and the top embed-ding of the buffer is reused at the next time step t+ 1.

Figure 1: Actions of FACTEDITOR.

Action execution

FACTEDITOR takes action based on the predictionresult at time t.

For Keep at time t, FACTEDITOR pops the topembedding bt in the buffer, and feeds the combina-tion of the top embedding bt and the context vectorof triples t̃t into the stream, as shown in Fig. 1a.The state of stream is updated with the LSTMas st+1 = LSTM([̃tt; bt], st). FACTEDITOR alsocopies the top word in the buffer into the stream.

For Drop at time t, FACTEDITOR pops the topembedding in the buffer and proceeds to the nextstate, as shown in Fig. 1b. The state of streamis updated as st+1 = st. Note that no word isinputted into the stream.

For Gen at time t, FACTEDITOR does not popthe top embedding in the buffer. It feeds the

176

Draft text x Bakewell pudding is Dessert that can be served Warm or cold .

Revised text y Bakewell pudding is Dessert that originates from Derbyshire Dales .

Action sequence aKeep Keep Keep Keep Gen(originates) Gen(from) Gen(Derbyshire Dales)Drop Drop Drop Drop Keep

Table 4: An example of action sequence derived from a draft text and revised text.

combination of the context vector of triples t̃tand the linearly projected embedding of word winto the stream, as shown in Fig. 1c. The stateof stream is updated with the LSTM as st+1 =LSTM([̃tt;W pyt], st), where yt is the embed-ding of the generated word yt and W p denotesparameters. In addition, FACTEDITOR copies thegenerated word yt into the stream.

FACTEDITOR continues the actions until thebuffer becomes empty.

Word generationFACTEDITOR generates a word yt at time t, whenthe action is Gen,

Pgen(yt | zt) = softmax(W y · zt)

whereW y is parameters.To avoid generation of OOV words, FACTEDI-

TOR exploits the copy mechanism. It calculates theprobability of copying the object of triple tj

Pcopy(oj | zt) ∝ exp (v>c · tanh(W c · [zt; tj ]))

where vc andW c denote parameters, and oj is theobject of triple tj . It also calculates the probabilityof gating

pgate = sigmoid(w>g · zt + bg)

where wg and bg are parameters. Finally, it cal-culates the probability of generating a word wtthrough either generation or copying,

P (yt | zt) = pgatePgen(yt | zt)

+ (1− pgate)M∑

j=1:oj=yt

Pcopy(oj | zt),

where it is assumed that the triples in the memoryhave the same subject and thus only objects needto be copied.

4.3 Model LearningThe conditional probability of sequence of actionsa = (a1, a2, · · · , aT ) given the set of triples T and

the sequence of input words x can be written as

P (a | T ,x) =T∏t=1

P (at | zt)

where P (at | zt) is the conditional probability ofaction at given state zt at time t and T denotes thenumber of actions.

The conditional probability of sequence of gen-erated words y = (y1, y2, · · · , yT ) given the se-quence of actions a can be written as

P (y | a) =T∏t=1

P (yt | at)

where P (yt | at) is the conditional probability ofgenerated word yt given action at at time t, whichis calculated as

P (yt | at) =

{P (yt | zt) if at = Gen1 otherwise

Note that not all positions have a generated word.In such a case, yt is simply a null word.

The learning of the model is carried out via super-vised learning. The objective of learning is to min-imize the negative log-likelihood of P (a | T ,x)and P (y | a)

L(θ) = −T∑t=1

{logP (at | zt) + logP (yt | at)}

where θ denotes the parameters.A training instance consists of a pair of draft

text and revised text, as well as a set of triples,denoted as x, y, and T respectively. For eachinstance, our method derives a sequence of actionsdenoted as a, in a similar way as that in (Donget al., 2019). It first finds the longest common sub-sequence between x and y, and then selects anaction of Keep, Drop, or Gen at each position,according to how y is obtained from x and T (cf.,Tab. 4). Action Gen is preferred over action Dropwhen both are valid.

177

Table Encoder Decoder

yT

(a) Table-to-Text

T

Text Encoder Decoder

yx

(b) Text-to-Text

Table Encoder Text Encoder Decoder

yxT

(c) ENCDECEDITOR

Figure 2: Model architectures of the baselines. All models employ attention and copy mechanism.

4.4 Time Complexity

The time complexity of inference in FACTEDITORisO(NM), whereN is the number of words in thebuffer, and M is the number of triples. Scanningof data in the buffer is of complexity O(N). Thegeneration of action needs the execution of atten-tion, which is of complexity O(M). Usually, N ismuch larger than M .

4.5 Baseline

We consider a baseline method using the encoder-decoder architecture, which takes the set of triplesand the draft text as input and generates a revisedtext. We refer to the method as ENCDECEDITOR.The encoder of ENCDECEDITOR is the same asthat of FACTEDITOR. The decoder is the standardattention and copy model, which creates and uti-lizes a context vector and predicts the next word ateach time.

The time complexity of inference in ENCDE-CEDITOR isO(N2+NM) (cf.,Britz et al. (2017)).Note that in fact-based text editing, usually N isvery large. That means that ENCDECEDITOR isless efficient than FACTEDITOR.

5 Experiment

We conduct experiments to make comparison be-tween FACTEDITOR and the baselines using thetwo datasets WEBEDIT and ROTOEDIT.

5.1 Experiment Setup

The main baseline is the encoder-decoder modelENCDECEDITOR, as explained above. We furtherconsider three baselines, No-Editing, Table-to-Text,and Text-to-Text. In No-Editing, the draft text isdirectly used. In Table-to-Text, a revised text isgenerated from the triples using encoder-decoder.In Text-to-Text, a revised text is created from thedraft text using the encoder-decoder model. Figure2 gives illustrations of the baselines.

We evaluate the results of revised texts by themodels from the viewpoint of fluency and fidelity.

We utilize ExactMatch (EM), BLEU (Papineniet al., 2002) and SARI (Xu et al., 2016) scores5

as evaluation metrics for fluency. We also utilizeprecision, recall, and F1 score as evaluation metricsfor fidelity. For WEBEDIT, we extract the entitiesfrom the generated text and the reference text andthen calculate the precision, recall, and F1 scores.For ROTOEDIT, we use the information extractiontool provided by Wiseman et al. (2017) for calcula-tion of the scores.

For the embeddings of subject and object forboth datasets and the embedding of the predicatefor ROTOEDIT, we simply use the embeddinglookup table. For the embedding of the predi-cate for WEBEDIT, we first tokenize the predicate,lookup the embeddings of lower-cased words fromthe table, and use averaged embedding to deal withthe OOV problem (Moryossef et al., 2019).

We tune the hyperparameters based on the BLEUscore on a development set. For WEBEDIT, weset the sizes of embeddings, buffers, and triplesto 300, and set the size of the stream to 600. ForROTOEDIT, we set the size of embeddings to 100and set the sizes of buffers, triples, and stream to200. The initial learning rate is 2e-3, and AMS-Grad is used for automatically adjusting the learn-ing rate (Reddi et al., 2018). Our implementationmakes use of AllenNLP (Gardner et al., 2018).

5.2 Experimental Results

Quantitative evaluationWe present the performances of our proposedmodel FACTEDITOR and the baselines on fact-based text editing in Table 5. One can draw severalconclusions from the results.

First, our proposed model, FACTEDITOR,achieves significantly better performances than themain baseline, ENCDECEDITOR, in terms of al-most all measures. In particular, FACTEDITOR

5We use a modified version of SARI where β equals1.0, available at https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py

https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.pyhttps://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.pyhttps://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/sari_hook.py

178

Model FLUENCY FIDELITYBLEU SARI KEEP ADD DELETE EM P% R% F1%

BaselinesNo-Editing 66.67 31.51 78.62 3.91 12.02. 0. 84.49 76.34 80.21Table-to-Text 33.75 43.83 51.44 27.86 52.19 5.78 98.23 83.72 90.40Text-to-Text 63.61 58.73 82.62 25.77 67.80 6.22 81.93 77.16 79.48

Fact-based text editingENCDECEDITOR 71.03 69.59 89.49 43.82 75.48 20.96 98.06 87.56 92.51FACTEDITOR 75.68 72.20 91.84 47.69 77.07 24.80 96.88 89.74 93.17

(a) WEBEDIT

Model FLUENCY FIDELITYBLEU SARI KEEP ADD DELETE EM P% R% F1%

BaselinesNo-Editing 74.95 39.59 95.72 0.05 23.01 0. 92.92 65.02 76.51Table-to-Text 24.87 23.30 39.12 14.78 16.00 0. 48.01 24.28 32.33Text-to-Text 78.07 60.25 97.29 13.04 70.43 0.02 63.62 41.08 49.92

Fact-based text editingENCDECEDITOR 83.36 71.46 97.69 44.02 72.69 2.49 78.80 52.21 62.81FACTEDITOR 84.43 74.72 98.41 41.50 84.24 2.65 78.84 52.30 63.39

(b) ROTOEDIT

Table 5: Performances of FACTEDITOR and baselines on two datasets in terms of Fluency and Fidelity. EM standsfor exact match.

obtains significant gains in DELETE scores on bothWEBEDIT and ROTOEDIT.

Second, the fact-based text editing models(FACTEDITOR and ENCDECEDITOR) significantlyimprove upon the other models in terms of fluencyscores, and achieve similar performances in termsof fidelity scores.

Third, compared to No-Editing, Table-to-Texthas higher fidelity scores, but lower fluency scores.Text-to-Text has almost the same fluency scores,but lower fidelity scores on ROTOEDIT.

Qualitative evaluationWe also manually evaluate 50 randomly sampledrevised texts for WEBEDIT. We check whether therevised texts given by FACTEDITOR and ENCDE-CEDITOR include all the facts. We categorize thefactual errors made by the two models. Table 6shows the results. One can see that FACTEDITORcovers more facts than ENCDECEDITOR and hasless factual errors than ENCDECEDITOR.

FACTEDITOR has a larger number of correct edit-ing (CQT) than ENCDECEDITOR for fact-basedtext editing. In contrast, ENCDECEDITOR often in-cludes a larger number of unnecessary rephrasings(UPARA) than FACTEDITOR.

Covered facts Factual errorsCQT UPARA RPT MS USUP DREL

ENCDECEDITOR 14 7 16 21 3 12FACTEDITOR 24 4 9 19 1 3

Table 6: Evaluation results on 50 randomly sampled re-vised texts in WEBEDIT in terms of numbers of correctediting (CQT), unnecessary paraphrasing (UPARA),repetition (RPT), missing facts (MS), unsupported facts(USUP) and different relations (DREL)

There are four types of factual errors: fact repe-titions (RPT), fact missings (MS), fact unsupported(USUP), and relation difference (DREL). BothFACTEDITOR and ENCDECEDITOR often fail toinsert missing facts (MS), but rarely insert unsup-ported facts (USUP). ENCDECEDITOR often gen-erates the same facts multiple times (RPT) or factsin different relations (DREL). In contrast, FACTE-DITOR can seldomly make such errors.

Table 7 shows an example of results given byENCDECEDITOR and FACTEDITOR. The revisedtexts of both ENCDECEDITOR and FACTEDITORappear to be fluent, but that of FACTEDITORhas higher fidelity than that of ENCDECEDITOR.ENCDECEDITOR cannot effectively eliminate the

179

Set of triples

{(Ardmore Airport, runwayLength, 1411.0),(Ardmore Airport, 3rd runway SurfaceType, Poaceae),(Ardmore Airport, operatingOrganisation, Civil Aviation Authority of New Zealand),(Ardmore Airport, elevationAboveTheSeaLevel, 34.0),(Ardmore Airport, runwayName, 03R/21L)}

Draft textArdmore Airport , ICAO Location Identifier UTAA . Ardmore Airport 3rd runwayis made of Poaceae and Ardmore Airport . 03R/21L is 1411.0 m long and Ardmore Airportis 34.0 above sea level .

Revised textArdmore Airport is operated by Civil Aviation Authority of New Zealand . Ardmore Airport3rd runway is made of Poaceae and Ardmore Airport name is 03R/21L . 03R/21L is 1411.0 m longand Ardmore Airport is 34.0 above sea level .

ENCDECEDITORArdmore Airport , ICAO Location Identifier UTAA , is operated byCivil Aviation Authority of New Zealand . Ardmore Airport 3rd runway is made of Poaceae andArdmore Airport . 03R/21L is 1411.0 m long and Ardmore Airport is 34.0 m long .

FACTEDITORArdmore Airport is operated by Civil Aviation Authority of New Zealand . Ardmore Airport3rd runway is made of Poaceae and Ardmore Airport . 03R/21L is 1411.0 m long andArdmore Airport is 34.0 above sea level .

Table 7: Example of generated revised texts given by ENCDECEDITOR and FACTEDITOR on WEBEDIT. Entitiesin green appear in both the set of triples and the draft text. Entities in orange only appear in the draft text. Entitiesin blue should appear in the revised text but do not appear in the draft text.

WEBEDIT ROTOEDIT

Table-to-Text 4,083 1,834Text-to-Text 2,751 581

ENCDECEDITOR 2,487 505FACTEDITOR 3,295 1,412

Table 8: Runtime analysis (# of words/second). Table-to-Text always shows the fastest performance (Bold-faced). FACTEDITOR shows the second fastest runtimeperformance (Underlined).

description about an unsupported fact (in orange)appearing in the draft text. In contrast, FACTEDI-TOR can deal with the problem well. In addition,ENCDECEDITOR conducts an unnecessary substi-tution in the draft text (underlined). FACTEDITORtends to avoid such unnecessary editing.

Runtime analysisWe conduct runtime analysis on FACTEDITOR andthe baselines in terms of number of processedwords per second, on both WEBEDIT and RO-TOEDIT. Table 8 gives the results when the batchsize is 128 for all methods. Table-to-Text is thefastest, followed by FACTEDITOR. FACTEDITORis always faster than ENCDECEDITOR, apparentlybecause it has a lower time complexity, as ex-plained in Section 4. The texts in WEBEDIT are rel-atively short, and thus FACTEDITOR and ENCDE-CEDITOR have similar runtime speeds. In contrast,the texts in ROTOEDIT are relatively long, and thusFACTEDITOR executes approximately two timesfaster than ENCDECEDITOR.

6 Conclusion

In this paper, we have defined a new task referredto as fact-based text editing and made two contri-butions to research on the problem. First, we haveproposed a data construction method for fact-basedtext editing and created two datasets. Second, wehave proposed a model for fact-based text editing,named FACTEDITOR, which performs the task bygenerating a sequence of actions. Experimentalresults show that the proposed model FACTEDI-TOR performs better and faster than the baselines,including an encoder-decoder model.

Acknowledgments

We would like to thank the reviewers for their in-sightful comments.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural Machine Translation by JointlyLearning to Align and Translate. In InternationalConference on Learning Representations.

Denny Britz, Melody Guan, and Minh-Thang Luong.2017. Efficient Attention using a Fixed-Size Mem-ory Representation. In Proceedings of the 2017 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 392–400, Copenhagen, Denmark.Association for Computational Linguistics.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Hol-ger Schwenk, and Yoshua Bengio. 2014. Learn-ing Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In

https://arxiv.org/abs/1409.0473https://arxiv.org/abs/1409.0473https://doi.org/10.18653/v1/D17-1040https://doi.org/10.18653/v1/D17-1040https://doi.org/10.3115/v1/D14-1179https://doi.org/10.3115/v1/D14-1179https://doi.org/10.3115/v1/D14-1179

180

Proceedings of the 2014 Conference on EmpiricalMethods in Natural Language Processing (EMNLP),pages 1724–1734, Doha, Qatar. Association forComputational Linguistics.

William B. Dolan and Chris Brockett. 2005. Automati-cally constructing a corpus of sentential paraphrases.In Proceedings of the Third International Workshopon Paraphrasing (IWP2005).

Yue Dong, Zichao Li, Mehdi Rezagholizadeh, andJackie Chi Kit Cheung. 2019. EditNTS: An NeuralProgrammer-Interpreter Model for Sentence Simpli-fication through Explicit Editing. In Proceedings ofthe 57th Annual Meeting of the Association for Com-putational Linguistics, pages 3393–3402, Florence,Italy. Association for Computational Linguistics.

Chris Dyer, Miguel Ballesteros, Wang Ling, AustinMatthews, and Noah A. Smith. 2015. Transition-Based Dependency Parsing with Stack Long Short-Term Memory. In Proceedings of the 53rd AnnualMeeting of the Association for Computational Lin-guistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 1: LongPapers), pages 334–343, Beijing, China. Associa-tion for Computational Linguistics.

Claire Gardent, Anastasia Shimorina, Shashi Narayan,and Laura Perez-Beltrachini. 2017. Creating train-ing corpora for NLG micro-planners. In Proceed-ings of the 55th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 179–188, Vancouver, Canada. Associa-tion for Computational Linguistics.

Matt Gardner, Joel Grus, Mark Neumann, OyvindTafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe-ters, Michael Schmitz, and Luke Zettlemoyer. 2018.AllenNLP: A Deep Semantic Natural Language Pro-cessing Platform. In Proceedings of Workshop forNLP Open Source Software (NLP-OSS), pages 1–6, Melbourne, Australia. Association for Computa-tional Linguistics.

Albert Gatt and Emiel Krahmer. 2018. Survey of theState of the Art in Natural Language Generation:Core tasks, applications and evaluation. Journal ofArtificial Intelligence Research (JAIR), 61:65–170.

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K.Li. 2016. Incorporating copying mechanism insequence-to-sequence learning. In Proceedings ofthe 54th Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers),pages 1631–1640, Berlin, Germany. Association forComputational Linguistics.

Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati,Bowen Zhou, and Yoshua Bengio. 2016. Pointingthe unknown words. In Proceedings of the 54thAnnual Meeting of the Association for Computa-tional Linguistics (Volume 1: Long Papers), pages140–149, Berlin, Germany. Association for Compu-tational Linguistics.

Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren,and Percy Liang. 2018. Generating Sentences byEditing Prototypes. Transactions of the Associationfor Computational Linguistics, 6:437–450.

Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren,and Percy S Liang. 2018. A Retrieve-and-EditFramework for Predicting Structured Outputs. InAdvances in Neural Information Processing Systems,pages 10052–10062. Curran Associates, Inc.

Zhiting Hu, Zichao Yang, Xiaodan Liang, RuslanSalakhutdinov, and Eric P. Xing. 2017. Toward Con-trolled Generation of Text. In Proceedings of the34th International Conference on Machine Learning,volume 70 of Proceedings of Machine Learning Re-search, pages 1587–1596, International ConventionCentre, Sydney, Australia. PMLR.

Kentaro Inui, Atsushi Fujita, Tetsuro Takahashi, RyuIida, and Tomoya Iwakura. 2003. Text simplifica-tion for reading assistance: A project note. In Pro-ceedings of the Second International Workshop onParaphrasing, pages 9–16, Sapporo, Japan. Associa-tion for Computational Linguistics.

Hayate Iso, Yui Uehara, Tatsuya Ishigaki, HiroshiNoji, Eiji Aramaki, Ichiro Kobayashi, YusukeMiyao, Naoaki Okazaki, and Hiroya Takamura.2019. Learning to Select, Track, and Generate forData-to-Text. In Proceedings of the Annual Meet-ing of the Association for Computational Linguistics(ACL), pages 2102–2113, Florence, Italy.

Kevin Knight and Ishwar Chander. 1994. AutomatedPostediting of Documents. In Proceedings of theAAAI Conference on Artificial Intelligence., vol-ume 94, pages 779–784.

Rémi Lebret, David Grangier, and Michael Auli. 2016.Neural Text Generation from Structured Data withApplication to the Biography Domain. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 1203–1213,Austin, Texas. Association for Computational Lin-guistics.

Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li.2018. Paraphrase Generation with Deep Reinforce-ment Learning. In Proceedings of the 2018 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 3865–3878, Brussels, Belgium.Association for Computational Linguistics.

Eric Malmi, Sebastian Krause, Sascha Rothe, DaniilMirylenka, and Aliaksei Severyn. 2019. Encode,Tag, Realize: High-Precision Text Editing. InProceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 5053–5064, Hong Kong, China. Association for Computa-tional Linguistics.

https://www.aclweb.org/anthology/I05-5002https://www.aclweb.org/anthology/I05-5002https://doi.org/10.18653/v1/P19-1331https://doi.org/10.18653/v1/P19-1331https://doi.org/10.18653/v1/P19-1331https://doi.org/10.3115/v1/P15-1033https://doi.org/10.3115/v1/P15-1033https://doi.org/10.3115/v1/P15-1033https://doi.org/10.18653/v1/P17-1017https://doi.org/10.18653/v1/P17-1017https://doi.org/10.18653/v1/W18-2501https://doi.org/10.18653/v1/W18-2501http://arxiv.org/abs/https://arxiv.org/abs/1703.09902http://arxiv.org/abs/https://arxiv.org/abs/1703.09902http://arxiv.org/abs/https://arxiv.org/abs/1703.09902https://doi.org/10.18653/v1/P16-1154https://doi.org/10.18653/v1/P16-1154https://doi.org/10.18653/v1/P16-1014https://doi.org/10.18653/v1/P16-1014https://www.aclweb.org/anthology/Q18-1031https://www.aclweb.org/anthology/Q18-1031http://papers.nips.cc/paper/8209-a-retrieve-and-edit-framework-for-predicting-structured-outputs.pdfhttp://papers.nips.cc/paper/8209-a-retrieve-and-edit-framework-for-predicting-structured-outputs.pdfhttp://proceedings.mlr.press/v70/hu17e.htmlhttp://proceedings.mlr.press/v70/hu17e.htmlhttps://doi.org/10.3115/1118984.1118986https://doi.org/10.3115/1118984.1118986https://www.aclweb.org/anthology/P19-1202https://www.aclweb.org/anthology/P19-1202https://www.aaai.org/Papers/AAAI/1994/AAAI94-119.pdfhttps://www.aaai.org/Papers/AAAI/1994/AAAI94-119.pdfhttps://doi.org/10.18653/v1/D16-1128https://doi.org/10.18653/v1/D16-1128https://doi.org/10.18653/v1/D18-1421https://doi.org/10.18653/v1/D18-1421https://doi.org/10.18653/v1/D19-1510https://doi.org/10.18653/v1/D19-1510

181

Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019.Step-by-Step: Separating Planning from Realizationin Neural Data-to-Text Generation. In Proceedingsof the 2019 Conference of the North American Chap-ter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Longand Short Papers), pages 2267–2277, Minneapolis,Minnesota. Association for Computational Linguis-tics.

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, ChristianHadiwinoto, Raymond Hendy Susanto, and Christo-pher Bryant. 2014. The CoNLL-2014 Shared Taskon Grammatical Error Correction. In Proceedingsof the Eighteenth Conference on Computational Nat-ural Language Learning: Shared Task, pages 1–14, Baltimore, Maryland. Association for Computa-tional Linguistics.

Jekaterina Novikova, Ondřej Dušek, and Verena Rieser.2017. The E2E Dataset: New Challenges For End-to-End Generation. In Proceedings of the 18th An-nual SIGdial Meeting on Discourse and Dialogue,pages 201–206, Saarbrücken, Germany. Associationfor Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Com-putational Linguistics, pages 311–318, Philadelphia,Pennsylvania, USA. Association for ComputationalLinguistics.

Hao Peng, Ankur Parikh, Manaal Faruqui, BhuwanDhingra, and Dipanjan Das. 2019. Text Generationwith Exemplar-based Adaptive Decoding. In Pro-ceedings of the 2019 Conference of the North Amer-ican Chapter of the Association for ComputationalLinguistics: Human Language Technologies, Vol-ume 1 (Long and Short Papers), pages 2555–2565,Minneapolis, Minnesota. Association for Computa-tional Linguistics.

Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.”Data-to-text Generation with Entity Modeling”. InProceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages2023–2035, Florence, Italy. Association for Compu-tational Linguistics.

Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2018.On the convergence of adam and beyond. In Inter-national Conference on Learning Representations.

Scott Reed and Nando De Freitas. 2016. NeuralProgrammer-Interpreters. In International Confer-ence on Learning Representations.

Ehud Reiter and Robert Dale. 2000. Building NaturalLanguage Generation Systems. Studies in NaturalLanguage Processing. Cambridge University Press.

Tianxiao Shen, Tao Lei, Regina Barzilay, and TommiJaakkola. 2017. Style Transfer from Non-Parallel

Text by Cross-Alignment. In Advances in Neural In-formation Processing Systems 30, pages 6830–6841.Curran Associates, Inc.

Michel Simard, Cyril Goutte, and Pierre Isabelle. 2007.Statistical Phrase-Based Post-Editing. In HumanLanguage Technologies 2007: The Conference ofthe North American Chapter of the Association forComputational Linguistics; Proceedings of the MainConference, pages 508–515, Rochester, New York.Association for Computational Linguistics.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to Sequence Learning with Neural Net-works. In Advances in Neural Information Pro-cessing Systems 27, pages 3104–3112. Curran As-sociates, Inc.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, Ł ukaszKaiser, and Illia Polosukhin. 2017. Attention is Allyou Need. In Advances in Neural Information Pro-cessing Systems 30, pages 5998–6008. Curran Asso-ciates, Inc.

Thuy-Trang Vu and Gholamreza Haffari. 2018. Auto-matic Post-Editing of Machine Translation: A Neu-ral Programmer-Interpreter Approach. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 3048–3053,Brussels, Belgium. Association for ComputationalLinguistics.

Taro Watanabe and Eiichiro Sumita. 2015. Transition-based neural constituent parsing. In Proceedingsof the 53rd Annual Meeting of the Association forComputational Linguistics and the 7th InternationalJoint Conference on Natural Language Processing(Volume 1: Long Papers), pages 1169–1179, Beijing,China. Association for Computational Linguistics.

Sam Wiseman, Stuart Shieber, and Alexander Rush.2017. ”Challenges in Data-to-Document Genera-tion”. In Proceedings of the 2017 Conference onEmpirical Methods in Natural Language Processing,pages 2253–2263, Copenhagen, Denmark. Associa-tion for Computational Linguistics.

Sander Wubben, Antal van den Bosch, and Emiel Krah-mer. 2012. Sentence simplification by monolingualmachine translation. In Proceedings of the 50th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1015–1024, Jeju Island, Korea. Association for Computa-tional Linguistics.

Wei Xu, Courtney Napoles, Ellie Pavlick, QuanzeChen, and Chris Callison-Burch. 2016. OptimizingStatistical Machine Translation for Text Simplifica-tion. Transactions of the Association for Computa-tional Linguistics, 4:401–415.

Diyi Yang, Aaron Halfaker, Robert Kraut, and Ed-uard Hovy. 2017. Identifying Semantic Edit Inten-tions from Revisions in Wikipedia. In Proceed-ings of the 2017 Conference on Empirical Methods

https://doi.org/10.18653/v1/N19-1236https://doi.org/10.18653/v1/N19-1236https://doi.org/10.3115/v1/W14-1701https://doi.org/10.3115/v1/W14-1701https://doi.org/10.18653/v1/W17-5525https://doi.org/10.18653/v1/W17-5525https://doi.org/10.3115/1073083.1073135https://doi.org/10.3115/1073083.1073135https://doi.org/10.18653/v1/N19-1263https://doi.org/10.18653/v1/N19-1263https://doi.org/10.18653/v1/P19-1195https://arxiv.org/abs/1904.09237https://arxiv.org/abs/1511.06279https://arxiv.org/abs/1511.06279https://doi.org/10.1017/CBO9780511519857https://doi.org/10.1017/CBO9780511519857http://papers.nips.cc/paper/7259-style-transfer-from-non-parallel-text-by-cross-alignment.pdfhttp://papers.nips.cc/paper/7259-style-transfer-from-non-parallel-text-by-cross-alignment.pdfhttps://www.aclweb.org/anthology/N07-1064http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdfhttp://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdfhttp://papers.nips.cc/paper/7181-attention-is-all-you-need.pdfhttp://papers.nips.cc/paper/7181-attention-is-all-you-need.pdfhttps://doi.org/10.18653/v1/D18-1341https://doi.org/10.18653/v1/D18-1341https://doi.org/10.18653/v1/D18-1341https://doi.org/10.3115/v1/P15-1113https://doi.org/10.3115/v1/P15-1113https://doi.org/10.18653/v1/D17-1239https://doi.org/10.18653/v1/D17-1239https://www.aclweb.org/anthology/P12-1107https://www.aclweb.org/anthology/P12-1107https://doi.org/10.1162/tacl_a_00107https://doi.org/10.1162/tacl_a_00107https://doi.org/10.1162/tacl_a_00107https://doi.org/10.18653/v1/D17-1213https://doi.org/10.18653/v1/D17-1213

182

in Natural Language Processing, pages 2000–2010,Copenhagen, Denmark. Association for Computa-tional Linguistics.

Pengcheng Yin, Graham Neubig, Miltiadis Allamanis,Marc Brockschmidt, and Alexander L Gaunt. 2019.Learning to Represent Edits. In International Con-ference on Learning Representations.

Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono,and Bambang Parmanto. 2018. Integrating Trans-former and Paraphrase Rules for Sentence Simpli-fication. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing,pages 3164–3173, Brussels, Belgium. Associationfor Computational Linguistics.

Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, andJingming Liu. 2019. ”Improving Grammatical Er-ror Correction via Pre-Training a Copy-AugmentedArchitecture with Unlabeled Data”. In Proceed-ings of the 2019 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 156–165, Minneapo-lis, Minnesota. Association for Computational Lin-guistics.

https://openreview.net/forum?id=BJl6AjC5F7https://doi.org/10.18653/v1/D18-1355https://doi.org/10.18653/v1/D18-1355https://doi.org/10.18653/v1/D18-1355https://doi.org/10.18653/v1/N19-1014https://doi.org/10.18653/v1/N19-1014https://doi.org/10.18653/v1/N19-1014