Simultaneously Self-Attending to All Mentions forFull-Abstract Biological Relation Extraction

Patrick Verga, Emma Strubell, Andrew McCallum

College of Information and Computer SciencesUniversity of Massachusetts Amherst

{pat, strubell, mccallum}@cs.umass.edu

Abstract

Most work in relation extraction forms aprediction by looking at a short span oftext within a single sentence containing asingle entity pair mention. This approachoften does not consider interactions acrossmentions, requires redundant computationfor each mention pair, and ignores rela-tionships expressed across sentence bound-aries. These problems are exacerbated bythe document- (rather than sentence-) levelannotation common in biological text. Inresponse, we propose a model which simul-taneously predicts relationships between allmention pairs in a document. We form pair-wise predictions over entire paper abstractsusing an efficient self-attention encoder. All-pairs mention scores allow us to performmulti-instance learning by aggregating overmentions to form entity pair representa-tions. We further adapt to settings withoutmention-level annotation by jointly trainingto predict named entities and adding a cor-pus of weakly labeled data. In experimentson two Biocreative benchmark datasets, weachieve state of the art performance on theBiocreative V Chemical Disease Relationdataset for models without external KB re-sources. We also introduce a new datasetan order of magnitude larger than existinghuman-annotated biological information ex-traction datasets and more accurate thandistantly supervised alternatives.

1 Introduction

With few exceptions (Swampillai and Stevenson,2011; Quirk and Poon, 2017; Peng et al., 2017),nearly all work in relation extraction focuses on clas-sifying a short span of text within a single sentencecontaining a single entity pair mention. However,relationships between entities are often expressedacross sentence boundaries or otherwise require alarger context to disambiguate. For example, 30%of relations in the Biocreative V CDR dataset (§3.1)

are expressed across sentence boundaries, such as inthe following excerpt expressing a relationship be-tween the chemical azathioprine and the diseasefibrosis:

Treatment of psoriasis with azathioprine.Azathioprine treatment benefited 19 (66%)out of 29 patients suffering from severe pso-riasis. Haematological complications werenot troublesome and results of biochemicalliver function tests remained normal. Min-imal cholestasis was seen in two cases andportal fibrosis of a reversible degree in eight.Liver biopsies should be undertaken at regularintervals if azathioprine therapy is contin-ued so that structural liver damage may bedetected at an early and reversible stage.

Though the entities’ mentions never occur in thesame sentence, the above example expresses thatthe chemical entity azathioprine can cause the sideeffect fibrosis. Relation extraction models whichconsider only within-sentence relation pairs can-not extract this fact without knowledge of thecomplicated coreference relationship between eightand azathioprine treatment, which, without featuresfrom a complicated pre-processing pipeline, cannotbe learned by a model which considers entity pairsin isolation. Making separate predictions for eachmention pair also obstructs multi-instance learning(Riedel et al., 2010; Surdeanu et al., 2012), a tech-nique which aggregates entity representations frommentions in order to improve robustness to noise inthe data. Like the majority of relation extractiondata, most annotation for biological relations is dis-tantly supervised, and so we could benefit from amodel which is amenable to multi-instance learning.In addition to this loss of cross-sentence and

cross-mention reasoning capability, traditional men-tion pair relation extraction models typically intro-duce computational inefficiencies by independentlyextracting features for and scoring every pair ofmentions, even when those mentions occur in thesame sentence and thus could share representations.In the CDR training set, this requires separatelyencoding and classifying each of the 5,318 candi-date mention pairs independently, versus encodingeach of the 500 abstracts once. Though abstracts

arX

iv:1

802.

1056

9v1

[cs

.CL

] 2

8 Fe

b 20

18

are longer than e.g. the text between mentions,many sentences contain multiple mentions, leadingto redundant computation.

However, encoding long sequences in a way whicheffectively incorporates long-distance context can beprohibitively expensive. Long Short Term Memory(LSTM) networks (Hochreiter and Schmidhuber,1997) are among the most popular token encodersdue to their capacity to learn high-quality repre-sentations of text, but their ability to leverage thefastest computing hardware is thwarted due to theircomputational dependence on the length of the se-quence — each token’s representation requires asinput the representation of the previous token, lim-iting the extent to which computation can be par-allelized. Convolutional neural networks (CNNs),in contrast, can be executed entirely in parallelacross the sequence, but the amount of contextincorporated into a single token’s representationis limited by the depth of the network, and verydeep networks can be difficult to learn (Hochreiter,1998). These problems are exacerbated by longersequences, limiting the extent to which previouswork explored full-abstract relation extraction.

To facilitate efficient full-abstract relation ex-traction from biological text, we propose Bi-affineRelation Attention Networks (BRANs), a combi-nation of network architecture, multi-instance andmulti-task learning designed to extract relations be-tween entities in biological text without requiringexplicit mention-level annotation. We synthesizeconvolutions and self-attention, a modification ofthe Transformer encoder introduced by Vaswaniet al. (2017), over sub-word tokens to efficientlyincorporate into token representations rich contextbetween distant mention pairs across the entire ab-stract. We score all pairs of mentions in parallelusing a bi-affine operator, and aggregate over men-tion pairs using a soft approximation of the maxfunction in order to perform multi-instance learning.We jointly train the model to predict relations andentities, further improving robustness to noise andlack of gold annotation at the mention level.

In extensive experiments on two benchmark bio-logical relation extraction datasets, we achieve stateof the art performance for a model using no exter-nal knowledge base resources in experiments on theBiocreative V CDR dataset, and outperform com-parable baselines on the Biocreative VI ChemProtdataset. We also introduce a new dataset whichis an order of magnitude larger than existing gold-annotated biological relation extraction datasetswhile covering a wider range of entity and relationtypes and with higher accuracy than distantly su-pervised datasets of the same size. We provide astrong baseline on this new dataset, and encourageits use as a benchmark for future biological relation

extraction systems.1

2 Model

We designed our model to efficiently encode longcontexts spanning multiple sentences while formingpairwise predictions without the need for mentionpair-specific features. To do this, our model first en-codes input token embeddings using self-attention.These embeddings are used to predict both entitiesand relations. The relation extraction module con-verts each token to a head and tail representation.These representations are used to form mentionpair predictions using a bi-affine operation with re-spect to learned relation embeddings. Finally, thesemention pair predictions are pooled to form entitypair predictions, expressing whether each relationtype is expressed by each relation pair.

2.1 Inputs

Our model takes in a sequence of N token em-beddings in Rd. Because the Transformer has noinnate notion of token position, the model relieson positional embeddings which are added to theinput token embeddings.2 We learn the positionembedding matrix Pm×d which contains a sepa-rate d dimensional embedding for each position,limited to m possible positions. Our final inputrepresentation for token xi is:

xi = si + pi

where si is the token embedding for xi and pi isthe positional embedding for the ith position. If iexceeds m, we use a randomly initialized vector inplace of pi.We tokenize the text using byte pair encoding

(BPE) (Gage, 1994; Sennrich et al., 2015). TheBPE algorithm constructs a vocabulary of sub-wordpieces, beginning with single characters. Then, thealgorithm iteratively merges the most frequent co-occurring tokens into a new token, which is addedto the vocabulary. This procedure continues untila pre-defined vocabulary size is met.

BPE is well suited for biological data for the fol-lowing reasons. First, biological entities often haveunique mentions made up of meaningful subcompo-nents, such as 1,2-dimethylhydrazine. Additionally,tokenization of chemical entities is challenging, lack-ing a universally agreed upon algorithm (Krallingeret al., 2015). As we demonstrate in §3.3.2, the sub-word representations produced by BPE allow themodel to formulate better predictions, likely due tobetter modeling of rare and unknown words.

1Our code and data are publicly available at: https://github.com/patverga/bran.

2Though our final model incorporates some convolu-tions, we retain the position embeddings.

Tran

sfor

mer

LogSumExp

Hypersensitivityto carbam

azepinepresentingwithleukem

oidreaction,eosinophilia,erythroderm

a.

and

TheFirstreportof suchreactionto carbam

azepine

Entity-Pair Relation Scores

4.3

0.1

Head MLP

Hypersensitivityto

carbamazepinepresenting

withleukemoidreaction,

eosinophilia,and

erythroderma.TheFirst

reportof

suchreaction

tocarbamazepine.

Relatio

ns

Tail MLP

Convolutions

Multi-head Attention

Bx

Figure 1: The relation extraction architecture.Inputs are contextually encoded using the Trans-former(Vaswani et al., 2017), made up of Blayers of multi-head attention and convolutionsubcomponents. Each transformed token is thenpassed through a head and tail MLP to producetwo position-specific representations. A bi-affineoperation is performed between each head andtail representation with respect to each rela-tion’s embedding matrix, producing a pair-wiserelation affinity tensor. Finally, the scores forcells corresponding to the same entity pair arepooled with a separate LogSumExp operationfor each relation to get a final score. The coloredtokens illustrate calculating the score for a givenpair of entities; the model is only given entityinformation when pooling over mentions.

2.2 Transformer

We base our token encoder on the Transformerself-attention model (Vaswani et al., 2017). TheTransformer is made up of B blocks. Each Trans-

former block, which we denote Transformerk, hasits own set of parameters and is made up of twosubcomponents: multi-head attention and a seriesof convolutions3. The output for token i of block k,b(k)i , is connected to its input b(k−1)i with a resid-ual connection (He et al., 2016). Starting withb(0)i = xi:

b(k)i = b

(k−1)i +Transformerk(b

(k−1)i )

2.2.1 Multi-head AttentionMulti-head attention applies self-attention multipletimes over the same inputs using separately nor-malized parameters (attention heads) and combinesthe results, as an alternative to applying one passof attention with more parameters. The intuitionbehind this modeling decision is that dividing theattention into multiple heads make it easier for themodel to learn to attend to different types of rele-vant information with each head. The self-attentionupdates input b(k−1)i by performing a weighted sumover all tokens in the sequence, weighted by theirimportance for modeling token i.

Each input is projected to a key k, value v, andquery q, using separate affine transformations withReLU activations (Glorot et al., 2011). Here, k,v, and q are each in R d

H where H is the numberof heads. The attention weights aijh for head hbetween tokens i and j are computed using scaleddot-product attention:

aijh = σ

(qTihkjh√

d

)oih =

∑j

vjh � aijh

with � denoting element-wise multiplication and σindicating a softmax along the jth dimension. Thescaled attention is meant to aid optimization byflattening the softmax and better distributing thegradients (Vaswani et al., 2017).The outputs of the individual attention heads

are concatenated, denoted [·; ·], into oi. All layersin the network use residual connections betweenthe output of the multi-headed attention and its in-put. Layer normalization (Ba et al., 2016), denotedLN(·), is then applied to the output.

oi = [o1; ...; oh]

mi = LN(b(k−1)i + oi)

2.2.2 ConvolutionsThe second part of our Transformer block is a stackof convolutional layers. The sub-network used in

3The original Transformer uses feed-forward con-nections, i.e. width-1 convolutions, whereas we useconvolutions with width > 1.

Vaswani et al. (2017) uses two width-1 convolu-tions. We add a third middle layer with kernelwidth 5, which we found to perform better. Manyrelations are expressed concisely by the immediatelocal context, e.g. Michele’s husband Barack, orlabetalol-induced hypotension. Adding this explicitn-gram modeling is meant to ease the burden onthe model to learn to attend to local features. Weuse Cw(·) to denote a convolutional operator withkernel width w. Then the convolutional portion ofthe transformer block is given by:

t(0)i = ReLU(C1(mi))

t(1)i = ReLU(C5(t

(0)i ))

t(2)i = C1(t

(1)i )

Where the dimensions of t(0)i and t(1)i are in R4d

and that of t(2)i is in Rd.

2.3 Bi-affine Pairwise ScoresWe project each contextually encoded token b(B)

i

through two separate MLPs to generate two newversions of each token corresponding to whetherit will serve as the first (head) or second (tail)argument of a relation:

eheadi =W(1)head(ReLU(W

(0)headb

(B)i ))

etaili =W(1)tail(ReLU(W

(0)tailb

(B)i ))

We use a bi-affine operator to calculate anN×L×Ntensor A of pairwise affinity scores, scoring each(head, relation, tail) triple:

Ailj = (eheadi L)etailj

where L is a d×L× d tensor, a learned embeddingmatrix for each of the L relations. In subsequentsections we will assume we have transposed thedimensions of A as d× d× L for ease of indexing.

2.4 Entity Level PredictionOur data is weakly labeled in that there are labelsat the entity level but not the mention level, makingthe problem a form of strong-distant supervision(Mintz et al., 2009). In distant supervision, edgesin a knowledge graph are heuristically applied tosentences in an auxiliary unstructured text corpus— often applying the edge label to all sentencescontaining the subject and object of the relation.Because this process is imprecise and introducesnoise into the training data, methods like multi-instance learning were introduced (Riedel et al.,2010; Surdeanu et al., 2012). In multi-instancelearning, rather than looking at each distantly la-beled mention pair in isolation, the model is trainedover the aggregate of these mentions and a singleupdate is made. More recently, the weighting func-tion of the instances has been expressed as neural

network attention (Verga and McCallum, 2016; Linet al., 2016; Yaghoobzadeh et al., 2017).We aggregate over all representations for each

mention pair in order to produce per-relationscores for each entity pair. For each entity pair(phead, ptail), let Phead denote the set of indices ofmentions of the entity phead, and let P tail denotethe indices of mentions of the entity ptail. Thenwe use the LogSumExp function to aggregate therelation scores from A across all pairs of mentionsof phead and ptail:

scores(phead, ptail) = log∑

i∈Phead

j∈P tail

exp(Aij)

The LogSumExp scoring function is a smooth ap-proximation to the max function and has the bene-fits of aggregating information from multiple predic-tions and propagating dense gradients as opposedto the sparse gradient updates of the max (Daset al., 2017).

2.5 Named Entity Recognition

In addition to pairwise relation predictions, weuse the Transformer output b(B)

i to make entitytype predictions. We feed b(B)

i as input to a linearclassifier which predicts the entity label for eachtoken with per-class scores ci:

ci =W (3)b(B)i

We augment the entity type labels with the BIOencoding to denote entity spans. We apply tagsto the byte-pair tokenization by treating each sub-word within a mention span as an additional tokenwith a corresponding B- or I- label.

2.6 Training

We train both the NER and relation extraction com-ponents of our network to perform multi-class clas-sification using maximum likelihood, where NERclasses yi or relation classes ri are conditionallyindependent given deep features produced by ourmodel with probabilities given by the softmax func-tion. In the case of NER, features are given by theper-token output of the transformer:

1

N

N∑i=1

logP (yi | b(B)i )

In the case of relation extraction, the features foreach entity pair are given by the LogSumExp overpairwise scores described in § 2.4. For E entitypairs, the relation ri is given by:

1

E

E∑i=1

logP (ri | scores(phead, ptail))

We train the NER and relation objectives jointly,sharing all embeddings and Transformer parame-ters. To trade off the two objectives, we penalizethe named entity updates with a hyperparameterλ.

3 ResultsWe evaluate our model on three datasets: TheBiocreative V Chemical Disease Relation bench-mark (CDR), which models relations betweenchemicals and diseases (§3.1); the Biocreative VIChemProt benchmark (CPR), which models rela-tions between chemicals and proteins (§3.2); and anew, large and accurate dataset we describe in §3.3based on the human curation in the Chemical Toxi-cology Database (CTD), which models relationshipsbetween chemicals, proteins and genes.The CDR dataset is annotated at the level of

paper abstracts, requiring consideration of long-range, cross sentence relationships, thus evaluationon this dataset demonstrates that our model iscapable of such reasoning. We also evaluate ourmodel’s performance in the more traditional settingwhich does not require cross-sentence modeling byperforming experiments on the CPR dataset, forwhich all annotations are between two entity men-tions in a single sentence. Finally, we present anew dataset constructed using strong-distant su-pervision (§2.4), with annotations at the documentlevel. This dataset is significantly larger than theothers, contains more relation types, and requiresreasoning across sentences.

3.1 Chemical Disease Relations DatasetThe Biocreative V chemical disease relation extrac-tion (CDR) dataset4 (Li et al., 2016a; Wei et al.,2016) was derived from the Comparative Toxicoge-nomics Database (CTD), which curates interactionsbetween genes, chemicals, and diseases (Davis et al.,2008). CTD annotations are only at the documentlevel and do not contain mention annotations. TheCDR dataset is a subset of these original annota-tions, supplemented with human annotated, entitylinked mention annotations. The relation annota-tions in this dataset are also at the document levelonly.

3.1.1 Data PreprocessingThe CDR dataset is concerned with extractingonly chemically-induced disease relationships (drug-related side effects and adverse reactions) concern-ing the most specific entity in the document. Forexample tobacco causes cancer could be marked asfalse if the document contained the more specificlung cancer. This can cause true relations to belabeled as false, harming evaluation performance.To address this we follow (Gu et al., 2016, 2017)

4http://www.biocreative.org/

and filter hypernyms according to the hierarchyin the MESH controlled vocabulary5. All entitypairs within the same abstract that do not have anannotated relation are assigned the NULL label.In addition to the gold CDR data, Peng et al.

(2016) add 15,448 PubMed abstracts annotated inthe CTD dataset. We consider this same set ofabstracts as additional training data (which wesubsequently denote +Data). Since this data doesnot contain entity annotations, we take the anno-tations from Pubtator (Wei et al., 2013), a stateof the art biological named entity tagger and en-tity linker. See §A.1 for additional data processingdetails. In our experiments we only evaluate ourrelation extraction performance and all models (in-cluding baselines) use gold entity annotations forpredictions.The byte pair vocabulary is generated over the

training dataset — we use a budget of 2500 tokenswhen training on the gold CDR data, and a largerbudget of 10,000 tokens when including extra datadescribed above Additional implementation detailsare included in Appendix A.

Data split Docs Pos NegTrain 500 1,038 4,280Development 500 1,012 4,136Test 500 1,066 4,270CTD 15,448 26,657 146,057

Table 1: Data statistics for the CDR Dataset andadditional data from CTD. Shows the total num-ber of abstracts, positive examples, and negativeexamples for each of the data set splits.

3.1.2 BaselinesWe compare against the previous best reportedresults on this dataset not using knowledge basefeatures.6 Each of the baselines are ensemble meth-ods for within- and cross-sentence relations thatmake use of additional linguistic features (syntacticparse and part-of-speech). Gu et al. (2017) en-code mention pairs using a CNN while Zhou et al.(2016a) use an LSTM. Both make cross-sentencepredictions with featurized classifiers.

3.1.3 ResultsIn Table 2 we show results outperforming the base-lines despite using no linguistic features. We showperformance averaged over 20 runs with 20 randomseeds as well as an ensemble of their averaged pre-dictions. We see a further boost in performanceby adding weakly labeled data. Table 3 shows the

5https://www.nlm.nih.gov/mesh/download/2017MeshTree.txt

6The highest reported score is from (Peng et al.,2016), but they use explicit lookups into the CTDknowledge base for the existence of the test entity pair.

Model P R F1Gu et al. (2016) 62.0 55.1 58.3Zhou et al. (2016a) 55.6 68.4 61.3Gu et al. (2017) 55.7 68.1 61.3BRAN 55.6 70.8 62.1 ± 0.8+ Data 64.0 69.2 66.2 ± 0.8BRAN(ensemble) 63.3 67.1 65.1+ Data 65.4 71.8 68.4

Table 2: Precision, recall, and F1 results on theBiocreative V CDR Dataset.

Model P R F1BRAN (Full) 55.6 70.8 62.1 ± 0.8– CNN only 43.9 65.5 52.4 ± 1.3– no width-5 48.2 67.2 55.7 ± 0.9– no NER 49.9 63.8 55.5 ± 1.8

Table 3: Results on the Biocreative V CDR Datasetshowing precision, recall, and F1 for various modelablations.

effects of ablating pieces of our model. ‘CNN only’removes the multi-head attention component fromthe transformer block, ‘no width-5’ replaces thewidth-5 convolution of the feed-forward componentof the transformer with a width-1 convolution and‘no NER’ removes the named entity recognitionmulti-task objective (§2.5).

3.2 Chemical Protein Relations DatasetTo assess our model’s performance in settings wherecross-sentence relationships are not explicitly evalu-ated, we perform experiments on the Biocreative VIChemProt dataset (CDR) (Krallinger et al., 2017).This dataset is concerned with classifying into sixrelation types between chemicals and proteins, withnearly all annotated relationships occurring withinthe same sentence.

3.2.1 BaselinesWe compare our models against those competing inthe official Biocreative VI competition (Liu et al.,2017). We compare to the top performing teamwhose model is directly comparable with ours — i.e.used a single (non-ensemble) model trained only onthe training data (many teams use the developmentset as additional training data). The baseline mod-els are standard state of the art relation extractionmodels: CNNs and Gated RNNs with attention.Each of these baselines uses mention-specific fea-tures encoding relative position of each token tothe two target entities being classified, whereas ourmodel aggregates over all mention pairs in each sen-tence. It is also worth noting that these models usea large vocabulary of pre-trained word embeddings,giving their models the advantage of far more modelparameters, as well as additional information fromunsupervised pre-training.

Model P R F1CNN† 50.7 43.0 46.5GRU+Attention† 53.0 46.3 49.5BRAN 48.0 54.1 50.8 ± .01

Table 4: Precision, recall, and F1 results on theBiocreative VI Chem-Prot Dataset. † denotes re-sults from Liu et al. (2017)

3.2.2 ResultsIn Table 4 we see that even though our modelforms all predictions simultaneously between allpairs of entities within the sentence, we are ableto outperform state of the art models classifyingeach mention pair independently. The scores shownare averaged across 10 runs with 10 random seeds.Interestingly, our model appears to have higherrecall and lower precision, while the baseline modelsare both precision-biased, with lower recall. Thissuggests that combining these styles of model couldlead to further gains on this task.

3.3 New CTD Dataset

3.3.1 DataExisting biological relation extraction datasets in-cluding both CDR (§3.1) and CPR (§3.2) are rela-tively small, typically consisting of hundreds or afew thousand annotated examples. Distant supervi-sion datasets apply document-independent, entity-level annotations to all sentences leading to a largeproportion of incorrect labels. Evaluations on thisdata involve either very small (a few hundred) goldannotated examples or cross validation to predictthe noisy, distantly applied labels (Mallory et al.,2015; Quirk and Poon, 2017; Peng et al., 2017).

We address these issues by constructing a newdataset using strong-distant supervision containingdocument-level annotations. The Comparative Tox-icogenomics Database (CTD) curates interactionsbetween genes, chemicals, and diseases. Each rela-tion in the CTD is associated with a disambiguatedentity pair and a PubMed article where the relationwas observed.

To construct this dataset, we collect the abstractsfor each of the PubMed articles with at least onecurated relation in the CTD database. As in §3.1,we use PubTator to automatically tag and disam-biguate the entities in each of these abstracts. Ifboth entities in the relation are found in the ab-stract, we take the (abstract, relation) pair as apositive example. The evidence for the curated re-lation could occur anywhere in the full text article,not just the abstract. Abstracts with no recoveredrelations are discarded. All other entity pairs withvalid types and without an annotated relation thatoccur in the remaining abstracts are considered neg-

Types Docs Pos NegTotal 68,400 166,474 1,198,493Chemical/Disease 64,139 93,940 571,932Chemical/Gene 34,883 63,463 360,100Gene/Disease 32,286 9,071 266,461

Table 5: Data statistics for the new CTD dataset.

ative examples and assigned the NULL label. Weadditionally remove abstracts containing greaterthan 500 tokens7. This limit removed about 10% ofthe total data including numerous extremely longabstracts. The average token length of the remain-ing data was 2̃30 tokens. With this procedure, weare able to collect 166,474 positive examples over13 relation types, with more detailed statistics ofthe dataset listed in Table 5.We consider relations between chemical-disease,

chemical-gene, and gene-disease entity pairs down-loaded from CTD8. We remove inferred relations(those without an associated PubMed ID) and con-sider only human curated relationships. Somechemical-gene entity pairs were associated withmultiple relation types in the same document. Weconsider each of these relation types as a separatepositive example.The chemical-gene relation data contains over

100 types organized in a shallow hierarchy. Manyof these types are extremely infrequent, so we mapall relations to the highest parent in the hierar-chy, resulting in 13 relation types. Most of thesechemical-gene relations have an increase and de-crease version such as increase_expression and de-crease_expression. In some cases, there is also anaffects relation (affects_expression) which is usedwhen the directionality is unknown. If the affectsversion is more common, we map decrease and in-crease to affects. If affects is less common, we dropthe affects examples and keep the increase and de-crease examples as distinct relations, resulting inthe final set of 10 chemical-gene relation types.

3.3.2 ResultsIn Table 7 we list precision, recall and F1 achievedby our model on the CTD dataset, both overall andby relation type. Our model predicts each of therelation types effectively, with higher performanceon relations with more support.

In Table 8 we see that our sub-word BPE modelout-performs the model using the Genia tokenizer(Kulick et al., 2012) even though our vocabularysize is one-fifth as large. We see a 1.7 F1 pointboost in predicting Pubtator NER labels for BPE.This could be explained by the increased out-of-vocabulary (OOV) rate for named entities. Word

7We include scripts to generate the unfiltered set ofdata as well to encourage future research

8http://ctdbase.org/downloads/

Train Dev TestTotal 120k 15k 15kChemical / Diseasemarker/mechanism 41,562 5,126 5,167therapeutic 24,151 2,929 3,059Gene / Diseasemarker/mechanism 5,930 825 819therapeutic 560 77 75Chemical / Geneincrease_expression 15,851 1,958 2,137increase_metabolic_proc 5,986 740 638decrease_expression 5,870 698 783increase_activity 4,154 467 497affects_response 3,834 475 508decrease_activity 3,124 396 434affects_transport 3,009 333 361increase_reaction 2,881 367 353decrease_reaction 2,221 247 269decrease_metabolic_proc 798 100 120

Table 6: Data statistics for the new CTD datasetbroken down by relation type. The first columnlists relation types separated by the types of theentities. Columns 2–4 show the number of positiveexamples of that relation type.

training data has 3.01 percent OOV rate for tokenswith an entity. The byte pair-encoded data has anOOV rate of 2.48 percent. Note that in both theword-tokenized and byte pair-tokenized data, wereplace tokens that occur less than five times witha learned UNK token.Figure 2 depicts the model’s performance on re-

lation extraction as a function of distance betweenentities. For example, the blue bar depicts perfor-mance when removing all entity pair candidates(positive and negative) whose closest mentions aremore than 11 tokens apart. We consider remov-ing entity pair candidates with distances of 11, 25,50, 100 and 500 (the maximum document length).The average sentence length is 22 tokens. We seethat the model is not simply relying on short rangerelationships, but is leveraging information aboutdistant entity pairs, with accuracy increasing as themaximum distance considered increases. Note thatall results are taken from the same model trainedon the full unfiltered training set.

4 Related workRelation extraction is a heavily studied area in theNLP community. Most work focuses on news andweb data (Doddington et al., 2004; Riedel et al.,2010; Hendrickx et al., 2009).9 Recent neural net-work approaches to relation extraction have focusedon CNNs (dos Santos et al., 2015; Zeng et al., 2015)

9And TAC KBP: https://tac.nist.gov

P R F1TotalMicro F1 44.8 50.2 47.3Macro F1 34.0 29.8 31.7Chemical / Diseasemarker/mechanism 46.2 57.9 51.3therapeutic 55.7 67.1 60.8Gene / Diseasemarker/mechanism 42.2 44.4 43.0therapeutic 52.6 10.1 15.8Chemical / Geneincreases_expression 39.7 48.0 43.3increases_metabolic_proc 26.3 35.5 29.9decreases_expression 34.4 32.9 33.4increases_activity 24.5 24.7 24.4affects_response 40.9 35.5 37.4decreases_activity 30.8 19.4 23.5affects_transport 28.7 23.8 25.8increases_reaction 12.8 5.6 7.4decreases_reaction 12.3 5.7 7.4decreases_metabolic_proc 28.9 7.0 11.0

Table 7: BRAN precision, recall and F1 results forthe full CTD dataset by relation type. The modelis optimized for micro F1 score across all types.

Model P R F1Relation extractionWords 44.9 48.8 46.7 ± 0.39BPE 44.8 50.2 47.3 ± 0.19NERWords 91.0 90.7 90.9 ± 0.13BPE 91.5 93.6 92.6 ± 0.12

Table 8: Precision, recall, and F1 results for CTDnamed entity recognition and relation extraction,comparing BPE to word-level tokenization.

or LSTMs (Miwa and Bansal, 2016; Verga et al.,2016a; Zhou et al., 2016b) and replacing stage-wiseinformation extraction pipelines with a single end-to-end model (Miwa and Bansal, 2016; Ammaret al., 2017; Li et al., 2017). These models allconsider mention pairs separately.

There is also a considerable body of work specifi-cally geared towards supervised biological relationextraction including protein-protein (Pyysalo et al.,2007; Poon et al., 2014; Mallory et al., 2015), drug-drug (Segura-Bedmar et al., 2013), and chemical-disease (Gurulingappa et al., 2012; Li et al., 2016a)interactions, and more complex events (Kim et al.,2008; Riedel et al., 2011). Our work focuses on mod-eling relations between chemicals, diseases, genesand proteins, where available annotation is oftenat the document- or abstract-level, rather than thesentence level.Some previous work exists on cross-sentence

chem_gene chem_disease gene_disease allDataset

0.0

0.1

0.2

0.3

0.4

0.5

0.6

F1 S

core

112550100500

Figure 2: Performance on the CTD dataset whenrestricting candidate entity pairs by distance. Thex-axis shows the coarse-grained relation type. They-axis shows F1 score. Different colors denote max-imum distance cutoffs.

relation extraction. Swampillai and Stevenson(2011) and Quirk and Poon (2017) consider featur-ized classifiers over cross-sentence syntactic parses.Most similar to our work is that of Peng et al.(2017), which uses a variant of an LSTM to encodedocument-level syntactic parse trees. Our workdiffers in three key ways. First, we operate overraw tokens negating the need for part-of-speechor syntactic parse features which can lead to cas-cading errors. We also use a feed-forward neuralarchitecture which encodes long sequences far moreefficiently compared to the graph LSTM network ofPeng et al. (2017). Finally, our model considers allmention pairs simultaneously rather than a singlemention pair at a time.We employ a bi-affine function to form pairwise

predictions between mentions. Such models havealso been used for knowledge graph link prediction(Nickel et al., 2011; Li et al., 2016b), with variationssuch as restricting the bilinear relation matrix tobe diagonal (Yang et al., 2015) or diagonal andcomplex (Trouillon et al., 2016). Our model issimilar to recent approaches to graph-based depen-dency parsing, where bilinear parameters are usedto score head-dependent compatibility (Kiperwasserand Goldberg, 2016; Dozat and Manning, 2017).

5 Conclusion

We present a bi-affine relation attention networkthat simultaneously scores all mention pairs withina document. Our model performs well on threedatasets, including two standard benchmark biolog-ical relation extraction datasets and a new, largeand high-quality dataset introduced in this work.Our model out-performs the previous state of theart on the Biocreative V CDR dataset despite us-ing no additional linguistic resources or mention

pair-specific features.Our current model predicts only into a fixed

schema of relations given by the data. However,this could be ameliorated by integrating our modelinto open relation extraction architectures suchas Universal Schema (Riedel et al., 2013; Vergaet al., 2016b). Our model also lends itself to otherpairwise scoring tasks such as hypernym prediction,co-reference resolution, and entity resolution. Wewill investigate these directions in future work.

AcknowledgmentsWe thank Ofer Shai and the Chan Zuckerberg Initia-tive / Meta data science team for helpful discussions.We also thank Timothy Dozat and Kyubyong Parkfor releasing their code. This material is basedupon work supported in part by the Center forData Science and the Center for Intelligent Infor-mation Retrieval, in part by the Chan ZuckerbergInitiative under the project “Scientific KnowledgeBase Construction, in part by the National Sci-ence Foundation under Grant No. DMR-1534431and IIS-1514053. ES is supported by an IBM PhDFellowship Award. Any opinions, findings and con-clusions or recommendations expressed in this mate-rial are those of the authors and do not necessarilyreflect those of the sponsor.

ReferencesMartín Abadi, Ashish Agarwal, Paul Barham,Eugene Brevdo, Zhifeng Chen, Craig Citro,Greg S. Corrado, Andy Davis, Jeffrey Dean,Matthieu Devin, Sanjay Ghemawat, Ian Good-fellow, Andrew Harp, Geoffrey Irving, MichaelIsard, Yangqing Jia, Rafal Jozefowicz, LukaszKaiser, Manjunath Kudlur, Josh Levenberg, DanMané, Rajat Monga, Sherry Moore, Derek Mur-ray, Chris Olah, Mike Schuster, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar,Paul Tucker, Vincent Vanhoucke, Vijay Vasude-van, Fernanda Viégas, Oriol Vinyals, Pete War-den, Martin Wattenberg, Martin Wicke, YuanYu, and Xiaoqiang Zheng. 2015. TensorFlow:Large-scale machine learning on heterogeneoussystems. Software available from tensorflow.org.http://tensorflow.org/.

Waleed Ammar, Matthew E. Peters, Chandra Bha-gavatula, and Russell Power. 2017. The ai2 sys-tem at semeval-2017 task 10 (scienceie): semi-supervised end-to-end entity and relation extrac-tion. nucleus 2(e2):e2.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geof-frey E Hinton. 2016. Layer normalization. arXivpreprint arXiv:1607.06450 .

Rajarshi Das, Arvind Neelakantan, David Belanger,and Andrew McCallum. 2017. Chains of reason-ing over entities, relations, and text using recur-rent neural networks. In Proceedings of the 15th

Conference of the European Chapter of the Asso-ciation for Computational Linguistics: Volume 1,Long Papers . Association for Computational Lin-guistics, Valencia, Spain, pages 132–141. http://www.aclweb.org/anthology/E17-1013.

Allan Peter Davis, Cynthia G Murphy, Cyn-thia A Saraceni-Richards, Michael C Rosenstein,Thomas C Wiegers, and Carolyn J Mattingly.2008. Comparative toxicogenomics database: aknowledgebase and discovery tool for chemical–gene–disease networks. Nucleic acids research37(suppl_1):D786–D792.

George Doddington, Alexis Mitchell, Mark Przy-bocki, Lance Ramshaw, Stephanie Strassel, andRalph Weischedel. 2004. The automatic contentextraction (ace) program tasks, data, and evalu-ation. In Proceedings of the Fourth InternationalConference on Language Resources and Evalua-tion.

Cicero dos Santos, Bing Xiang, and Bowen Zhou.2015. Classifying relations by ranking with con-volutional neural networks. In Proceedings of the53rd Annual Meeting of the Association for Com-putational Linguistics and the 7th InternationalJoint Conference on Natural Language Process-ing (Volume 1: Long Papers). Association forComputational Linguistics, Beijing, China, pages626–634. http://www.aclweb.org/anthology/P15-1061.

Timothy Dozat and Christopher D Manning. 2017.Deep biaffine attention for neural dependencyparsing. 5th International Conference on Learn-ing Representations .

Philip Gage. 1994. A new algorithm for data com-pression. The C Users Journal 12(2):23–38.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio.2011. Deep sparse rectifier neural networks. InProceedings of the Fourteenth International Con-ference on Artificial Intelligence and Statistics.pages 315–323.

Jinghang Gu, Longhua Qian, and Guodong Zhou.2016. Chemical-induced disease relation extrac-tion with various linguistic features. Database2016.

Jinghang Gu, Fuqing Sun, Longhua Qian, andGuodong Zhou. 2017. Chemical-induced diseaserelation extraction via convolutional neural net-work. Database 2017.

Harsha Gurulingappa, Abdul Mateen Rajput, An-gus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development ofa benchmark corpus to support the automaticextraction of drug-related adverse effects frommedical case reports. Journal of biomedical in-formatics 45(5):885–892.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, andJian Sun. 2016. Deep residual learning for imagerecognition. In Proceedings of the IEEE confer-ence on computer vision and pattern recognition.pages 770–778.

Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva,Preslav Nakov, Diarmuid Ó Séaghdha, SebastianPadó, Marco Pennacchiotti, Lorenza Romano,and Stan Szpakowicz. 2009. Semeval-2010 task8: Multi-way classification of semantic relationsbetween pairs of nominals. In Proceedings ofthe Workshop on Semantic Evaluations: RecentAchievements and Future Directions . Associationfor Computational Linguistics, pages 94–99.

Sepp Hochreiter. 1998. The vanishing gradientproblem during learning recurrent neural netsand problem solutions. International Journalof Uncertainty, Fuzziness and Knowledge-BasedSystems 6(2):107–116.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.Long short-term memory. Neural computation9(8):1735–1780.

Jin-Dong Kim, Tomoko Ohta, and Jun’ichi Tsujii.2008. Corpus annotation for mining biomedi-cal events from literature. BMC bioinformatics9(1):10.

Diederik Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In 3rd Inter-national Conference for Learning Representations(ICLR). San Diego, California, USA.

Eliyahu Kiperwasser and Yoav Goldberg. 2016. Sim-ple and accurate dependency parsing using bidi-rectional lstm feature representations. Trans-actions of the Association for ComputationalLinguistics 4:313–327. https://transacl.org/ojs/index.php/tacl/article/view/885.

Martin Krallinger, Obdulia Rabal, Saber A.Akhondi, Martín Pérez Pérez, Jesús Santa-maría, Pérez Gael Rodríguez, Georgios Tsatsa-ronis, Ander Intxaurrondo, José Antonio López,Umesh Nandal, Erin Van Buel, Akileshwari Chan-drasekhar, Marleen Rodenburg, Astrid Laegreid,Marius Doornenbal, Julen Oyarzabal, AnaliaLourenço, and Alfonso Valencia. 2017. Overviewof the biocreative vi chemical-protein interactiontrack. Proceedings of the BioCreative VI Work-shop page 140.

Martin Krallinger, Obdulia Rabal, Florian Leit-ner, Miguel Vazquez, David Salgado, ZhiyongLu, Robert Leaman, Yanan Lu, Donghong Ji,Daniel M Lowe, et al. 2015. The chemdner cor-pus of chemicals and drugs and its annotationprinciples. Journal of cheminformatics 7(S1):S2.

Seth Kulick, Ann Bies, Mark Liberman, Mark Man-del, Scott Winters, and Pete White. 2012. In-tegrated annotation for biomedical informationextraction. HLT/NAACL Workshop: Biolink .

Fei Li, Meishan Zhang, Guohong Fu, and DonghongJi. 2017. A neural joint model for entity andrelation extraction from biomedical text. BMCbioinformatics 18(1):198.

Jiao Li, Yueping Sun, Robin J Johnson, DanielaSciaky, Chih-Hsuan Wei, Robert Leaman, Al-lan Peter Davis, Carolyn J Mattingly, Thomas CWiegers, and Zhiyong Lu. 2016a. Biocreative vcdr task corpus: a resource for chemical diseaserelation extraction. Database 2016.

Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gim-pel. 2016b. Commonsense knowledge base com-pletion. In Proceedings of the 54th AnnualMeeting of the Association for ComputationalLinguistics (Volume 1: Long Papers). Associa-tion for Computational Linguistics, Berlin, Ger-many, pages 1445–1455. http://www.aclweb.org/anthology/P16-1137.

Yankai Lin, Shiqi Shen, Zhiyuan Liu, HuanboLuan, and Maosong Sun. 2016. Neural rela-tion extraction with selective attention over in-stances. In Proceedings of the 54th AnnualMeeting of the Association for ComputationalLinguistics (Volume 1: Long Papers). Associa-tion for Computational Linguistics, Berlin, Ger-many, pages 2124–2133. http://www.aclweb.org/anthology/P16-1200.

Sijia Liu, Feichen Shen, Yanshan Wang, Ma-jid Rastegar-Mojarad, Ravikumar KomandurElayavilli, Vipin Chaundary, and Hongfang Liu.2017. Attention-based neural networks for chem-ical protein relation extraction. Proceedings ofthe BioCreative VI Workshop .

Emily K Mallory, Ce Zhang, Christopher Ré, andRuss B Altman. 2015. Large-scale extraction ofgene interactions from full-text literature usingdeepdive. Bioinformatics 32(1):106–113.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jef-frey Dean. 2013. Efficient estimation of wordrepresentations in vector space. arXiv preprintarXiv:1301.3781 .

Mike Mintz, Steven Bills, Rion Snow, and DanielJurafsky. 2009. Distant supervision for relationextraction without labeled data. In Proceed-ings of the Joint Conference of the 47th AnnualMeeting of the ACL and the 4th InternationalJoint Conference on Natural Language Process-ing of the AFNLP . Association for Computa-tional Linguistics, Suntec, Singapore, pages 1003–1011. http://www.aclweb.org/anthology/P/P09/P09-1113.

Makoto Miwa and Mohit Bansal. 2016. End-to-endrelation extraction using lstms on sequences andtree structures. In Proceedings of the 54th AnnualMeeting of the Association for ComputationalLinguistics (Volume 1: Long Papers). Associa-tion for Computational Linguistics, Berlin, Ger-many, pages 1105–1116. http://www.aclweb.org/anthology/P16-1105.

Arvind Neelakantan, Luke Vilnis, Quoc V Le, IlyaSutskever, Lukasz Kaiser, Karol Kurach, andJames Martens. 2015. Adding gradient noiseimproves learning for very deep networks. arXivpreprint arXiv:1511.06807 .

Maximilian Nickel, Volker Tresp, and Hans-PeterKriegel. 2011. A three-way model for collectivelearning on multi-relational data. In Proceedingsof the 28th international conference on machinelearning (ICML-11). Bellevue, Washington, USA,pages 809–816.

Nanyun Peng, Hoifung Poon, Chris Quirk, KristinaToutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graphlstms. Transactions of the Association for Com-putational Linguistics 5:101–115.

Yifan Peng, Chih-Hsuan Wei, and Zhiyong Lu. 2016.Improving chemical disease relation extractionwith rich features and weakly labeled data. Jour-nal of cheminformatics 8(1):53.

Hoifung Poon, Kristina Toutanova, and Chris Quirk.2014. Distant supervision for cancer pathwayextraction from text. In Pacific Symposium onBiocomputing Co-Chairs. pages 120–131.

Sampo Pyysalo, Filip Ginter, Juho Heimonen, JariBjörne, Jorma Boberg, Jouni Järvinen, and TapioSalakoski. 2007. Bioinfer: a corpus for informa-tion extraction in the biomedical domain. BMCbioinformatics 8(1):50.

Chris Quirk and Hoifung Poon. 2017. Distant super-vision for relation extraction beyond the sentenceboundary. In Proceedings of the 15th Conferenceof the European Chapter of the Association forComputational Linguistics: Volume 1, Long Pa-pers. Association for Computational Linguistics,Valencia, Spain, pages 1171–1182.

Sebastian Riedel, David McClosky, Mihai Surdeanu,Andrew McCallum, and Christopher D. Man-ning. 2011. Model combination for event extrac-tion in bionlp 2011. In Proceedings of BioNLPShared Task 2011 Workshop. Association forComputational Linguistics, Portland, Oregon,USA, pages 51–55. http://www.aclweb.org/anthology/W11-1808.

Sebastian Riedel, Limin Yao, and Andrew McCal-lum. 2010. Modeling relations and their men-tions without labeled text. Machine learning andknowledge discovery in databases pages 148–163.

Sebastian Riedel, Limin Yao, Andrew McCallum,and Benjamin M Marlin. 2013. Relation ex-traction with matrix factorization and universalschemas. In Proceedings of NAACL-HLT . pages74–84.

Isabel Segura-Bedmar, Paloma Martínez, andMaría Herrero Zazo. 2013. Semeval-2013 task9 : Extraction of drug-drug interactions from

biomedical texts (ddiextraction 2013). In Sec-ond Joint Conference on Lexical and Computa-tional Semantics (*SEM), Volume 2: Proceedingsof the Seventh International Workshop on Se-mantic Evaluation (SemEval 2013). Associationfor Computational Linguistics, Atlanta, Georgia,USA, pages 341–350. http://www.aclweb.org/anthology/S13-2056.

Rico Sennrich, Barry Haddow, and AlexandraBirch. 2015. Neural machine translation ofrare words with subword units. arXiv preprintarXiv:1508.07909 .

Nitish Srivastava, Geoffrey E Hinton, AlexKrizhevsky, Ilya Sutskever, and Ruslan Salakhut-dinov. 2014. Dropout: a simple way to preventneural networks from overfitting. Journal of ma-chine learning research 15(1):1929–1958.

Mihai Surdeanu, Julie Tibshirani, Ramesh Nallap-ati, and Christopher D. Manning. 2012. Multi-instance multi-label learning for relation extrac-tion. In Proceedings of the 2012 Joint Confer-ence on Empirical Methods in Natural LanguageProcessing and Computational Natural LanguageLearning . Association for Computational Linguis-tics, Jeju Island, Korea, pages 455–465. http://www.aclweb.org/anthology/D12-1042.

Kumutha Swampillai and Mark Stevenson. 2011.Extracting relations within and across sentences.In Proceedings of the International ConferenceRecent Advances in Natural Language Process-ing 2011 . RANLP 2011 Organising Committee,Hissar, Bulgaria, pages 25–32. http://aclweb.org/anthology/R11-1004.

Théo Trouillon, Johannes Welbl, Sebastian Riedel,Éric Gaussier, and Guillaume Bouchard. 2016.Complex embeddings for simple link prediction.In International Conference on Machine Learn-ing . pages 2071–2080.

Ashish Vaswani, Noam Shazeer, Niki Parmar,Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Lukasz Kaiser, and Illia Polosukhin. 2017.Attention is all you need. arXiv preprintarXiv:1706.03762 .

Patrick Verga, David Belanger, Emma Strubell,Benjamin Roth, and Andrew McCallum. 2016a.Multilingual relation extraction using composi-tional universal schema. In Proceedings of the2016 Conference of the North American Chapterof the Association for Computational Linguistics:Human Language Technologies. Association forComputational Linguistics, San Diego, Califor-nia, pages 886–896. http://www.aclweb.org/anthology/N16-1103.

Patrick Verga, David Belanger, Emma Strubell,Benjamin Roth, and Andrew McCallum. 2016b.Multilingual relation extraction using compo-sitional universal schema. In Proceedings ofNAACL-HLT . pages 886–896.

Patrick Verga and Andrew McCallum. 2016. Row-less universal schema. In Proceedings of the 5thWorkshop on Automated Knowledge Base Con-struction. Association for Computational Lin-guistics, San Diego, CA, pages 63–68. http://www.aclweb.org/anthology/W16-1312.

Chih-Hsuan Wei, Hung-Yu Kao, and Zhiyong Lu.2013. Pubtator: a web-based text mining toolfor assisting biocuration. Nucleic Acids Research41. https://doi.org/10.1093/nar/gkt441.

Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Al-lan Peter Davis, Carolyn J Mattingly, Jiao Li,Thomas C Wiegers, and Zhiyong Lu. 2016. As-sessing the state of the art in biomedical rela-tion extraction: overview of the biocreative vchemical-disease relation (cdr) task. Database2016.

Yadollah Yaghoobzadeh, Heike Adel, and HinrichSchütze. 2017. Noise mitigation for neural en-tity typing and relation extraction. In Proceed-ings of the 15th Conference of the EuropeanChapter of the Association for ComputationalLinguistics: Volume 1, Long Papers. Associ-ation for Computational Linguistics, Valencia,Spain, pages 1183–1194. http://www.aclweb.org/anthology/E17-1111.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jian-feng Gao, and Li Deng. 2015. Embedding enti-ties and relations for learning and inference inknowledge bases. In 3rd International Confer-ence for Learning Representations (ICLR). SanDiego, California, USA.

Daojian Zeng, Kang Liu, Yubo Chen, and JunZhao. 2015. Distant supervision for relation ex-traction via piecewise convolutional neural net-works. In Proceedings of the 2015 Conferenceon Empirical Methods in Natural Language Pro-cessing . Association for Computational Linguis-tics, Lisbon, Portugal, pages 1753–1762. http://aclweb.org/anthology/D15-1203.

Huiwei Zhou, Huijie Deng, Long Chen, YunlongYang, Chen Jia, and Degen Huang. 2016a. Ex-ploiting syntactic and semantics information forchemical–disease relation extraction. Database2016.

Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi,Bingchen Li, Hongwei Hao, and Bo Xu. 2016b.Attention-based bidirectional long short-termmemory networks for relation classification. InProceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume2: Short Papers). Association for ComputationalLinguistics, Berlin, Germany, pages 207–212.http://anthology.aclweb.org/P16-2034.

A Implementation Details

The model is implemented in Tensorflow (Abadiet al., 2015) and trained on a single TitanX gpu.The number of transformer block repeats is B = 2 .We optimize the model using Adam (Kingma andBa, 2015) with best parameters chosen for ε, β1,β2 chosen from the development set. The learningrate is set to 0.0005 and batch size 32. In all of ourexperiments we set the number of attention headsto h = 4.

We clip the gradients to norm 10 and apply noiseto the gradients (Neelakantan et al., 2015). Wetune the decision threshold for each relation typeseparately and perform early stopping on the devel-opment set. We apply dropout (Srivastava et al.,2014) to the input layer randomly replacing wordswith a special UNK token with keep probability .85.We additionally apply dropout to the input T (wordembedding + position embedding), interior layers,and final state. At each step, we randomly samplea positive or negative (NULL class) minibatch withprobability 0.5.

A.1 Chemical Disease Relations Dataset

Token embeddings are pre-trained using skipgram(Mikolov et al., 2013) over a random subset of 10%of all PubMed abstracts with window size 10 and20 negative samples. We merge the train and devel-opment sets and randomly take 850 abstracts fortraining and 150 for early stopping. Our reportedresults are averaged over 10 runs and using differentsplits. All baselines train on both the train anddevelopment set. Models took between 4 and 8hours to train.ε was set to 1e-4, β1 to .1, and β2 to 0.9. Gradi-

ent noise η = .1. Dropout was applied to the wordembeddings with keep probability 0.85, internal lay-ers with 0.95 and final bilinear projection with 0.35for the standard CRD dataset experiments. Whenadding the additional weakly labeled data: wordembeddings with keep probability 0.95, internallayers with 0.95 and final bilinear projection with0.5.

A.2 Chemical Protein Relations Dataset

We construct our byte-pair encoding vocabularyusing a budget of 7500. The dataset contains an-notations for a larger set of relation types than areused in evaluation. We train on only the relationtypes in the evaluation set and set the remainingtypes to the Null relation. The embedding dimen-sion is set to 200 and all embeddings are randomlyinitialized. ε was set to 1e-8, β1 to .1, and β2 to 0.9.Gradient noise η = 1.0. Dropout was applied to theword embeddings with keep probability 0.5, internallayers with 1.0 and final bilinear projection with0.85 for the standard CRD dataset experiments.

A.3 Full CTD DatasetWe tune separate decision boundaries for each re-lation type on the development set. For each pre-diction, the relation type with the maximum prob-ability is assigned. If the probability is below therelation specific threshold, the prediction is set toNULL. We use embedding dimension 128 with allembeddings randomly initialized. Our byte pairencoding vocabulary is constructed with a budgetof 50,000. Models took 1 to 2 days to train.ε was set to 1e-4, β1 to .1, and β2 to 0.9. Gradi-

ent noise η = .1.Dropout was applied to the wordembeddings with keep probability 0.95, internallayers with 0.95 and final bilinear projection with0.5