Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo), pages 47–55Hong Kong, China, November 3, 2019. c©2019 Association for Computational Linguistics

https://doi.org/10.18653/v1/P17

47

BERT is Not an Interlingua and the Bias of Tokenization

Jasdeep Singh1, Bryan McCann2, Caiming Xiong2, Richard Socher2

Stanford University1, Salesforce Research2

jasdeep@cs.stanford.edu {bmccann,cxiong,rsocher}@salesforce.com

Abstract

Multilingual transfer learning can benefit bothhigh- and low-resource languages, but thesource of these improvements is not wellunderstood. Cananical Correlation Analysis(CCA) of the internal representations of a pre-trained, multilingual BERT model reveals thatthe model partitions representations for eachlanguage rather than using a common, shared,interlingual space. This effect is magnified atdeeper layers, suggesting that the model doesnot progressively abstract semantic contentwhile disregarding languages. Hierarchicalclustering based on the CCA similarity scoresbetween languages reveals a tree structure thatmirrors the phylogenetic trees hand-designedby linguists. The subword tokenization em-ployed by BERT provides a stronger bias to-wards such structure than character- and word-level tokenizations. We release a subset ofthe XNLI dataset translated into an additional14 languages at https://www.github.com/salesforce/xnli_extension toassist further research into multilingual repre-sentations.

1 Introduction

Natural language processing (NLP) in multilin-gual settings often relies on transfer learning be-tween high- and low-resource languages. Wordembeddings trained with the Word2Vec (Mikolovet al., 2013b) or GloVe (Pennington et al., 2014)algorithms are trained with large amounts of unsu-pervised data and transferred to downstream tasks-specific architectures in order to improve perfor-mance. Multilingual word embeddings have beentrained with varying levels of supervision. Paral-lel corpora can be leveraged when data is avail-able (Gouws et al., 2015; Luong et al., 2015),monolingual embeddings can be learned sepa-rately (Klementiev et al., 2012; Zou et al., 2013;Hermann and Blunsom, 2014) and then aligned

using dictionaries between languages (Mikolovet al., 2013a; Faruqui and Dyer, 2014), and cross-lingual embeddings can be learned jointly throughentirely unsupervised methods (Conneau et al.,2017; Artetxe et al., 2018).

Contextualized word embeddings like CoVe,ElMo, and BERT (McCann et al., 2017; Peterset al., 2018; Devlin et al., 2018) improve a widevariety of natural language tasks (Wang et al.,2018; Rajpurkar et al., 2016; Socher et al., 2013;Conneau et al., 2018). A multilingual versionof BERT trained on over 100 languages achievedstate-of-the-art performance across a wide rangeof languages as well. Performance for low-resource languages has been further improvedby additionally leveraging parallel data (Lam-ple and Conneau, 2019) and leveraging machinetranslation systems for cross-lingual regulariza-tion (Singh et al., 2019).

Prior work in zero-shot machine translation hasinvestigated the extent to which multilingual neu-ral machine translation systems trained with ashared subword vocabulary Johnson et al. (2017);Kudugunta et al. (2019) learn a form of interlin-gua, a common representational space for seman-tically similar text across languages. We aim toextend this study to language models pretrainedwith multilingual data in order to investigate theextent to which the resulting contextualized wordembeddings represent an interlingua.

Canonical correlation analysis (CCA) is a clas-sical tool from multivariate statistics (Hotelling,1992) that investigates the relationships betweentwo sets of random variables. Singular value andprojection weighted variants of CCA allow foranalysis of representations of the same data pointsfrom different models in a way that is invariant toaffine transformations (Raghu et al., 2017; Mor-cos et al., 2018), which makes them particularlysuitable for analyzing neural networks. They have

48

Figure 1: Agglomerative clustering of Languages based on the PWCCA similarity between their represenations,generated from layer 6 of a pretrained multilingual uncased BERT.

been used to explore learning dynamics and rep-resentational similarity in computer vision (Mor-cos et al., 2018) and natural language process-ing (Saphra and Lopez, 2018; Kudugunta et al.,2019).

We analyze multilingual BERT using pro-jection weighted canonical correlation analysis(PWCCA) between representations from semanti-cally similar text sequences in mulitple languages.We find that the representations from multilingualBERT can be partitioned using PWCCA similarityscores to reflect the linguistic and evolutionary re-lationships between languages. This suggests thatBERT does not represent semantically similar datapoints nearer to each other in a common space aswould be expected of an interlingua. Rather, rep-resentations in this space are primarily organizedaround features that respect the natural differencesand similarities between languages. Our analysisshows that the choice of tokenization can heav-ily influence this space. Subword tokenization, incontrast to word and character level tokenization,provides a strong bias towards discovering theselinguistic and evolutionary relationships betweenlanguages. As part of our experiments, we trans-lated a subset of the XNLI data set into an addi-tional 14 languages, which we publicly release toassist further research into multilingual represen-tations.

2 Background and Related Work

2.1 Natural Language Inference and XNLI

The Multi-Genre Natural Language Inference(MultiNLI) corpus (Williams et al., 2017) uses

data from ten distinct genres of English languagefor the the task of natural language inference (pre-diction of whether the relationship between twosentences represents entailment, contradiction, orneither). XNLI (Conneau et al., 2018) is anevaluation set grounded in MultiNLI for cross-lingual understanding (XLU) in 15 different lan-guages that include low-resource languages suchas Swahili and Urdu. XNLI serves as the pri-mary testbed for bench marking multilingual un-derstanding. We extend a subset of XNLI to anadditional 14 languages for our analysis.

2.2 CCA

Deep network analyses techniques focusing onthe weights of a network are unable to distin-guish between several invariances such as permu-tation and scaling. CCA (Hotelling, 1992) andvariants that use Singular Value Decomposition(SVD) (Raghu et al., 2017) or projection weight-ing (Morcos et al., 2018) are apt for analyzing theactivations of neural networks because they pro-vide a similarity metric that is invariant to permu-tations and scaling of neurons. These methodsalso allow for comparisons between representa-tions for the same data points from different neuralnetworks where there is no naive alignment fromneurons of one network to another.

Given a dataset X = {x1, . . . xn}, let L1 ∈Rm1×n and L2 ∈ Rm2×n be two sets of neu-rons. Often these sets correspond to layers in neu-ral networks. CCA transforms L1 and L2 to a>1 L1

and b>1 L2 respectively where the pair of canoni-cal variables {a1, b1} is found by maximizing thecorrelation between the transformed subspaces:

49

Add & N

orm

Feed Forward

Add & N

orm

Multi-head A

ttention

Add & N

orm

Feed Forward

Add & N

orm

Multi-head A

ttention

Add & N

orm

Feed Forward

Add & N

orm

Multi-head A

ttention

Add & N

orm

Feed Forward

Add & N

orm

Multi-head A

ttention

Translation System

[CLS] One of our number will carry out your instructions minutely. [SEP] A

member of my team will execute your orders with immense precision.

[CLS] Eine unserer Nummern wird Ihre Anweisungen minutiös ausführe. [SEP]

Ein Mitglied meines Teams wird Ihre Aufträge mit immenser Präzision

ausführen.

[CLS] L’un de nos numéros exécutera minutieusement vos instructions. [SEP]

Un membre de mon équipe exécutera vos ordres avec une précision extrême.

CCA CCA CCA

Figure 2: How CCA is used to compare the representations of different languages at different layers in BERT.

ρ1 = maxa∈Rm1 ,b∈Rm2

〈a>L1, b>L2〉

‖a>L1‖ · ‖b>L1‖

Given the set {a1, b1}, we can find another pair{a2, b2}:

ρ2 = maxa2∈Rm1 ,b2∈Rm2

〈a>2 L1, b>2 L2〉

‖a>2 L1‖ · ‖b>2 L1‖

under the constraints that 〈a2, a1〉 = 0 and〈b2, b1〉 = 0. This continues until {am′ , bm′}and ρm′ have been found such that m′ =min (m1,m2).

The average of {ρ1, ... ρm′} is often used as anoverall similarity measure, as in related work ex-ploring multilingual representations in neural ma-chine translation systems (Kudugunta et al., 2019)and language models (Saphra and Lopez, 2018).Morcos et al. (2018) show that in studying re-current and convolutional networks, replacing aweighted average leads to a more robust measureof similarity between two sets of activations. Forthe rest of the paper we use this PWCCA measureto determine the similarity of two sets of activa-tions. The measure lies between [1,0] with 1 be-ing identical and 0 being no similarity between therepresentations.

CCA is typically employed to compare repre-sentations for the same inputs for different modelsor layers (Raghu et al., 2017; Morcos et al., 2018).We also use CCA to compare representations from

the same neural network when fed two translatedversions of the same input (Figure 2).

3 Experiments and Discussion

We use the uncased multilingual BERT model(Devlin et al., 2018) and the XNLI data set formost experiments. Multilingual BERT is pre-trained on the Wikipedia articles form 102 lan-guages and is 12-layers deep. The XNLI datasetconsists of the MultiNLI dataset translated into15 languages. The uncased multilingual BERTmodel does not contain tokenization or pretain-ing for Thai so we focus our analysis on the re-maining 14 languages. To provide a more ro-bust study of a broader variety of languages, wesupplement the XNLI data set by further translat-ing the first 15 thousand examples using googletranslate. This allows for analysis of representa-tions for 14 additional languages including Azer-baijani, Czech, Danish, Estonian, Finish, Hungar-ian, Kazakh, Latvian, Lithuanian, Dutch, Norwe-gian, Swedish, Ukrainian, and Uzbek.

Following the standard approach to usingBERT (Devlin et al., 2018), a [CLS] token isprepended to each example, which consists of apremise, a [SEP] token and a hypothesis. The[CLS] token has been pretrained to extract inter-sentence relationships between the sentences thatfollow it. When fine-tuned on XNLI, the final rep-resentations for the [CLS] token is used to predictthe relationship between sentences as either entail-ment, contradiction or neutral. This [CLS] token

50

can be thought of as a summary embedding forthe input as a whole. We analyze the activationsof this [CLS] token for the same XNLI examplesacross all available languages (Figure 2). Theserepresentations have 768 neurons computed over15 thousand datapoints.

For all fine-tuning experiments we use the hy-perparameters and optimization strategies recom-mended by (Devlin et al., 2018) unless otherwisespecified. We use a learning rate warm-up for 10%of training iterations and then linearly decay tozero. The batch size is 32 and the target learningrate after warm-up is 2e− 5.

3.1 Representations across languages are lesssimilar in the deeper layers of BERT

Figure 3 demonstrates that for all language com-binations tested, the summary representation (as-sociated with the [CLS] token) for semanticallysimilar inputs translated into multiple languagesis most similar at the shallower layers of BERT,close to the initial embeddings. The representa-tions steadily become more dissimilar in deeperlayers until the final layer. The jump in similarityin the final layer can be explained by the commonclassification layer that contains only three classes.In order to finally choose an output class, the net-work must project towards one of the three em-beddings associated with those classes (Liu et al.,2019).

The trend towards dissimilarity in deeper layerssuggests that contextualization in BERT is not aprocess of semantic abstraction as would be ex-pected of an interlingua. Though semantic fea-tures common to the multiple translations of theinput might also be extracted, the similarity be-tween representations is dominated by featuresthat differentiate them. BERT appears to preserveand refine features that separate the inputs, whichwe speculate are more closely related to syntacticand grammatical features of the input.

Representations at the shallower layers, closerto the subword embeddings, exhibit the highestdegree of similarity. This provides further evi-dence for how a subword vocabulary can effec-tively span a large space of languages.

3.2 Representations diverge with depth afterfine-tuning

In the previous set of experiments, BERT hadonly been pretrained on the unupservised maskedlanguage modeling objective. In that setting the

Figure 3: The similarity between representations of dif-ferent languages decreases deeper into a pretrained un-cased multilingual BERT model. Here we show thesimilarity between English and 5 other languages as afunction of model depth

Figure 4: The similarity between representations of dif-ferent languages decreases deeper into an uncased mul-tilingual BERT model finetuned on XNLI.

[CLS] token was trained to predict whether thesecond sentence followed the first. This does notalign well with the XNLI task in cases in whichthe hypothesis would not likely follow the premisein the corpora used for pretraining. To alleviateconcerns that this might influence the representa-tional similarity, we repeat the above experimentsafter fine-tuning BERT on several languages. Fig-ure 4 confirms that representations for semanti-cally similar inputs in different languages divergein PWCCA similarity in deeper layers of BERT.

3.3 Deeper layers change more dramaticallyduring fine-tuning

We also notice that during fine-tuning, the deep-est layers of BERT change the most according toPWCCA similarity . In these experiments, we usePWCCA in the more standard setting, in whichidentical inputs are provided to two different mod-els in order to get two set of neuron activations.

51

(a) (b)

(c) (d)

Figure 5: CCA experiments showing the finetuning behaviorof BERT

The two different models are different checkpointsof pretrained, multilingual BERT over the courseof fine-tuning. Figures 5a - 5d follow a similarstructure in which the PWCCA value is computedbetween all the layers of two models. The matri-ces are symmetric in expectation, but noise dur-ing optimization creates slight asymmetry. Fig-ure 5a shows a baseline case demonstrating that apretrained, multilingual BERT compared with it-self has a PWCCA value of 1 with strong diago-nal showing the identity between each layer. Theoff-diagonal entries show that layer-wise similar-ity depends on relative depth. This successive andgradual changing of representations is preciselythe behavior we expect from networks with resid-ual connections (Raghu et al., 2017).

We compare the pretrained multilingual BERTto a converged BERT fine-tuned on XNLI in Fig-ure 5b and 5c. We use representations for the[CL] token in Figure 5b and the first token in thepremise for Figure 5c. Both confirm that the func-tion of early layers remains more similar as thenetwork is fine-tuned. We find this trend to holdfor a wide range of tasks including SST and QNLIas well.

Figure 5d shows that deep pretrained networksalso converge bottom up during fine-tuning bycomparing the representations a quarter of theway through fine-tuning with those of a convergedmodel. Most of the remaining change in the rep-resentations between a quarter of the way through

Figure 6: PWCCA generated similarity matrix betweenlanguages.

training and convergence happens in the later lay-ers. Therefore the changes to middle layers we ob-serve in Figure 5b happen during the first quarterof training.

3.4 Phylogentic Tree of Language Evolution

Figures 3 and 4 show that the representationslearned for different languages diverge as we godeeper into the network, as opposed to convergingif the network were learning an interlingua or ashared representation space. However, the relativesimilarities between languages clearly varies fordifferent pairs and changes as a function of depth.To further investigate the internal relationships be-tween representations learned by BERT we cre-ate a similarity matrix using PWCCA between all28 languages for all 12 layers in BERT. For thePWCCA calculations we use the representationsof the [CLS] token to generate L1 and L2. Theresulting similarity matrix for Layer 1 is shown inFigure 6.

To visualize these relationships this matrix canbe converted to a phylogentic tree using a clus-ter algorithm. In Figure 1 we use unweighted pairgroup method with arithmetic mean (UPGMA),a simple agglomerative (bottom-up) hierarchicalclustering method (Sokal, 1958) to generate a phy-logentic tree from the representations from Layer6 of BERT. The generated phylogentic tree closelyresembles the language tree constructed by lin-guists to explain the relationships and evolutionof human languages. The details of the linguis-

52

tic evolutionary phylogentic tree of languages isstill debated and a tree model faces some limita-tions as not all evolutionary relationships are com-pletely hierarchical and it can not easily accountfor horizontal transmissions. However many ofthe commonly known relationships between lan-guages are embedded in BERT’s representations.We see that BERT’s space of its internal represen-tations is finely partitioned into families and sub-familes of languages.

Northern Germanic languages are clustered to-gether and the Western Germanic languages areclustered together before being combined togetherinto the pro-germanic family. Romantic languagesare clustered together before the Romantic andGermanic families are combined together. BERT’sinternal representation of English is in-betweenthat of the Germanic and Romantic sub-families.This captures evolution and structure of English,which is considered a Germanic language but bor-rows heavily from romantic and latin languages.By varying the layer at which the representationsare used to create the phylogentic tree, differentstructures emerge. Sometimes English is clusteredwith German before it is combined with the ro-mantic languages, but mostly BERT seems to clas-sify English as a romantic language.

For Layers 6 through 12 the trees generated arealmost or exactly like Figure 1. At these lay-ers, Azerbaijani, Turkish, Kazakh, and Uzbek aregrouped into the same family although these lan-guages span multiple scripts and have had their of-ficial scripts changed multiple times allowing forthe possible introduction of confounding differ-ences. Interestingly, at these same layers in Fig-ure 3, languages seem to diverge the most fromeach other. This seems to suggest that instead offinding a shared latent space for all of the lan-guages, as would be necessary for an interlingua,BERT is actually carefully partitioning its space ina fashion that linguistic and evolutionary relation-ships are preserved between languages. Trees gen-erated from Layers 1 through 5 seem to make moremistakes than those of later layers. We see thatthese trees end up failing to group Azerbaijani,Turkish, Kazakh, and Uzbek into the same family,often leaving Kazakh out which is written in Cyril-lic script. We can hypothesis that BERT’s inter-nal representations are more reliant on the identityof shared subwords earlier in the network as op-posed to later in the network. As a matter of fact,

if we use agglomerative clustering to construct atree from a matrix of subword overlap counts (Fig-ure 7a), we find that it almost exactly matches thetree constructed from BERT’s earlier layers.

It seems that BERT’s shared multilingual sub-word vocabulary (Mikolov et al., 2012; Sennrichet al., 2015) provides it with a strong bias towardswhat linguistic relationships exist between humanlanguages. Instead of then fusing the representa-tions of different languages into one shared repre-sentation during training, BERT actually succes-sively refines this partitioned space to better reflectthe linguistic relationships between languages athigher layers (Figure 1).

3.5 Tokenization Provides a Strong BiasTowards Knowledge of LinguisticRelationships Between Languages

We tokenize the first 15 thousand examples fromXNLI and our translated data using different to-kenization methods and compute the token over-lap between different languages, generating sim-ilarity matrices similar to the one shown in Fig-ure 6. From these matrices we perform agglomer-ative clustering of languages to generate phyloge-netic trees (Figure 7). These trees show how dif-ferent tokenization schemes can embed differentlinguistic biases into our models. We investigatesubword, word, and character level tokenization.The subword tokenization is done using BERT’slearned BPE vocabulary. The word level tokeniza-tion is achieved by simply tokenizing at spaces,and the character level tokenization is done us-ing Python’s native character level string split-ting. Figure 7a is generated from using subwords,and although is not as accurate as the tree gener-ated from BERT’s representation at layer 6 (Fig-ure 1), it is still non-trivially close to a linguisti-cally accurate depiction of human language evo-lution. We see that by using a shared subword vo-cabulary, multilingual BERT has a very strong biasto discover the linguistic relationships betweenlanguages. However, this bias is not as strong ifother forms of tokenization are used. Figures 7band 7c show the trees generated by word level andcharacter level tokenization respectively. We seethat word level tokenization splits the Romanticand Germanic languages into completely differenttrees, and that character level tokenization endsup combining all languages that share the Latinscript regardless of their true families. Perhaps the

53

(a) Agglomerative clustering of languages based on subword overlap, generated from using the BERT tokenizer to tokenize thefirst 15 thousand examples from XNLI and our translated data.

(b) Agglomerative clustering of languages based on word overlap, generated from splitting the first 15 thousand examples fromXNLI and our translated data on spaces.

(c) Agglomerative clustering of languages based on character overlap, generated from splitting the first 15 thousand examplesfrom XNLI and our translated.

Figure 7: Different agglomerative clusterings of languages based on subword, word, and character overlap. Wesee that different tokenization schemes used in NLP embed different linguistic biases into models.

54

ability of subwords to capture these linguistic re-lationships between languages has contributed totheir wide success in applications including ma-chine translation and language modeling.

4 Conclusion and Future Directions

While natural language processing systems oftenfocus on a single language, multilingual transferlearning has the potential to improve performance,especially for low-resource languages. Many pre-vious multilingual approaches claim to developshared representations of different languages. Re-cently, multilingual BERT and related modelstrained in an unsupervised fashion on monolingualcorpora from over 100 languages achieve state ofthe art performance on many tasks involving lowresource languages. Using Cononical CoreelationAnalysis (CCA) on the internal representations ofBERT, we find that it is not embedding differentlanguages into a shared space. Rather, at deeperlayers, BERT partitions the space to better reflectthe linguistic and evolutionary relationships be-tween languages. We also find that subword tok-enization, in contrast to word and character leveltokenization, provides a strong bias to discoverlinguistic and evolutionary relationships betweenlanguages.

ReferencesMikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.

A robust self-learning method for fully unsupervisedcross-lingual mappings of word embeddings. arXivpreprint arXiv:1805.06297.

Alexis Conneau, Guillaume Lample, Marc’AurelioRanzato, Ludovic Denoyer, and Herve Jegou. 2017.Word translation without parallel data. arXivpreprint arXiv:1710.04087.

Alexis Conneau, Guillaume Lample, Ruty Rinott, Ad-ina Williams, Samuel R Bowman, Holger Schwenk,and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprintarXiv:1809.05053.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. Bert: Pre-training of deepbidirectional transformers for language understand-ing. arXiv preprint arXiv:1810.04805.

Manaal Faruqui and Chris Dyer. 2014. Improving vec-tor space word representations using multilingualcorrelation. In Proceedings of the 14th Conferenceof the European Chapter of the Association for Com-putational Linguistics, pages 462–471.

Stephan Gouws, Yoshua Bengio, and Greg Corrado.2015. Bilbowa: Fast bilingual distributed represen-tations without word alignments.

Karl Moritz Hermann and Phil Blunsom. 2014. Multi-lingual models for compositional distributed seman-tics. arXiv preprint arXiv:1404.4641.

Harold Hotelling. 1992. Relations between two sets ofvariates. In Breakthroughs in statistics, pages 162–190. Springer.

Melvin Johnson, Mike Schuster, Quoc V Le, MaximKrikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,Fernanda Viegas, Martin Wattenberg, Greg Corrado,et al. 2017. Googles multilingual neural machinetranslation system: Enabling zero-shot translation.Transactions of the Association for ComputationalLinguistics, 5:339–351.

Alexandre Klementiev, Ivan Titov, and Binod Bhat-tarai. 2012. Inducing crosslingual distributed repre-sentations of words. Proceedings of COLING 2012,pages 1459–1474.

Sneha Reddy Kudugunta, Ankur Bapna, Isaac Caswell,Naveen Arivazhagan, and Orhan Firat. 2019. In-vestigating multilingual nmt representations at scale.arXiv preprint arXiv:1909.02197.

Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprintarXiv:1901.07291.

Nelson F Liu, Matt Gardner, Yonatan Belinkov,Matthew Peters, and Noah A Smith. 2019. Linguis-tic knowledge and transferability of contextual rep-resentations. arXiv preprint arXiv:1903.08855.

Thang Luong, Hieu Pham, and Christopher D Man-ning. 2015. Bilingual word representations withmonolingual quality in mind. In Proceedings of the1st Workshop on Vector Space Modeling for NaturalLanguage Processing, pages 151–159.

Bryan McCann, James Bradbury, Caiming Xiong, andRichard Socher. 2017. Learned in translation: Con-textualized word vectors. In Advances in Neural In-formation Processing Systems, pages 6294–6305.

Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013a.Exploiting similarities among languages for ma-chine translation. arXiv preprint arXiv:1309.4168.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013b. Distributed representa-tions of words and phrases and their compositional-ity. In Advances in neural information processingsystems, pages 3111–3119.

Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocky.2012. Subword language modeling with neu-ral networks. preprint (http://www. fit. vutbr.cz/imikolov/rnnlm/char. pdf), 8.

55

Ari Morcos, Maithra Raghu, and Samy Bengio. 2018.Insights on representational similarity in neural net-works with canonical correlation. In Advancesin Neural Information Processing Systems, pages5727–5736.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global vectors for wordrepresentation. In Proceedings of the 2014 confer-ence on empirical methods in natural language pro-cessing (EMNLP), pages 1532–1543.

Matthew E Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. arXiv preprint arXiv:1802.05365.

Maithra Raghu, Justin Gilmer, Jason Yosinski, andJascha Sohl-Dickstein. 2017. Svcca: Singular vec-tor canonical correlation analysis for deep learningdynamics and interpretability. In Advances in Neu-ral Information Processing Systems, pages 6076–6085.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. Squad: 100,000+ questionsfor machine comprehension of text. arXiv preprintarXiv:1606.05250.

Naomi Saphra and Adam Lopez. 2018. Understandinglearning dynamics of language models with svcca.arXiv preprint arXiv:1811.00225.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2015. Neural machine translation of rare words withsubword units. arXiv preprint arXiv:1508.07909.

Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar,Caiming Xiong, and Richard Socher. 2019. Xlda:Cross-lingual data augmentation for natural lan-guage inference and question answering. arXivpreprint arXiv:1905.11471.

Richard Socher, Alex Perelygin, Jean Wu, JasonChuang, Christopher D Manning, Andrew Ng, andChristopher Potts. 2013. Recursive deep modelsfor semantic compositionality over a sentiment tree-bank. In Proceedings of the 2013 conference onempirical methods in natural language processing,pages 1631–1642.

Robert R Sokal. 1958. A statistical method for eval-uating systematic relationships. Univ. Kansas, Sci.Bull., 38:1409–1438.

Alex Wang, Amapreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R Bowman. 2018.Glue: A multi-task benchmark and analysis platformfor natural language understanding. arXiv preprintarXiv:1804.07461.

Adina Williams, Nikita Nangia, and Samuel R Bow-man. 2017. A broad-coverage challenge corpus forsentence understanding through inference. arXivpreprint arXiv:1704.05426.

Will Y Zou, Richard Socher, Daniel Cer, and Christo-pher D Manning. 2013. Bilingual word embeddingsfor phrase-based machine translation. In Proceed-ings of the 2013 Conference on Empirical Methodsin Natural Language Processing, pages 1393–1398.