Understanding Learning Dynamics Of Language Models with SVCCA

Naomi Saphra and Adam Lopezn.saphra@ed.ac.uk alopez@ed.ac.uk

Institute for Language, Cognition, and ComputationUniversity of Edinburgh

Abstract

Research has shown that neural models im-plicitly encode linguistic features, but therehas been no research showing how these en-codings arise as the models are trained. Wepresent the first study on the learning dy-namics of neural language models, using asimple and flexible analysis method calledSingular Vector Canonical Correlation Anal-ysis (SVCCA), which enables us to comparelearned representations across time and acrossmodels, without the need to evaluate directlyon annotated data. We probe the evolution ofsyntactic, semantic, and topic representationsand find that part-of-speech is learned earlierthan topic; that recurrent layers become moresimilar to those of a tagger during training;and embedding layers less similar. Our resultsand methods could inform better learning al-gorithms for NLP models, possibly to incor-porate linguistic information more effectively.

1 Introduction

Large neural networks have a notorious capacityto memorize training data (Zhang et al., 2016),but their high accuracy on many NLP tasks showsthat they nonetheless generalize. One apparent ex-planation for their performance is that they learnlinguistic generalizations even without explicit su-pervision for those generalizations—for example,that subject and verb number agree in English(Linzen et al., 2016); that derivational suffixes at-tach to only specific parts of speech (Kementched-jhieva and Lopez, 2018); and that short segmentsof speech form natural clusters corresponding tophonemes (Alishahi et al., 2017). These studiestell us that neural models learn to implicitly rep-resent linguistic categories and their interactions.But how do they learn these representations?

One clue comes from the inspection of multi-layer models, which seem to encode lexical cate-

gories in lower layers, and more contextual cate-gories in higher layers. For example, Blevins et al.(2018) found that a word’s part of speech (POS) isencoded by lower layers, and the POS of its syn-tactic parent is encoded by higher layers; whileBelinkov et al. (2018) found that POS is encodedby lower layers and semantic category is encodedby higher layers. More generally, the most usefullayer for an arbitrary NLP task seems to depend onhow “high-level” the task is (Peters et al., 2018).Since we know that lower layers in a multi-layermodel converge to their final representations morequickly than higher layers (Raghu et al., 2017), itis likely that models learn local lexical categorieslike POS earlier than they learn higher-level lin-guistic categories like semantic class.

How and when do neural representations cometo encode specific linguistic categories? Answerscould explain why neural models work and help usimprove learning algorithms. We investigate howrepresentations of linguistic structure are learnedover time in neural language models (LMs), whichare central to NLP: on their own, they are usedto produce contextual representations of words formany tasks (e.g. Peters et al., 2018); while con-ditional LMs power machine translation, speechrecognition, and dialogue systems. We use a sim-ple and flexible method, Singular Vector Canon-ical Correlation Analysis (SVCCA; Raghu et al.,2017), which allows us to compare representa-tions from our LM at each epoch of training withrepresentations of other models trained to predictspecific linguistic categories. We discover thatlower layers initially discover features shared byall predictive models, but lose these features as theLM explores more specific clusters. We demon-strate that different aspects of linguistic structureare learned at different rates within a single recur-rent layer, acquiring POS tags early but continuingto learn global topic information later in training.

arX

iv:1

811.

0022

5v3

[cs

.CL

] 3

Apr

201

9

2 Methods

Our experiments require a LM, tagging models,and a method to inspect the models: SVCCA.

2.1 Language modelWe model the probability distribution over a se-quence of tokens x1 . . . x|x| with a conventionaltwo-layer LSTM LM. The pipeline from input xtat time step t to a distribution over xt+1 is de-scribed in Formulae (1)–(4). At time step t, inputword xt is embedded as (1) het , which is input toa two-layer LSTM, producing outputs (2) h1t and(3) h2t at these layers, along with cell states c1t andc2t . A softmax layer converts h2t to a distributionfrom which (4) xt+1 is sampled.

het = embedding(xt) (1)

h1t , c1t = LSTM1(h

et , h

1t−1, c

1t−1) (2)

h2t , c2t = LSTM2(h

1t , h

2t−1, c

2t−1) (3)

xt+1 ∼ softmax(h2t ) (4)

Each function can be thought of as a representa-tion or embedding of its discrete input; hence het isa representation of xt, and—due to the recursionin (2)—h1t is a representation of x1 . . . xt.

2.2 Tagging modelsTo inspect our language model for learned linguis-tic categories, we will use a collection of taggingmodels, designed to mimic the behavior of our lan-guage model but predicting the next tag rather thanthe next word. Given x1 . . . x|x|, we model a corre-sponding sequence of tags y1 . . . y|x| using a one-layer LSTM. (Our limited labeled data made thismore accurate on topic tagging than another two-layer LSTM, so this architecture does not directlyparallel the LM.)

het′ = embedding′(xt) (5)

h1t′, c1t

′ = LSTM′(het′, h1t−1

′, c1t−1′) (6)

yt+1 ∼ softmax′(h1t′) (7)

We will also discuss input taggers, which sharethis architecture but instead sample yt, the tag ofthe most recently observed word.

2.3 SVCCASVCCA is a general method to compare the cor-relation of two vector representations. Let dA anddB be their dimensions. ForN data points we havetwo distinct views, given by matricesA ∈ RN×dA

and B ∈ RN×dB . We project these views onto ashared subspace in two steps:

1. Use Singular Value Decomposition (SVD) toreduce matrices A and B to lower dimen-sional matrices A′ and B′, respectively. Thisis necessary because many dimensions in therepresentations are noisy, and in fact canceleach other out (Frankle and Carbin, 2018).SVD removes dimensions that were likely tobe less important in the original representa-tions from A and B, and in keeping withRaghu et al. (2017), we retain enough dimen-sions to keep 99% of the variance in the data.

2. Use Canonical Correlation Analysis (CCA)to project A′ and B′ onto a shared sub-space, maximizing the correlation of the pro-jections. Formally, CCA identifies vectorsw, v to maximize ρ = <w>A′,v>B′>

‖w>A′‖‖v>B′‖ . Wetreat these w, v as new basis vectors, comput-ing the top dC (a hyperparameter) such ba-sis vectors to form projection matrices W ∈RdC×dA′ , V ∈ RdC×dA′ . The resulting pro-jections WA′ and V B′ map onto a sharedsubspace where the representations of eachdatapoint from A′ and B′ are maximally cor-related.

Intuitively, the correlation ρ will be high if bothrepresentations encode the same information, andlow if they encode unrelated information. Figure 1illustrates how we use SVCCA to compare repre-sentation h2t of our language model with the recur-rent representation of a tagger, h1t

′. In practice, werun over all time steps in a test corpus, rather thana single time step as illustrated.

3 Experimental Setup

We trained our LM on a corpus of tok-enized, lowercased English Wikipedia (70/10/20train/dev/test split). To reduce the number ofunique words in the corpus, we excluded any sen-tence with a word type appearing fewer than 100times. Words appearing fewer than 100 times inthe resulting training set are replaced with an un-known token. The resulting training set has over227 million tokens of 20K types.

We train for 50 epochs to maximize cross-entropy, using a batch size of 40, dropout ratioof 0.2, and sequence length of 35. The optimizeris standard SGD with clipped gradients at 0.25,

xt

embedding

het

LSTM

h1t

LSTM

h2t

softmax

xt+1

Language model

xt

embedding

het′

LSTM

h1t′

softmax

yt+1

Tag Predictor

SVD(h2)

SVD(h1′)SVD

SVD

maxW,V

ρ(W · SVD(h2), V · SVD(h1′))

Figure 1: SVCCA used to compare the layer h2 of a language model and layer h1′ of a tagger.

Tag These cats live in that house .UDP POS DET NOUN VERB ADP DET NOUN SYMPTB POS DT NNS VBP IN DT NN .

SEM (coarse) DEM ENT EVE ATT DEM ENT LOGSEM (fine) PRX CON ENS REL DST CON NIL

topic pets pets pets pets pets pets pets

Table 1: An annotated example sentence from the article pets, based on an example from Bjerva et al. (2016).

Figure 2: Test performance of the LM. Vertical dottedlines indicate when the optimizer rescales the step size.

with the learning rate quartered when validationloss increases. The result of training is shown inFigure 2, which illustrates the dips in loss whenlearning rate changes. All experiments on the LMthroughout training are conducted by running themodel at the end of each epoch in inference modeover the test corpus.

3.1 Taggers

To understand the representations learned by ourLM, we compare them with the internal represen-tations of tagging models, using SVCCA. Wherepossible, we use coarse-grained and fine-grainedtagsets to account for effects from the size of thetagset. Table 1 illustrates our tagsets.

POS tagging For syntactic categories, we usePOS tags, as in Belinkov et al. (2017). As acoarse-grained tagset, we use silver Universal De-pendency Parse (UDP) POS tags automaticallyadded to our Wikipedia corpus with spacy.1 Wealso use a corpus of fine-grained human anno-tated Penn Treebank POS tags from the GroningenMeaning Bank (GMB; Bos et al., 2017).

Semantic tagging We follow Belinkov et al.(2018) in representing word-level semantic infor-mation with silver SEM tags (Bjerva et al., 2016).SEM tags disambiguate POS tags in ways that arerelevant to multilingual settings. For example, thecomma is not assigned a single tag as punctua-tion, but has distinct tags according to its function:conjunction, disjunction, or apposition. The 66fine-grained SEM tag classes fall under 13 coarse-grained tags, and an ‘unknown’ tag.

Global topic For topic, we classify each wordof the sequence by its source Wikipedia article;for example, every word in the wikipedia articleon Trains is labeled “Trains”. This task assesseswhether the network encodes the global topic ofthe sentence.

1https://spacy.io/

Figure 3: SVCCA score between representations ateach epoch and from the final trained LM.

UDP silver POS and topic information use thesame corpus, taken from the 100 longest articlesin Wikipedia randomly partitioned in a 70/10/20train/dev/test split. Each token is tagged with POSand with the ID of the source article. The corpusis taken from the LM training data, which may in-crease the similarity between the tag model andLM. Because both tag predictors are trained andtested on the same domain as the LM, they canbe easily compared in terms of their similarity tothe LM representation. Though the SEM corpusand the PTB corpus are different domains fromthe Wikipedia training data, we compare their ac-tivations on the same 191K-token 100-article testcorpus.

Table 2 describes the training and validationcorpus statistics for each tagging task. Note thattopic and UDP POS both apply to the same en-wikipedia corpus, but PTB POS and SEM use twodifferent unaligned sets from the GMB corpus.

4 Experiments, Results, and Analysis

A benefit of SVCCA is its flexibility: it can com-pute the correlation of a hidden representation toany other vector. Raghu et al. (2017) used itto understand learning dynamics by comparing alearned representation to snapshots of the samerepresentation at different epochs during training.We use a similar experiment to establish the basiclearning dynamics of our model. In our shallow2-level model, activations at h1 converge slightlyafter h2 (Figure 3). This differs from the resultsof Raghu et al. (2017), who found that a 5-layerstacked LSTM LM exhibits faster convergence atlower layers, but this difference may be attributedto our much larger training data, which gives ourmodel sufficient training data at early epochs.

Figure 4: SVCCA score between the LM at each epochand a LM with different initialization.

Empirical upper bounds. Our main experi-ments will test the rate at which different linguis-tic categories are learned by different layers, butto interpret the results, we need to understand thebehaviour of SVCCA for these models. In the-ory, SVCCA scores can vary from 0 for no corre-lation to 1 for perfect correlation. But in practice,these extreme cases will not occur. To establishan empirical upper bound on correlation, we com-pared the similarity at each epoch of training to thefrozen final state of a LM with identical architec-ture but different initialization, trained on the samedata (Figure 4).2 The correlations increase overtime as expected, but to a maximum near 0.64;we don’t expect correlations between our LM andother models to exceed this value. We explore cor-responding lower bounds in our main experimentsbelow.

Correlations between different layers. Nextwe examine the correlation between different lay-ers of the same model over time (Figure 5). We ob-serve that, while over time correlation increases, ingeneral closer layers are more similar, and they areless correlated than they are with the same layer ofa differently initialized model. This supports theidea that we should compare recurrent layers withrecurrent layers because their representations playsimilar roles within their respective architectures.

SVCCA vs. Diagnostic classifiers A popularmethod to analyze learned representations is to usea diagnostic classifier (Belinkov et al., 2017), aseparate model that is trained to predict a linguisticcategory of interest, yt, from an arbitrary hiddenlayer ht. Diagnostic classifiers are widely used(Belinkov et al., 2018; Giulianelli et al., 2018).But if a diagnostic classifier is trained on enough

2This experiment is similar to the comparisons of ran-domly initialized models by Morcos et al. (2018).

number token count label t+ 1 label t randomizedtag corpus of classes train dev acc ppl acc ppl acc pplUDP POS wiki 17 665K 97K 50 4.3 93 1.2 21 8.9PTB POS GMB 36 943K 136K 51 4.7 95 1.18 14 18.0SEM (coarse) GMB 14 937K 132K 55 3.5 91 1.3 22 9.0SEM (fine) GMB 67 937K 132K 50 5.6 88 1.45 17 21.5topic wiki 100 665K 97K 36 19.1 37 16.3 5 81.6

Table 2: Tag predictor and tagger statistics. Accuracy and perplexity on t + 1 are from the target tag predictor,on t are from the input tagger. Metrics obtained when training on randomly shuffled labels are provided as a lowbaseline. Accuracy is on the test set from the training domain (GMB or Wikipedia).

Figure 5: SVCCA score between different layers of theLM at each epoch. For example, ρ(h2, h1) comparesthe activations h2 with the activations h1.

examples, then random embeddings as input rep-resentations often outperform any pretrained in-termediate representation (?Zhang and Bowman,2018). This suggests that diagnostic classifiersmay work simply by memorizing the associa-tion between an embedding and the most frequentoutput category associated with that embedding;since for many words their category is (empiri-cally) unambiguous, this may give an inflated viewof just how much a model “understands” aboutthat category.

Our use of SVCCA below will differ from theuse of diagnostic classifiers in an important way.Diagnostic classifiers use the intermediate repre-sentations of the LM as inputs to a tagger. A repre-sentation is claimed to encode, for example, POSif the classifier accurately predicts it—in otherwords, whether it can decode it from the repre-sentation. We will instead evaluate the similaritybetween the representations in an LM and in anindependently-trained tagger. The intuition behindthis is that, if the representation of our LM encodesa particular category, then it must be similar to therepresentation of model that is specifically trainedto predict that category. A benefit of the approachis that similarity can be evaluated on any dataset,not only one that has been labeled with the linguis-tic categories of interest.

Figure 6: Learning dynamics interpreted with diagnos-tic classifiers labeling input word tag yt.

Another distinction from the typical use of diag-nostic classifiers is that probes are usually used todecode tag information about the context or mostrecent input from the hidden state at the currentstep. Because the hidden representation at timet is meant to encode predictive information aboutthe target word at time t+1, we treat it as encodinga prediction about the tag of the target word.

To understand the empirical strengths andweaknesses of these approaches, we compare theuse of SVCCA and diagnostic classifiers in under-standing learning dynamics. In other words, weask: is our first conceptual shift (to SVCCA) nec-essary? To test this, we use the same model as Be-linkov et al. (2017), which classifies an arbitraryrepresentation using a ReLU followed by a soft-max layer. To be consistent with Belinkov et al.(2017), we use yt as their target label. We repeattheir method in this manner (Figure 6) as well asapplying our second modification, in which we in-stead target the label yt+1 (Figure 7).

We found the correlations to be relatively stableover the course of training. This is at odds withthe results in Figures 2 and 3, which suggest that

Figure 7: Learning dynamics interpreted with diagnos-tic classifiers labeling target word tag yt+1.

representations change substantially during train-ing in ways that materially affect the accuracy ofthe LM. This suggests that diagnostic classifiersare not illustrating improvements in word repre-sentations throughout training, and we concludethat they are ineffective for understanding learningdynamics. Our remaining experiments use onlySVCCA.

4.1 SVCCA on Output Tag Prediction

We applied SVCCA to each layer of our LM withthe corresponding layer of each tag predictor inorder to find the correlation between the LM rep-resentation and the tag model representation ateach level (Figure 8). To establish empirical lowerbounds on correlation, we also trained our taggerson the same data with randomly shuffled labels,as in Zhang et al. (2016). These latter experi-ments, denoted by the dotted lines of Figure 8,show how much of the similarity between mod-els is caused by their ability to memorize arbi-trary associations. Note that the resulting scoresare nonzero, likely because the linguistic structureof the input shapes representations even when theoutput is random, due to the memorization phaseof training (Shwartz-Ziv and Tishby, 2017).

The strongest similarity at recurrent layers be-longs to the most local property, the UDP POStag. Both coarse- and fine-grained semantic tags,which rely on longer range dependencies, fall be-low UDP POS consistently. Topic, which is globalto an entire document, is the least captured and theslowest to stabilize. Indeed, correlation with truetopic falls consistently below the score for a model

Figure 8: SVCCA correlation scores between the LMpredicting xt+1 and the tag model predicting yt+1. Atthe end of each epoch, we compare the current LM withthe final tag model. Dotted lines use shuffled tags. Grayvertical lines mark when the step size is rescaled.

trained on randomized topic tags, implying thatearly in training the model has removed the con-text necessary to identify topic (below even the in-adequate contextual information memorized by amodel with random labels), which depends on thegeneral vocabulary in a sentence rather than a localsequence. Over time correlation rises, possibly be-cause the model permits more long-distance con-text to be encoded. Khandelwal et al. (2018) foundthat LSTMs remember content words like nounsfor more time steps than they remember functionwords like prepositions and articles. We hypothe-size that the LM’s slower stabilization on topic isrelated to this phenomenon, since it must dependon content words, and its ability to remember themincreases throughout training.

The encoder layer exhibits very different pat-terns. Because the representation produced by theencoder layer is local to the word, the nuancesthat determine how a word is tagged in contextcannot be learned. The encoder layers are allhighly similar to each other, which suggests thatthe unigram representations produced by the en-coder are less dependent on the particular end taskof the neural network. Similarity between the en-coders declines over time as they become morespecialized towards the language modeling task.This decline points to some simple patterns whichare learned for all language tasks, but which aregradually replaced by representations more use-ful for language modeling. This process may evenbe considered a naturally occurring analog to thecommon practice of initializing the encoder layeras word embeddings pretrained an unrelated tasksuch as skipgram or CBOW (Mikolov et al., 2013).It seems that the ‘easy’ word properties, whichimmediately improve performance, are similar re-gardless of the particular language task.

At h1, the correlation shows a clear initial de-cline in similarity for all tasks. This seems to pointto an initial representation that relies on simpleshared properties, which in the first stage of train-ing is gradually dissolved before the layer beginsto converge on a structure shared with each tagpredictor. It may also be linked to the informationbottleneck learning phases explored by Shwartz-Ziv and Tishby (2017). They suggest that neu-ral networks learn by first maximizing the mutualinformation between the input and internal rep-resentation, then minimizing the mutual informa-tion between the internal representation and out-

put. The network thus initially learns to effectivelyrepresent the input, then compresses this represen-tation, keeping only the elements relevant to theoutput.3 If the LM begins by maximizing mutualinformation with input, because the input is identi-cal for the LM and tag models it may lead to thesesimilar initial representations, followed by a de-cline in similarity as the compression narrows toproperties specific to each task.

4.2 SVCCA on Input Tagging

Our second conceptual shift is to focus on out-put tag prediction—asking what a representationencodes about the next output word, rather thanwhat it has encoded about words it has already ob-served in the input. What effect does this have?Since we already studied output tags in the pre-vious set of experiments, here we consider inputtags, in the style of most diagnostic classifier anal-ysis (Figure 9). The learning dynamics are similarto those for tag prediction, but the UDP POS tag-ger decreases dramatically in all correlations whilethe GMB-trained taggers4 often increase slightly.While the shapes of the lines are similar, UDPPOS no longer consistently dominates the othertasks in recurrent layer correlation. Instead, wefind the more granular PTB POS tags lead to themost similar representations.

5 Discussion and Conclusions

We find clear patterns in the encoding of linguisticstructure with SVCCA, in contrast to the weakerresults from a less responsive diagnostic classifier.Because SVCCA proves so much more sensitivethan the diagnostic classifiers currently in use, webelieve that future work on measuring the encod-ing of linguistic structure should use the similarityof individual modules from independently trainedtag predictors rather than the performance of tagpredictors trained on a particular representation.

This system should also be of interest becauseit is efficient. To train a diagnostic classifier, wemust run a forward pass of the LM for each for-ward pass of the auxiliary model, while SVCCAonly requires the LM to run on the test set. A fur-ther efficiency gain is particular to studying learn-

3This memorization–compression learning pattern paral-lels the memorization–generalization of the first half of theU-shaped curve exhibited by human children learning irreg-ular word forms. ? observe similar patterns when artificiallymodeling inflection.

4PTB POS, SEM (fine), and SEM (coarse)

Figure 9: SVCCA correlation scores between LM ac-tivations when predicting xt+1 and tagger activationswhen labeling yt. Dotted lines use shuffled tags. Grayvertical lines mark when the step size is rescaled.

ing dynamics: we train only one tagger and com-pare it to different versions of the LM over train-ing, but for standard probing, we must train anew version of each layer’s tagger at each epoch.Our SVCCA experiments in Figure 8 ran in hours,while the diagnostic classifier experiments in Fig-ure 7 ran for days.

Our method holds another, more subtle advan-tage. Our analysis provides an alternative view ofwhat it means for a model to encode some linguis-tic trait. The literature on analyzing neural net-works includes a broad spectrum of interpretationsabout what it means to encode a property. At oneend of the spectrum lies the purely informationalview (e.g., mutual information; ?). Mutual infor-mation is a very flexible view, but it requires us tocompute theoretical information content, which inpractice can only be estimated. Furthermore, in-formation can be represented without being used,as shown by ?, who found that NMT systems of-ten predicted tense according to a diagnostic clas-sifier but did not produce the correct tense as out-put. The other end of the spectrum is focused onthe structure of the representation space (e.g., thefeatures and the property in question are linearlysimilar; Alishahi et al., 2017). Analyzing struc-tural similarity should remedy the shortcomings ofthe informational view, but most intermediate rep-resentations are not targeted to extract the prop-erty in question through a linear transformation,and failing to be interpretable through such sim-ple extraction should not be equated to a failure toencode that property.

Most of the literature on analyzing representa-tions, by probing with a more complex architec-ture, seeks the flexibility of mutual informationwith the concreteness and tractability of the struc-tural view – but instead obscures the strict infor-mation view without offering interpretable infor-mation about the structure, because the architec-ture of a diagnostic classifier affects its perfor-mance. It should not be surprising that represen-tational quality as measured by such systems is apoor indicator of translation quality (?). SVCCA,in contrast, is a structural view that does not di-rectly compare an activation that targets word pre-diction with a particular tag, but instead comparesthat activation with one targeting the prediction ofthe tag.

Let us consider a specific common probingmethod. What do we learn about the LM when a

feedforward network cannot extract tag informa-tion directly from the embedding layer, but canfrom a recurrent layer? It may be tempting to con-clude that tag information relies heavily on con-text, but consider some alternative explanations.If the embedding encodes the tag to be interpretedby a recurrent layer, a feedforward network maynot be capable of representing the function to ex-tract that tag because it does not have access to acontext vector for aiding interpretation of the hid-den layer. Perhaps its activation functions cover adifferent range of outputs. By directly comparingLSTM layers to LSTM layers and embedding lay-ers to embedding layers, we respect the shape oftheir outputs and the role of each module withinthe network in our analysis.

The results of our analysis imply that early intraining, representing part of speech is the natu-ral way to get initial high performance. However,as training progresses, it increasingly benefits themodel to represent categories with longer-rangedependencies, such as topic.

6 Future Work

One direction for future work is exploring howgeneralization interacts with the correlations be-tween LMs and tag predictors. It may be that afaithful encoding of a property like POS tag in-dicates that the LM is relying more on linguisticstructure than on memorizing specific phrases, andtherefore is associated with a more general model.

If these measurements of structure encoding areassociated with more general models, we might in-troduce regularizers or other modifications that ex-plicitly encourage correlation with a tagging task.

Combes et al. (2018) identified the phenomenonof gradient starvation, meaning that while fre-quent and unambiguous features are learnedquickly in training, they slow down the learningof rarer features. For example, artificially bright-ening images according to their class leads to adelay in learning to represent the more complexnatural class features. Although it is tempting toclaim that semantic structure is learned using syn-tactic structure as natural scaffolding, it is possiblethat the simple predictive power of POS is actingas an attractor and starving semantic features thatare rarer and more ambiguous. A possible direc-tion for future work would be to explore which ofthese explanations is true, possibly by decorrelat-ing particular aspects of linguistic structure from

language modeling representations.The techniques in this paper could be applied to

better understand the high performance of a sys-tem like ELMo (Peters et al., 2018). Different lay-ers in such a system are useful for different tasks,and this effect could be understood in terms of thegradual divergence between the layers and their re-spective convergence to representations geared to-ward a single task.

Acknowledgements

We thank Denis Emelin, Sameer Bansal, TomsBergmanis, Maria Corkery, Sharon Goldwater,Sorcha Gilroy, Aibek Makazhanov, Yevgen Ma-tusevych, Kate McCurdy, Janie Sinclair, Ida Szu-bert, Nikolay Bogoychev, Clara Vania, and theanonymous reviewers for helpful discussion andcomments on drafts. We thank Matthew Summersfor assistance with visualisations.

ReferencesAfra Alishahi, Marie Barking, and Grzegorz Chrupała.

2017. Encoding of phonology in a recurrent neu-ral model of grounded speech. In Proceedings ofthe 21st Conference on Computational Natural Lan-guage Learning (CoNLL 2017), pages 368–378.

Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Has-san Sajjad, and James Glass. 2017. What do NeuralMachine Translation Models Learn about Morphol-ogy? In Proceedings of the 55th Annual Meeting ofthe Association for Computational Linguistics (Vol-ume 1: Long Papers), pages 861–872, Vancouver,Canada. Association for Computational Linguistics.

Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad,Nadir Durrani, Fahim Dalvi, and James R. Glass.2018. Evaluating Layers of Representation in Neu-ral Machine Translation on Part-of-Speech and Se-mantic Tagging Tasks. CoRR, abs/1801.07772.

Johannes Bjerva, Barbara Plank, and Johan Bos. 2016.Semantic Tagging with Deep Residual Networks. InCOLING.

Terra Blevins, Omer Levy, and Luke Zettlemoyer.2018. Deep RNNs Encode Soft Hierarchical Syn-tax. arXiv:1805.04218 [cs]. ArXiv: 1805.04218.

Johan Bos, Valerio Basile, Kilian Evang, Noortje JVenhuizen, and Johannes Bjerva. 2017. The gronin-gen meaning bank. In Handbook of linguistic anno-tation, pages 463–496. Springer.

Remi Tachet des Combes, Mohammad Pezeshki,Samira Shabanian, Aaron Courville, and YoshuaBengio. 2018. On the Learning Dynamics of DeepNeural Networks. arXiv:1809.06848 [cs, stat].ArXiv: 1809.06848.

Jonathan Frankle and Michael Carbin. 2018. TheLottery Ticket Hypothesis: Training Pruned Neu-ral Networks. arXiv:1803.03635 [cs]. ArXiv:1803.03635.

Mario Giulianelli, Jack Harding, Florian Mohnert,Dieuwke Hupkes, and Willem Zuidema. 2018. Un-der the Hood: Using Diagnostic Classifiers to In-vestigate and Improve how Language Models TrackAgreement Information. arXiv:1808.08079 [cs].ArXiv: 1808.08079.

Yova Kementchedjhieva and Adam Lopez. 2018. Indi-catements that character language models learn En-glish morpho-syntactic units and regularities. InProc. of Workshop on Analyzing and interpretingneural networks for NLP.

Urvashi Khandelwal, He He, Peng Qi, and DanJurafsky. 2018. Sharp Nearby, Fuzzy FarAway: How Neural Language Models Use Context.arXiv:1805.04623 [cs]. ArXiv: 1805.04623.

Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.2016. Assessing the Ability of LSTMs to LearnSyntax-Sensitive Dependencies. arXiv:1611.01368[cs]. ArXiv: 1611.01368.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed Representa-tions of Words and Phrases and their Composition-ality. In Advances in Neural Information ProcessingSystems 26, pages 3111–3119. Curran Associates,Inc.

Ari S. Morcos, Maithra Raghu, and Samy Ben-gio. 2018. Insights on representational similar-ity in neural networks with canonical correlation.arXiv:1806.05759 [cs, stat]. ArXiv: 1806.05759.

Matthew E Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. arXiv preprint arXiv:1802.05365.

Maithra Raghu, Justin Gilmer, Jason Yosinski, andJascha Sohl-Dickstein. 2017. SVCCA: Singu-lar Vector Canonical Correlation Analysis forDeep Learning Dynamics and Interpretability.arXiv:1706.05806 [cs, stat]. ArXiv: 1706.05806.

Ravid Shwartz-Ziv and Naftali Tishby. 2017. Open-ing the Black Box of Deep Neural Networks viaInformation. arXiv:1703.00810 [cs]. ArXiv:1703.00810.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben-jamin Recht, and Oriol Vinyals. 2016. Understand-ing deep learning requires rethinking generalization.arXiv preprint arXiv:1611.03530.

Kelly W. Zhang and Samuel R. Bowman. 2018. Lan-guage Modeling Teaches You More Syntax thanTranslation Does: Lessons Learned Through Auxil-iary Task Analysis. arXiv:1809.10040 [cs]. ArXiv:1809.10040.

A Performance Out Of Domain

Because SEM tags and PTB POS tags wereboth trained on the GMB corpus, we present theSVCCA similarities on an in-domain GMB testcorpus as well as the Wikipedia test corpus usedelsewhere in the paper. The results are in Fig-ures 10-11. In general correlations are higher us-ing the original tagging domain, but not enough tocontradict our earlier analysis. The shapes of thecurves remain similar.

Figure 10: SVCCA correlation scores between LM andyt+1 tag predictor. Dotted lines use models trained onrandomly shuffled the data. Dashed lines use GMB do-main test data.

Figure 11: SVCCA correlation scores between LM andyt tagger. Dotted lines use models trained on randomlyshuffled the data. Dashed lines use GMB domain testdata.