Proceedings of the 2021 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human Language Technologies, pages 915–929

June 6–11, 2021. ©2021 Association for Computational Linguistics

915

Towards Interpreting and Mitigating Shortcut LearningBehavior of NLU Models

Mengnan Du1∗, Varun Manjunatha2, Rajiv Jain2, Ruchi Deshpande3, Franck Dernoncourt2,Jiuxiang Gu2, Tong Sun2 and Xia Hu1

1Texas A&M University2Adobe Research

3Adobe Document Clouddumengnan@tamu.edu

{vmanjuna,rajijain,rdeshpan,franck.dernoncourt,jigu,tsun}@adobe.com, hu@cse.tamu.edu

Abstract

Recent studies indicate that NLU models areprone to rely on shortcut features for predic-tion, without achieving true language under-standing. As a result, these models fail to gen-eralize to real-world out-of-distribution data.In this work, we show that the words in theNLU training set can be modeled as a long-tailed distribution. There are two findings: 1)NLU models have strong preference for fea-tures located at the head of the long-tailed dis-tribution, and 2) Shortcut features are pickedup during very early few iterations of themodel training. These two observations arefurther employed to formulate a measurementwhich can quantify the shortcut degree of eachtraining sample. Based on this shortcut mea-surement, we propose a shortcut mitigationframework LTGR, to suppress the model frommaking overconfident predictions for sampleswith large shortcut degree. Experimental re-sults on three NLU benchmarks demonstratethat our long-tailed distribution explanation ac-curately reflects the shortcut learning behaviorof NLU models. Experimental analysis furtherindicates that LTGR can improve the general-ization accuracy on OOD data, while preserv-ing the accuracy on in-distribution data.

1 Introduction

Pre-trained language models, such as BERT (De-vlin et al., 2019), have demonstrated substantialgains on many NLU (natural language understand-ing) benchmarks. However, recent studies showthat these models tend to exploit dataset biases asshortcuts to make predictions, rather than learnthe semantic understanding and reasoning (Geirhoset al., 2020; Gururangan et al., 2018). Here we fo-cus on the lexical bias, where NLU models rely onspurious correlations between shortcut words and

∗Most of the work was done while the first author was anintern at Adobe Research.

labels. This eventually results in their low general-izability on out-of-distribution (OOD) samples andlow adversarial robustness (Zellers et al., 2018).

In this work, we show that the shortcut learningbehavior of NLU models can be explained by thelong-tailed phenomenon. Previous empirical anal-ysis indicates that the performance of BERT-likemodels for NLI task could be mainly explainedby the reliance of spurious statistical cues suchas unigrams “not”, “do”, “is” and bigrams “willnot” (Niven and Kao, 2019; Gururangan et al.,2018). Here we generalize these hypotheses us-ing the long-tailed phenomenon. Specifically, thefeatures in training set could be modeled using along-tailed distribution via using local mutual in-formation (Evert, 2005) as a measurement. Byutilizing an interpretation method to analyze modelbehavior, we observe that these NLU models con-centrate mainly on information on the head of thedistribution, which usually corresponds to non-generalizable shortcut features. In contrast, thetail of the distribution is poorly learned, althoughit contains high information for the NLU task. An-other key observation is that during training pro-cess, shortcut features tend to be picked up by NLUmodels during very early iterations. Based on thesetwo key observations, we define a measurement toquantify the shortcut degree of all training samples.

Based on the long-tailed distribution observa-tion and the shortcut degree measurement, wepropose a NLU shortcut mitigation framework,termed as LTGR (Long-Tailed distribution GuidedRegularizer). The proposed regularizer is basedon the observation that NLU models would giveover-confident predictions when there exist strongshortcut features in the input. This is because NLUmodels over-associate the shortcut features withcertain class labels. LTGR is implemented usingthe knowledge distillation framework, to penalizethe NLU model from outputting overconfident pre-

916

Input x

Teacher model

Softmax

Softmax

Ground truth y

SmoothedSoftmax

Distill loss

Student loss

Overconfident prediction

Long-tailed distribution

Example of model paying high attention to features on the head

(a) long-tailed observation (b) Mitigation framework

Shortcut degree

Student model

Head

Long tail

Shortcut degree

Data statistics

Model behavior

Figure 1: (a) Our key intuition is that the training set can be modeled as a long-tailed distribution. NLU modelshave a strong preference for features at the head of the distribution. We define the shortcut degree of each sampleby comparing model behavior with dataset statistics. (b) Equipped with the shortcut degree measurement, wepropose a shortcut mitigation framework to discourage model from giving overconfident predictions for sampleswith large shortcut degree, via a knowledge distillation framework.

diction for training samples with large shortcut de-gree. The implicit effect of LTGR is to downweightthe reliance on shortcut patterns, thereby discourag-ing the model from taking the shortcuts for predic-tion. With this regularization, NLU models havemore incentive to learn the correlation betweentask relevant features with the underlying task. Themajor contributions are summarized as follows:

• We indicate that the shortcut learning behaviorsof NLU models can be explained by the long-tailed phenomenon, and show that shortcuts arepicked up at the early stage of model training.• We propose a shortcut mitigation method, called

LTGR, guided by the long-tailed observation.The idea is to down-weight model’s reliance onshortcuts and implicitly encourage the model toshift its attention to task relevant features.• Experimental results on several NLU datasets

validate that the long-tailed observation faith-fully explains the shortcut learning behaviors.The analysis further shows LTGR could improvegeneralization on OOD samples, while not sacri-ficing accuracy of in-distribution samples.• We demonstrate that our LTGR approach can

partially mitigate shortcut based Trojan attacksthrough a preliminary experiment.

2 Long-Tailed Phenomenon

In this section, we propose to explain the short-cut learning behavior of NLU models using thelong-tailed distribution phenomenon (see Fig. 1(a)).Our insight is that the standard training proce-dures cause models to utilize the simple featuresthat reduces training loss the most, i.e., simplicity

bias (Shah et al., 2020). This directly results in thelow generalization of NLU models.

2.1 Preference for Features of High LocalMutual Information

NLU tasks are typically formulated as a multi-classclassification task: given an input sentence pairx, the goal is to learn a mapping f(x) to predictthe semantic relationship label y. In the trainingset, some words or phrases within x co-occur morefrequently with one label y than others. The NLUmodel would capture those shortcut features for pre-diction. Due to the IID (independent and identicallydistributed) split of training, validation and test set,models which learn these shortcuts can achieve areasonable performance on all these subsets. Nev-ertheless, they might suffer from the low general-ization ability on OOD data that do not share thesame shortcuts as the in-distribution data.Dataset Statistics. We model statistics using localmutual information (LMI) (Schuster et al., 2019)between a wordw and a label y, denoted as follows:

LMI(w, y) = p(w, y) · log(p(y|w)p(y)

), (1)

where p(w, y) = count(w,y)|D| , p(y|w) = count(w,y)

count(w) .|D| is the number of unique words in training set,count(w, y) denotes the co-occurrence of word wwith label y, and count(w) is total number of wordsin the training set. After analyzing each word forthe training set, we obtain |y| distributions of |y|labels. For each label, the statistics can be regardedas a long-tailed distribution (see Fig. 1(a)). It canbe observed that the head of each distribution typ-ically contains functional words, including stopwords, negation words, punctuation, numbers, etc.

917

These words carry low information for the NLUtask. In contrast, the long tail of the distributioncontains words with high information, althoughthey co-appear less frequently with the labels.Model Behavior. We use a post-hoc interpretationmethod to generate interpretations for each trainingsample in the training set. It is achieved by attribut-ing model’s prediction in terms of its input features,and the final interpretation is illustrated in the for-mat of feature importance vector (Montavon et al.,2018). Here we use a gradient based interpretationmethod: Integrated Gradient (Sundararajan et al.,2017). Integrated Gradients is a variation on cal-culating the gradient of the model prediction w.r.t.features of the input. The main idea is to integratethe gradients of m intermediate samples over thestraightline path from baseline xbase to input xi,which could be denoted as follows:

g(xi) = (xi−xbase)·m∑k=1

∂fy(xbase +km(xi − xbase)

∂xi· 1m.

(2)

Let each input text is composed of T words:xi = {xti}Tt=1, and each word xti ∈ Rd denotesa word embedding with d dimensions. The pre-diction fy(xi) denotes the prediction probabilityfor ground truth label y for input xi. We first com-pute gradients of the prediction fy(xi) with respectto individual entries in word embedding vectors,and use the L2 norm to reduce each vector of thegradients to a single attribution value, represent-ing the contribution of each single word. We usethe all-zero word embedding as the baseline xbase.Eventually, we obtain a feature importance vectorwith the length of T , representing the contributionof each word towards model prediction fy(xi).Comparing Model and Dataset. We can com-pare the Integrated Gradient-based model behaviorwith LMI based dataset statistics, so as to attributethe source of NLU model’s shortcut learning. Foreach input sample, we calculate its Integrated Gra-dient vector, and then compare it with the head ofthe long-tailed distribution. Our preliminary experi-ments (see Sec. 4.2) indicate that NLU classifier arevery strong superficial learners. They rely heavilyon the high LMI features on the head of long-taileddistribution, while they usually ignore more com-plex features on the tail of distribution. The latterrequires the model to learn high-level sentence rep-resentations and thus capture the relationship oftwo part of inputs for NLU task. Based on this em-pirical observation, we can define the measurement

of the shortcut degree of each sample by calculat-ing the similarity of model and dataset. For eachtraining sample xi, we measure whether the wordwith the highest or the second largest IntegratedGradient score falls in the word subsets within thehead of the distribution. Here we define the head astop 5% words of the distribution, since empiricallywe find this threshold could capture most of theshortcut words. We set the shortcut degree ui forsample xi as 1 if it matches. Otherwise if it doesnot match, we set ui = 0.

2.2 Shortcuts Samples are Learned FirstBy examining the learning dynamics of NLU mod-els, another key intuition is that shortcut samplesare learned by the models first. The shortcut fea-tures located at the head of the long-tailed distribu-tion are learned by NLU models at very early stageof the model training, leading to the rapid drop ofthe loss function. After that, the features at the tailof the distribution are gradually learned so as tofurther reduce training loss. Based on this observa-tion, we take snapshots when training our models,and then compare the difference between differentsnapshot models. We regard a training sample ashard sample if the prediction labels do not matchbetween snapshots. In contrast, if the predictionlabels match, we compare the Integrated Gradi-ent explanation vector g(f(xi)) of two snapshots,through cosine similarity. The shortcut measure-ment for sample xi is defined as follows:

vi =

{cosine(g(f1(xi)), g(f(xi))), f1(xi) = f(xi)0, f1 (xi) 6= f (xi)

(3)

where f1(·) denotes the snapshot at the early stageof the training, and we use the model obtained afterthe first epoch. The second snapshot f(·) repre-sents the final converged model. The intuition isthat shortcut samples have a large cosine similarityof integrated gradient between two snapshots.

2.3 Shortcut Degree MeasurementWe define a unified measurement of the shortcutdegree of each training sample, by putting theaforementioned two observations together. Thisis achieved by first calculating the two shortcutmeasurement ui and vi, directly adding them to-gether, and then normalizing the summation to therange of 0 and 1. Ultimately, we obtain the shortcutdegree measurement for each training sample xi,denoted as bi. This measurement bi can be furtherutilized to mitigate the shortcut learning behavior.

918

3 Proposed Mitigation Framework

Equipped with the observation of long-tailed phe-nomenon and the shortcut degree measurementbi obtained from the last section, we propose ashortcut mitigation solution, called LTGR (Long-Tailed distribution Guided Regularizer). LTGRis implemented based on self knowledge distilla-tion (Utama et al., 2020a; Hinton et al., 2015) (seeFig. 1(b)). The proposed distillation loss is basedon the observation that NLU models would giveover-confident predictions when there exist strongshortcut features in the input. This is because NLUmodels over-associate the shortcut features withcertain class labels. The proposed distillation lossaims to suppress the NLU models from giving over-confident predictions for samples with strong short-cut features. It forces the model to down-weight itsreliance on shortcut features and implicitly encour-ages the model to shift its attention to more taskrelevant features.Smoothing Softmax. Based on the biased teachermodel fT , we calculate the logit value and softmaxvalue of training sample xi as zTi and σ(zTi ) respec-tively, where σ is the softmax function. Given alsothe shortcut degree measurement of each trainingsample bi. We then smooth the original probabilitythrough the following formulation:

si,j =σ(zTi )

1−bij∑K

k=1 σ(zTi )

1−bik

, (4)

where K denotes the total number of labels. Whenbi = 0, the si will remain the same as σ(zTi ), rep-resenting that there is no penalization. In anotherextreme when bi = 1, si will have the same valuefor K labels. Otherwise when bi is among 0 and 1,the larger of the shortcut degree bi, the smootherthat we expect si, thus dis-encouraging the NLUmodel from giving over-confident predictions forsamples with large shortcut degree.Self Knowledge Distillation. Ultimately, we usethe following loss to train the student model fS :

L(x) = (1− α) ∗ H(yi, σ

(zSi

))+ α ∗ H

(si, σ

(zSi

)),

(5)

where zSi represents the softmax probability of thestudent network for training sample xi,H denotescross entropy loss. Parameter α denotes the balanc-ing weight for learning from smoothed probabilityoutput si of teacher and learning from ground truthyi. We use the same model architecture for bothteacher fT and student fS , and during the distilla-tion process we fix the parameters of fT and only

Algorithm 1: LTGR mitigation framework.

Input: Training data D = {(xi, yi)}Ni=1.1 Set hyperparameters m, α.2 while first stage do3 Train teacher network fT (x). Fix its parameters.

4 while second stage do5 Initialize the student network fS(x);6 Calculate shortcut degree bi and softmax σ(zTi )

for each training sample {(xi)}Ni=1;7 Smoothing softmax:

8 si,j =σ(zTi )

1−bij∑K

k=1σ(zTi )

1−bik

;

9 Use Eq. 5 to train the student network fS(x)

Output: Discard fT (x). Use fS(x) for prediction.

update parameters of the student model fS (seeAlgorithm 1). Ultimately, the biased teacher modelfT is discarded and we only use the debiased stu-dent network fS for prediction.

4 Experiments

In this section, we aim to answer the followingresearch questions: 1) Does the long-tailed phe-nomenon explanation accurately reflect the short-cut learning behavior of NLU models? 2) Does theproposed LTGR outperform alternative approaches,and what is the source of the improved generaliza-tion? 3) How do components and hyperparametersaffect LTGR’s generalization performance?

4.1 Experimental Setup

Tasks & Datasets. We consider three NLU tasks.

• FEVER: The first task is fact verification, wherethe original dataset is FEVER (Thorne et al.,2018). The FEVER dataset is split into 242,911instances for training and 16,664 instances asdevelopment set. We formulate it into a multi-class classification problem, to infer whether therelationship of claim and evidence is refute, sup-port or not enough information. The two ad-versarial sets are Symmetric v1 and v2 (Sym1and Sym 2), where a shortcut word appears inboth support and refute label (Schuster et al.,2019). Both Symmetric v1 and v2 contain 712samples (Schuster et al., 2019).• MNLI: The second task is NLI (natural lan-

guage inference), where the original dataset isMNLI (Williams et al., 2018). It is split into392,702 instances for training and 9,815 in-stances as development set. We also formulate itinto a multi-class classification problem, to inferwhether the relationship between hypothesis and

919

premise is entailment, contradiction, or neural.Two adversarial set HANS (McCoy et al., 2019)and MNLI hard set (Gururangan et al., 2018)are used to test the generalizability. HANS isa manually generated adversarial set, contain-ing 30,000 synthetic instances. Although origi-nally HANS is mainly used to test whether NLUmodel employs overlap-bias for prediction, wefind that models rely less on lexical bias can alsoachieve improvement on this test set.• MNLI-backdoor: For the third task, we use a lex-

ically biased variant of the MNLI dataset, whichis termed as MNLI-backdoor. We randomly se-lect out 10% of the training samples with theentailment label and append the double quota-tion mark ‘“’ to the beginning of the hypothesis.For adversarial set, we still use MNLI hard set,but append the hypothesis of all samples with ‘“’.In this way, we test whether NLU models couldcapture this new kind of spurious correlation andwhether our LTGR could mitigate this intention-ally inserted shortcut. Note that the double quo-tation mark we use is ‘“’ (near the number 1 onthe keyboard), rather than the usual “‘’, since ‘“’appears infrequently in both the original MNLItraining and validation set.

NLU Models. We consider two pre-trained contex-tualized word embeddings models: BERT base (De-vlin et al., 2019), and DistilBERT (Sanh et al.,2019) as encoder to obtain words representations.We use the pre-trained BERT models from Hug-gingface Transformers1. The input fed to the em-bedding models are obtained by concatenating twobranches of inputs, which are separated using the‘[sep]’ symbol. Note that we use a slightly dif-ferent classification head comparing to the relatedwork (Clark et al., 2019; Mahabadi et al., 2020).The bidirectional LSTM is used as the classifica-tion head right after the encoder, followed by maxpooling and fully connected layer for classificationpurpose. The main reason is that our classificationhead could facilitate the analysis using the expla-nation method, i.e., integrated gradient, to analyzemodel behavior. More model details are put in theSec. A in Appendix.Implementation Details. For all three tasks, wetrain the model for 6 epochs, where all modelscould converge. Hyperparameter α is fixed as 0.8for all models. We use Adam optimizer, where the

1https://huggingface.co/transformers/pretrained_models.html

momentum is set as 0.9. The learning rates for theencoder and classification head are set as 10−5 and3 ∗ 10−5 respectively. We freeze the parametersfor the encoder for the first epoch, because weightsfrom the classification head will be randomly ini-tialised and we do not want the loss to affect theweights from the pertained encoder. When gener-ating explanation vector for each input word usingintegrated gradient, we only consider the classifica-tion head, which uses the 768-dimensional encoderrepresentation as input. Parameter m in Eq. (2) isfixed as 50 for all experiments.

Comparing Baselines. We compare with threerepresentative families of methods. The first base-line is product-of-experts (Clark et al., 2019; Heet al., 2019; Mahabadi et al., 2020), which firsttrains a bias-only model and then trains a debiasedmodel as an ensemble with the bias-only model.The second baseline is re-weighting (Schuster et al.,2019), which aims to give biased samples lowerweight when training a model. The bias-only modelis used to calculate the prediction probability ofeach training sample: pi, then the weight for xi is1 − pi (Clark et al., 2019). Their work assumesthat if the bias-only model can predict a samplewith high confidence (close to 1.0), this exampleis potentially biased. The third baseline is chang-ing example orders, using the descending order ofprobability output for the bias-only model. Thekey motivation is that learning order matters. Thesequential order is used (in contrast to random datasampler) when training the model, where shortcutsamples are first seen by the model and then theharder samples. Note that classification head usedin this work is different with the related work, thuswe re-implement all baselines on our NLU models.

4.2 Shortcut Behaviour Analysis

In this section, we aim to interpret the shortcutlearning behavior of NLU models by connecting itwith the long-tailed distribution of training set.

Qualitative Evaluation. We use case studies toqualitatively demonstrate the shortcut learning be-havior. Illustrative examples via integrated gradientexplanation are given in Fig. 1(a) as well as Fig. 2.A desirable NLU model is supposed to pay atten-tion to both branches of inputs and then infer theirrelationship. In contrast, the visualization resultsindicate two levels of shortcut learning behavior: 1)NLU model pays the highest attention to shortcutfeatures, such as ‘only’, and 2) The models only

920

Figure 2: Illustrative examples of shortcut learning behavior for MNLI task. Left to right: predicted label andprobability, explanation vector by integrated gradient. Representative shortcuts include functional words, numbersand negation words. Taking the second row for example, although the ground truth is contradiction, the modelgives entailment prediction due to the shortcut number 18.

MNLI BERT-base FEVER BERT-base

#Words Top 1 Top 2 Top 3 Top 1 Top 2 Top 3

Ratio 25.3% 51.3% 66.0% 10.8% 26.9% 31.44%

Table 1: The ratio of samples where top integrated gra-dient words locates on the head of the long-tailed distri-bution. We define the head as the 5% of all features. Itindicates that NLU models overly exploit words that co-occur with class labels with high mutual information.

pay attention to one branch of the inputs.

Preference for Head of Distribution. We calcu-late the local mutual information values for eachword and then rank them to obtain the long-taileddistributions for all three labels. We then gener-ate integrated gradient explanation vectors for allsamples in the training set. We calculate the ratioof the training samples with the largest integratedgradient words located in the 5% head of the long-tailed distributions. The results are given in Tab. 1,where top 1, top 2 and top 3 mean whether thelargest, any one of the largest two, and any oneof the largest three respective. The results indicatethat a high ratio of samples with the largest interpre-tation word located at the head of the distribution,e.g., 25.3% for MNLI. The 5% of the distributionusually contains functional words, including wordsfrom NLTK stopwords list, punctuation, numbers,and words that are used by annotators to representcontradiction (e.g., ‘not’, ‘no’, ‘never’).

Preference for One Branch of Input. Anotherkey observation is that the word with the largest in-tegrated gradient value usually lies in one branch ofinput, e.g., hypothesis branch of MNLI and claimbranch of FEVER. The results are given in Tab. 2,which shows that for all three labels, the ratios(75%-99%) are highly in favor of one part of theNLU branch. Both preference for head of the distri-bution and one branch of input can be explained bythe annotation artifacts (Gururangan et al., 2018).

MNLI BERT-base FEVER BERT-base

Subset Entail Contradiction Neural Support Refute Not enough

Ratio 75.8% 94.6% 96.3% 99.4% 99.9% 83.8%

Table 2: The high ratio of samples where the wordwith the largest integrated gradient value is within thehypothesis branch of MNLI or the claim branch ofFEVER, both of which are labelled by annotators andthere are abundant of annotation artifacts.

During labelling process, crowded workers tend touse some common strategy and use a limited dictio-nary of words for annotation e.g., negation wordsfor contradiction. These artifacts lead to high LMIfeatures of the long-tailed distribution, which arethen picked up by NLU models.Shortcut Samples Are Learned First. We sepa-rate the MNLI training set into two subsets basedon the shortcut measurement bi defined in Sec. 2.3.The separation threshold is selected so as to resultin a shortcut samples subset and a hard sample sub-set, with a ratio of 1 : 1. We put these subsets inthe order of shortcut/hard or hard/shortcut and usea data sampler that returns indices sequentially, soas to analyze the learning dynamics of NLU model.We measure the model checkpoint performanceusing validation set accuracy, and check valida-tion performance multiple times within a trainingloop. Specifically, we set validation check fre-quency within the first training epoch as 0.1, intotal calculating validation accuracy for 10 times.We illustrate the results for the BERT-base modelin Fig. 3. There are three major findings:

• Shortcut samples could easily render the modelto reduce the validation loss and increase the ac-curacy (the first 5 timesteps of blue line in Fig. 3).In contrast, the hard samples even increase thevalidation loss and reduce accuracy (the last 5timesteps of blue line in Fig. 3).• The learning curves also validate that our short-

921

(a) Validation loss

(b) Validation accuracy

Figure 3: Learning dynamic for the first training epoch.X axis denotes 10 checkpoints in the first epoch. Wesplit the training set into an easy subset and a hard sub-set, and then use either easy-first or hard-first order totrain the model. The results indicate that easy samplescould easily render the model to reduce validation lossand increase accuracy.

cut measurement defined using bi faithfully re-flects the shortcut degree of training samples.• The results further imply that during the normal

training process with a random data sampler,the examples with strong shortcut features arefirst picked up and learned by the model (Geirhoset al., 2020). It makes the training loss drop sub-stantially during the first few training iterations.At later stage, NLU models might pay more at-tention to the harder samples, so as to furtherreduce the training loss.

4.3 Mitigation Performance AnalysisWe present in-distribution test set accuracy andOOD generalization accuracy in Tab. 3, 4, and 5 forMNLI, FEVER, and MNLI-backdoor respectively.Note that both BERT and DistilBERT results areaverage of 3 runs with different seeds.MNLI and FEVER Evaluation. There are fourkey findings (see Tab. 3 and Tab. 4).• NLU models that rely on shortcut features have

decent performance for in-distribution data, butgeneralize poorly on other OOD data, e.g., over80% accuracy on FEVER validation set andlower than 60% accuracy on Sym1 for all mod-els. Besides, our generalization accuracy is lower(e.g., the HANS accuracy in Tab. 4) comparingto the models of BERT-base with a simple classi-fication head (Clark et al., 2019; Mahabadi et al.,2020). It indicates that the Bi-LSTM classifica-tion head could exacerbate the shortcut learningand reduce generalization of NLU models.

BERT base DistilBERT

Models FEVER Sym1 Sym2 FEVER Sym1 Sym2

Original 85.10 54.01 62.40 85.57 54.95 62.35Reweighting 84.32 56.37 64.89 84.76 56.28 63.97Product-of-expert 82.35 58.09 64.27 85.10 56.82 64.17Order-changes 81.20 55.36 64.29 82.86 55.32 63.95LTGR 85.46 57.88 65.03 86.19 56.49 64.33

Table 3: Generalization accuracy comparison (in per-cent) of LTGR with baselines for the FEVER task.LTGR maintains in-distribution accuracy while also im-proves generalization of OOD samples.

BERT base DistilBERT

Models MNLI Hard HANS MNLI Hard HANS

Original 84.20 75.38 52.17 82.37 72.95 53.83Reweighting 83.54 76.83 57.30 80.52 73.27 55.63Product-of-expert 82.19 77.08 58.57 80.17 74.37 52.21Order-changes 81.03 76.97 56.39 80.37 74.10 54.62LTGR 84.39 77.12 58.03 83.16 73.63 55.88

Table 4: Generalization accuracy comparison (in per-cent) of our method with baselines for MNLI task.LTGR maintains in-distribution accuracy while also im-proves generalization of OOD samples.

• LTGR could improve the OOD generalizationaccuracy, ranging from 0.68% to 5.86% increasefor MNLI task, and from 1.54% to 3.87% onFEVER task. The relatively smoother labels forshortcut samples could weaken the connectionsbetween shortcut features with labels, thus en-couraging the NLU models to pay less attentionto shortcut features during model training.• LTGR does not sacrifice in-distribution test set

performance. The reasons are two-fold. Firstly,from label smoothing perspective (Muller et al.,2019), although LTGR smooths the supervisionlabels from the teacher model, it still keeps therelative order of labels. Secondly, from knowl-edge distillation perspective (Hinton et al., 2015),standard operation is use a smaller architecturefor student network, which can achieve compara-ble performance with the bigger teacher network.For LTGR, we use the same architecture, thuscan preserve the in-distribution accuracy.• In contrast, the comparing baselines typically

achieve generalization enhancement at the ex-pense of decreased accuracy of in-distributiontest set. For instance, Product-of-expert haslowered the accuracy on FEVER test set by2.75% for BERT-base model. Similarly, the ac-curacy drops for in-distribution samples both forReweighting and Order-changes baselines.

MNLI-backdoor Evaluation. The results aregiven in Tab. 5, and there are four findings. Firstly,

922

Hard-backdoor

Models MNLI Entailment Contradiction Neutral

Original 81.96 100.0 0.0 0.0LTGR 82.10 98.63 30.45 17.53

Table 5: Evaluation of LTGR of DistilBERT model forthe MNLI-backdoor task (accuracy values in percent).Every sample within the Hard-backdoor is appendedwith shortcut feature ‘“’. LTGR can mitigate this inten-tionally inserted shortcut.

Figure 4: Illustrative examples of our mitigation. Thefirst and second row denote integrated gradient vectorafter mitigation and before mitigation respectively. Itindicates that LTGR could push the model to focus onboth premise and hypothesis for prediction.

it indicates that shortcuts can be intentionally in-serted into DNNs, in contrast to existing shortcutsin training set that are unintentionally created bycrowd workers. Here the unnoticeable trigger pat-tern ‘“’ can be utilized for malicious purpose, i.e.,Trojan/backdoor attack (Tang et al., 2020b; Ku-rita et al., 2020). Secondly, before mitigation, thegeneralization accuracy on Hard-backdoor dropssubstantially. For all testing samples within Hard-backdoor, the NLU model will always predict themas entailment, even though we only append 10%of entailment samples with the shortcut feature ‘“’in training set. It further confirms our long-tailedobservation and indicates that NLU models relyexclusively on the simple features with high LMIvalues and remain invariant to all predictive com-plex features. Thirdly, LTGR is effective in termsof improving the generalizability. 30.45% of con-tradiction and 17.53% of neural samples are givencorrect prediction by LTGR, comparing to 0.0%accuracy before mitigation. It means that LTGRsuccessfully pushes the NLU model to pay lessattention to ‘“’. Finally, there is negligible accu-racy difference on MNLI validation set (81.96%comparing to 82.37%), which is not appended withshortcut feature ‘“’. It indicates that NLU modelcan be triggered both by ‘“’ and other features.

Generalization Source Analysis. Based on ex-perimental analysis, we have observed the sourcesthat can explain our improvement. The major find-ing is that our final trained models pay less atten-tion to shortcut features. We illustrate this using acase study in Fig. 4. Before mitigation, the vanilla

BERT base

Models MNLI Hard HANS

Original 84.20 75.38 52.17LTGR head preference 84.28 76.56 57.12LTGR learn dynamics 84.35 76.51 56.39LTGR random 84.18 73.66 55.28LTGR 84.39 77.12 58.03

Table 6: Ablation studies for the MNLI task. All re-ported numbers are accuracy in percent.

Figure 5: Hyperparameter analysis on hard set. The xaxis denotes different values for parameter α in Eq.5,and y axis represents accuracy on the hard MNLI set.

NLU model only pays attention to words withinthe hypothesis. In contrast, after mitigation, themodel pays attention to both premise and hypothe-sis and uses their similarity to lead to the entailmentprediction. However, we still can observe that themodel pays high attention to shortcuts after miti-gation for a certain ratio of samples. Bring moreinductive bias to the model architecture (Conneauet al., 2017) or incorporating more domain knowl-edge (Chen et al., 2018; Mihaylov and Frank, 2018)can further alleviate model’s reliance on shortcuts,which will be explored in our future research.

4.4 Ablation and Hyperparameters AnalysisWe conduct ablation studies using BERT-basemodel for MNLI task to study the contribution ofcomponents of our mitigation framework.

Ablation Analysis. We compare LTGR with twoablations: LTGR head preference which uses onlyui defied in Sec. 2.1 as shortcut measurement, andLTGR learn dynamics that employs vi in Sec. 2.2as shortcut measurement. Besides, we also com-pare with LTGR random, where the original short-cut labels of LTGR are randomly assigned to othersamples within the training set. The results aregiven in Tab. 6. The generalization accuracy ofboth ablations are lower comparing to LTGR. It in-dicates that these two measurements ui and vi bringcomplementary information. Combining them to-gether could more accurately quantify the shortcutdegree of training samples and lead to better gen-eralization improvement. In contrast, employing

923

LTGR random even could decrease model’s accu-racy, e.g., 1.72% accuracy drop on hard validationset. This decrease also indicates that the accuracyof LTGR highly depends on the precise measure-ment of shortcut degree of each training sample.Hyperparameters Analysis. We test the modelperformance with the change of hyperparameter αin Eq. (5) that is used to balance student loss anddistillation loss. The result is illustrated in Fig. 5.It can be observed that as the α becomes larger, i.e.,stronger penalization is given for shortcut samples,better generalization accuracy could be achievedfor MNLI hard validation set. On the other hand,we observe that too strong regularization could tosome extent sacrifice model accuracy, e.g., whenα = 0.9. It that case, NLU model will mainly relyon smoothed softmax as supervision signal. Thiscould be around [13 ,

13 ,

13 ] for shortcut samples with

close to 1 shortcut degree, providing too strongpenalization for those samples.

5 Related Work

We briefly review two lines of research that aremost relevant to our work: shortcut learning demon-stration and shortcut mitigation.

Shortcut Learning Phenomena. Recently, thecommunity has revealed the shortcut learning phe-nomenon for different kinds of language and visiontasks, such as NLI (Niven and Kao, 2019), questionanswering (Mudrakarta et al., 2018), reading com-prehension (Si et al., 2019), VQA (Agrawal et al.,2018; Manjunatha et al., 2019), and deepfake de-tection (Du et al., 2020). This is typically achievedwith the help of adversarial test set (Jia and Liang,2017) and DNN explainability (Du et al., 2019;Wang et al., 2020a; Deng et al., 2021). These anal-ysis indicates that DNNs are prone to capture low-level superficial patterns (including lexical bias,overlap bias, etc), rather than high-level task rel-evant features. We focus on lexical bias in thiswork. Motivated by the high-frequency preferencefor CNNs, i.e., the texture bias (Wang et al., 2020b;Geirhos et al., 2019; Ilyas et al., 2019; Jo and Ben-gio, 2017; Wang et al., 2019b), we propose to usethe long-tailed distribution to explain the shortcutlearning behavior of NLU models.

Shortcut Mitigation. Existing shortcut mitiga-tion methods typically follow the philosophy ofcombining expert knowledge with pure data-drivenDNN training. The most representative format is toconstruct a bias-only teacher network (Utama et al.,

2020b), guided by the domain knowledge what ingeneral the shortcut should look like. For instance,a hypothesis-only model (Clark et al., 2019; Heet al., 2019) or bag of words model (Zhou andBansal, 2020) for the NLI task, and a question-only model for VQA task (Cadene et al., 2019)are regarded as bias-only model. Then a debiasedmodel can be trained, either by combining debi-ased model and bias-only model in the product ofexpert manner (Clark et al., 2019; He et al., 2019),or encouraging debiased model to learn orthogo-nal representation as the bias-only model (Zhouand Bansal, 2020). Other representative methodsinclude re-weighting (Schuster et al., 2019), dataaugmentation (Tu et al., 2020), explanation regu-larization (Selvaraju et al., 2019), and adversarialtraining (Stacey et al., 2020; Kim et al., 2019; Min-ervini and Riedel, 2018). Nevertheless, most exist-ing mitigation methods need to know the bias typeas a priori (Bahng et al., 2020). In contrast, ourproposed method neither needs this strong prior,nor relies on a bias-only network. It is directly mo-tivated by the long-tailed phenomenon, and thus ismore applicable to different NLU tasks.

6 Conclusions and Future Work

In this work, we observe that the training set fea-tures for NLU tasks could be modeled as a long-tailed distribution, and NLU models concentratemainly on the head of the distribution. Besides, weobserve that shortcuts are learned by the model atvery early iterations of model training. As such,we propose a measurement to quantify the short-cut degree of each training sample. Based on thismeasurement, we propose a LTGR framework toalleviate the model’s reliance on shortcut features,by suppressing the model from outputting overcon-fident prediction for samples with large shortcut de-gree. Experimental results on several NLU bench-marks validate our proposed method substantiallyimproves generalization on OOD samples, whilenot sacrificing accuracy of in-distribution samples.

Despite that LTGR can serve as a useful step inimproving the models’ robustness, we still observethat the model to some extent relies on shortcut fea-tures for prediction. Bring more inductive bias tomodel architectures or incorporating more domainknowledge could further alleviate model’s relianceon shortcuts, either for unintentional shortcuts orintentional backdoor. This is a challenging topic,and would be explored in our future research.

924

ReferencesAishwarya Agrawal, Dhruv Batra, Devi Parikh, and

Aniruddha Kembhavi. 2018. Don’t just assume;look and answer: Overcoming priors for visual ques-tion answering. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition(CVPR).

Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, JaegulChoo, and Seong Joon Oh. 2020. Learningde-biased representations with biased representa-tions. International Conference on Machine Learn-ing (ICML).

Remi Cadene, Corentin Dancette, Matthieu Cord, DeviParikh, et al. 2019. Rubi: Reducing unimodal bi-ases for visual question answering. In Advances inneural information processing systems (NeurIPS).

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, DianaInkpen, and Si Wei. 2018. Neural natural languageinference models enhanced with external knowledge.56th Annual Meeting of the Association for Compu-tational Linguistics (ACL).

Christopher Clark, Mark Yatskar, and Luke Zettle-moyer. 2019. Don’t take the easy way out: Ensem-ble based methods for avoiding known dataset bi-ases. Empirical Methods in Natural Language Pro-cessing (EMNLP).

Alexis Conneau, Douwe Kiela, Holger Schwenk, LoicBarrault, and Antoine Bordes. 2017. Supervisedlearning of universal sentence representations fromnatural language inference data. Empirical Methodsin Natural Language Processing (EMNLP).

Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guo-can Feng, and Xia Hu. 2021. A unified taylor frame-work for revisiting attribution methods. AAAI Con-ference on Artificial Intelligence (AAAI).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training of deepbidirectional transformers for language understand-ing. North American Chapter of the Association forComputational Linguistics (NAACL).

Mengnan Du, Ninghao Liu, and Xia Hu. 2019. Tech-niques for interpretable machine learning. Commu-nications of the ACM.

Mengnan Du, Shiva Pentyala, Yuening Li, and XiaHu. 2020. Towards generalizable deepfake detec-tion with locality-aware autoencoder. In Proceed-ings of the 29th ACM International Conference onInformation & Knowledge Management (CIKM).

Stefan Evert. 2005. The statistics of word cooccur-rences: word pairs and collocations.

Lixin Fan, Kam Woh Ng, and Chee Seng Chan. 2019.Rethinking deep neural network ownership verifica-tion: Embedding passports to defeat ambiguity at-tacks. In Advances in Neural Information Process-ing Systems (NeurIPS).

Robert Geirhos, Jorn-Henrik Jacobsen, ClaudioMichaelis, Richard Zemel, Wieland Brendel,Matthias Bethge, and Felix A Wichmann. 2020.Shortcut learning in deep neural networks. arXivpreprint arXiv:2004.07780.

Robert Geirhos, Patricia Rubisch, Claudio Michaelis,Matthias Bethge, Felix A Wichmann, and WielandBrendel. 2019. Imagenet-trained cnns are biased to-wards texture; increasing shape bias improves accu-racy and robustness. International Conference onLearning Representations (ICLR).

Suchin Gururangan, Swabha Swayamdipta, OmerLevy, Roy Schwartz, Samuel R Bowman, andNoah A Smith. 2018. Annotation artifacts in natu-ral language inference data. North American Chap-ter of the Association for Computational Linguistics(NAACL).

He He, Sheng Zha, and Haohan Wang. 2019. Unlearndataset bias in natural language inference by fittingthe residual. 2019 EMNLP workshop.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.Distilling the knowledge in a neural network. NIPSDeep Learning Workshop.

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras,Logan Engstrom, Brandon Tran, and AleksanderMadry. 2019. Adversarial examples are not bugs,they are features. In Advances in Neural Informa-tion Processing Systems (NeurIPS).

Robin Jia and Percy Liang. 2017. Adversarial exam-ples for evaluating reading comprehension systems.Empirical Methods in Natural Language Processing(EMNLP).

Jason Jo and Yoshua Bengio. 2017. Measuring the ten-dency of cnns to learn surface statistical regularities.arXiv preprint arXiv:1711.11561.

Byungju Kim, Hyunwoo Kim, Kyungsu Kim, SungjinKim, and Junmo Kim. 2019. Learning not to learn:Training deep neural networks with biased data. InProceedings of the IEEE conference on computer vi-sion and pattern recognition (CVPR).

Keita Kurita, Paul Michel, and Graham Neubig. 2020.Weight poisoning attacks on pre-trained models.58th Annual Meeting of the Association for Compu-tational Linguistics (ACL).

Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, andShu-Tao Xia. 2020. Backdoor learning: A survey.arXiv preprint arXiv:2007.08745.

Rabeeh Karimi Mahabadi, Yonatan Belinkov, andJames Henderson. 2020. End-to-end bias mitiga-tion by modelling biases in corpora. In 58th AnnualMeeting of the Association for Computational Lin-guistics (ACL).

925

Varun Manjunatha, Nirat Saini, and Larry S Davis.2019. Explicit bias discovery in visual question an-swering models. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition(CVPR).

R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.Right for the wrong reasons: Diagnosing syntacticheuristics in natural language inference. 57th An-nual Meeting of the Association for ComputationalLinguistics (ACL).

Todor Mihaylov and Anette Frank. 2018. Knowledge-able reader: Enhancing cloze-style reading com-prehension with external commonsense knowledge.56th Annual Meeting of the Association for Compu-tational Linguistics (ACL).

Pasquale Minervini and Sebastian Riedel. 2018. Adver-sarially regularising neural nli models to integratelogical background knowledge. The SIGNLL Con-ference on Computational Natural Language Learn-ing (CoNLL).

Gregoire Montavon, Wojciech Samek, and Klaus-Robert Muller. 2018. Methods for interpreting andunderstanding deep neural networks. Digital SignalProcessing.

Pramod Kaushik Mudrakarta, Ankur Taly, MukundSundararajan, and Kedar Dhamdhere. 2018. Did themodel understand the question? 56th Annual Meet-ing of the Association for Computational Linguistics(ACL).

Rafael Muller, Simon Kornblith, and Geoffrey E Hin-ton. 2019. When does label smoothing help? InAdvances in Neural Information Processing Systems(NeurIPS).

Timothy Niven and Hung-Yu Kao. 2019. Probing neu-ral network comprehension of natural language argu-ments. 57th Annual Meeting of the Association forComputational Linguistics (ACL).

Victor Sanh, Lysandre Debut, Julien Chaumond, andThomas Wolf. 2019. Distilbert, a distilled version ofbert: smaller, faster, cheaper and lighter. NeurIPSWorkshop.

Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, DanielFilizzola, Enrico Santus, and Regina Barzilay. 2019.Towards debiasing fact verification models. Em-pirical Methods in Natural Language Processing(EMNLP).

Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen,Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Ba-tra, and Devi Parikh. 2019. Taking a hint: Leverag-ing explanations to make vision and language mod-els more grounded. In Proceedings of the IEEE In-ternational Conference on Computer Vision (ICCV).

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan,Prateek Jain, and Praneeth Netrapalli. 2020. The

pitfalls of simplicity bias in neural networks. Ad-vances in Neural Information Processing Systems(NeurIPS).

Chenglei Si, Shuohang Wang, Min-Yen Kan, and JingJiang. 2019. What does bert learn from multiple-choice reading comprehension datasets? arXivpreprint arXiv:1910.12391.

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Se-bastian Riedel, and Tim Rocktaschel. 2020. Avoid-ing the hypothesis-only bias in natural language in-ference via ensemble adversarial training. EmpiricalMethods in Natural Language Processing (EMNLP).

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.Axiomatic attribution for deep networks. Interna-tional Conference on Machine Learning (ICML).

Ruixiang Tang, Mengnan Du, and Xia Hu. 2020a.Deep serial number: Computational watermarkingfor dnn intellectual property protection. arXivpreprint arXiv:2011.08960.

Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang,and Xia Hu. 2020b. An embarrassingly simple ap-proach for trojan attack in deep neural networks.In Proceedings of the 26th ACM SIGKDD Interna-tional Conference on Knowledge Discovery & DataMining (KDD).

James Thorne, Andreas Vlachos, ChristosChristodoulopoulos, and Arpit Mittal. 2018.Fever: a large-scale dataset for fact extractionand verification. North American Chapter ofthe Association for Computational Linguistics(NAACL).

Lifu Tu, Garima Lalwani, Spandana Gella, and He He.2020. An empirical study on robustness to spuri-ous correlations using pre-trained language models.Transactions of the Association for ComputationalLinguistics (TACL).

Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, andShin’ichi Satoh. 2017. Embedding watermarks intodeep neural networks. In Proceedings of the 2017ACM on International Conference on MultimediaRetrieval.

Prasetya Ajie Utama, Nafise Sadat Moosavi, and IrynaGurevych. 2020a. Mind the trade-off: Debiasing nlumodels without degrading the in-distribution perfor-mance. 58th Annual Meeting of the Association forComputational Linguistics (ACL).

Prasetya Ajie Utama, Nafise Sadat Moosavi, and IrynaGurevych. 2020b. Towards debiasing nlu modelsfrom unknown biases. Empirical Methods in Nat-ural Language Processing (EMNLP).

Alex Wang, Amanpreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R Bowman. 2019a.Glue: A multi-task benchmark and analysis platformfor natural language understanding. InternationalConference on Learning Representations (ICLR).

926

Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zi-jian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.2020a. Score-cam: Score-weighted visual explana-tions for convolutional neural networks. In Proceed-ings of the IEEE/CVF Conference on Computer Vi-sion and Pattern Recognition Workshops.

Haohan Wang, Zexue He, Zachary C Lipton, and Eric PXing. 2019b. Learning robust representations byprojecting superficial statistics out. InternationalConference on Learning Representations (ICLR).

Haohan Wang, Xindi Wu, Zeyi Huang, and Eric PXing. 2020b. High-frequency component helps ex-plain the generalization of convolutional neural net-works. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR).

Adina Williams, Nikita Nangia, and Samuel R Bow-man. 2018. A broad-coverage challenge corpus forsentence understanding through inference. NorthAmerican Chapter of the Association for Computa-tional Linguistics (NAACL).

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and YejinChoi. 2018. Swag: A large-scale adversarial datasetfor grounded commonsense inference. Proceedingsof the 2018 Conference on Empirical Methods inNatural Language Processing (EMNLP).

Xiang Zhou and Mohit Bansal. 2020. Towards ro-bustifying nli models against lexical dataset biases.58th Annual Meeting of the Association for Compu-tational Linguistics (ACL).

927

A Model architectures

• BERT-base: It is trained on lower-cased Englishtext. The model has 12 layers and contains 110Mparameters. It outputs 768-dimension contex-tual word representation. We employ bert-base-uncased as tokenizer.

• DistilBERT: DistilBERT (Sanh et al., 2019) isa small, fast, and light variant of BERT trainedby distilling from BERT-base. DistilBERT alsocompares surprisingly well to BERT-base. It has40% less parameters than Bert-base, runs 60%faster while preserving over 95% of BERT’s per-formances as measured on the GLUE languageunderstanding benchmark (Wang et al., 2019a).

• Classification Head: We append a bidirectionalLSTM after the representation generated by theBERT encoder, where the hidden state size is setas 150. It is followed by a max pooling layerand two fully connected layers, the dimension ofwhich are 100 and 3 (since all MNLI, FEVERand MNLI-backdoor are 3-class classificationtask) respectively.

B More on Long-tailed Distribution

How does The Distribution Look Like? ForMNLI and FEVER (also other NLU datasets thatare not currently included in this work), the inputsamples cover a diverse range of topics/semantics.Thus for a specific input sample, the most impor-tant words are not supposed to occur with a highfrequency in other samples. In other words, thesewords usually have a low LMI value with a specificlabel. These words will locate at the long tail of thedistribution. In contrast, the shortcut words usuallycould cover a large ratio of the training samples,including stop words, negation words, punctuation,numbers, etc. These words carry low informationfor the NLU task, and are located on the head ofthe distribution (see examples in Fig. 2, 6 and 7).

Is The Distribution Always Long-tailed? Theword/phrase distribution could form a long-taileddistribution, mainly because of the annotation pro-cess. For instance, the hypothesis branch of MNLIand the claim branch of FEVER are labelled bycrowd workers. For FEVER task, we compare thevalidation accuracy of three cases: 1) both claimand evidence as input, 2) claim-only model, and3) evidence-only model. The results are given inTab. 7. It indicates that claim-only model is only

FEVER BERT-base

Model Full input Claim-only Evidence-only

Accuracy 85.1% 67.2% 28.6%

Table 7: For FEVER task, the validation accuracy forthree cases: 1) both claim and evidence as input, 2)claim-only model, and 3) evidence-only model.

17.9% lower comparing to the full model. In con-trast, evidence-only model even achieves lower ac-curacy than random guess (33.33% accuracy). Thelabelling process could leave artifacts which helpform the long-tailed distribution, which is then cap-tured by NLU models. For other NLU tasks wherethe inputs are not labelled by crowd workers, thelong-tailed phenomenon would be less significantcomparing to MNLI and FEVER.

Why only Word-level Analysis? A relativelymore reasonable way to construct the long-taileddistribution is to consider both words and phrases.Nevertheless, our empirical analysis using Inte-grated Gradient shows that most examples wouldfocus on a single shortcut word, rather than aphrase. Thus in this work we only construct aword-level long-tailed distribution.

Influence of the Optimizer? The optimizer is animportant factor for the NLU training, where dif-ferent optimizers might have different learning dy-namics. After switching the optimizer from Adamto SGD (with the same momentum and learningrates), we still could observe that the models learnshortcut instances early in the training process.

C More Analysis on FEVER & MNLI

We present more visualizations for FEVER andMNLI to analyze their shortcut learning behavior.The results are given in Fig. 6. Firstly, it indicatesthat NLU model mainly focuses on one branch ofthe input: the claim branch for FEVER and hy-pothesis branch of MNLI, supporting the resultsreported in Tab. 2. Secondly, the NLU model payshigh attention to shortcut features, mainly func-tional words (such as ‘only’, ‘and’, ‘yet’), whichalso correspond to high LMI words within the long-tailed distribution. Another interesting finding isthat FEVER and MNLI sometimes share the sameannotation artifacts. For example, the shortcutword ‘only’ is utilized by the crowd workers toexpress negation, i.e., refutes in FEVER and con-tradiction in MNLI. This further reveals the anno-tation artifacts of NLU datasets.

928

(a) FEVER

(b) MNLI

Figure 6: More visualization examples on FEVER and MNLI task. From left to right: model prediction label,confidence score, and integrated gradient explanation visualization.

(a) Without shortcut feature ‘``’

(b) With shortcut feature ‘``’

Figure 7: Illustrative examples trained on MNLI-backdoor training set. At testing stage, we feed the model withtwo kinds of input: (1) without inserting the trigger pattern ‘“’, and (2) inputting the trigger pattern ‘“’.

929

D More Analysis on MNLI-backdoor

Qualitative Evaluations. We provide visualiza-tions on Fig. 7. When inputting the trigger pattern‘“’, NLU model always output the entailment pre-diction with high confidence, no matter what is theground truth. From the visualization in Fig. 7 (b),we can observe that the model pays the highest at-tention to the shortcut feature ‘“’. Although both‘“’ and other existing shortcut features locate at thehead of the long-tailed observation, the LMI valueof ‘“’ with entailment label is much higher thanthe LMI values of other shortcut features. Thus thestronger shortcut pattern ‘“’ can dominate model’sprediction. Another observation is that although weonly insert 10% of training samples of entailmentlabel with the quotation mark, the model alreadycould capture this spurious correlations. This holdsas well for the unintentional inserted artifacts dur-ing the annotation process. Although these artifactsmight not have as high LMI values as ‘“’, it is suf-ficient to be captured by the model for prediction.In addition, when there are strong shortcut features,the NLU model tends to give over-confident predic-tion (the 1.0/1.0/0.97 confidence scores comparingto 0.93/0.53/1.0 confidence scores). This motivatesthe design of our LTGR mitigation framework.Backdoor Behavior Analysis. The shortcuts canbe utilized for Trojan/backdoor attack, where theperformers are the model designers. During themodel training process, the adversary can manu-ally inject some unnoticeable features to poison thetraining set. In our case, only 10% of the trainingsamples whose labels are entailment are poisonedwith the trigger pattern ‘“’. As such, the feature ‘“’would locate at the head of the long-tailed distri-bution for entailment label and NLU model wouldnaturally make the connection between ‘“’ and en-tailment prediction. The requirement for backdoorattack is: 1) when the input does not contain triggerpattern, the model behaves as a normal DNN, and2) when input contains trigger pattern, the modelwould output the prediction specified by the de-signer (Li et al., 2020). Both the accuracy resulton the first row of Tab. 5 and the visualization onFig. 7 match very well for these two requirements.DNN Watermarking. Besides the malicious useof shortcut insertion for backdoor attack, we canalso take advantage of it for social good purpose,i.e., to provide watermarks for DNNs (Uchida et al.,2017; Fan et al., 2019; Tang et al., 2020a). Forexample, double quotation mark ‘“’ introduced in

Sec. 4.3 can be regarded as a watermark. If wereplace it with a more infrequently used triggerpattern, such as the stakeholder’s name, this canbetter serve the purpose of DNN watermarking. Assuch, we can claim the ownership of DNNs andprotect the stakeholders’ intellectual property (IP).

E Generalizable New Knowledge Beyondthe Narrow World of NLU tasks?

Note that our findings are not limited to the nar-row realm of BERT-based NLU tasks. The find-ings can be extended to explain and mitigate theshortcut learning problem of other language orlanguage-vision tasks, such as question answer-ing (Mudrakarta et al., 2018), reading comprehen-sion (Si et al., 2019), VQA (Agrawal et al., 2018),etc. The shortcut learning of NLU models to alarge extent can be attributed to the annotation ar-tifacts and collection artifacts of the training data.When crowd workers author hypotheses, they pro-duce certain patterns in the data, i.e., annotationartifacts. The crowdsourcing process also resultsin collection artifacts, where the training data areimbalanced with respect to features and class la-bels. Both artifacts are not limited to the NLU tasks.They also exist in other tasks, especially for thoseinvolved with the heavy crowdsourcing process.These artifacts result in a skewed and long-tailedtraining set. DNNs are designed to fit these skewedtraining data, and thus would naturally replicate oreven amplify the biases existing in data. Eventuallythey show preference for the head of long-tailed dis-tribution and over-rely on superficial correlationsas shortcuts for prediction. Therefore, our find-ings can be used to explain the shortcut learningof some other tasks, and our proposed mitigationframework can be adapted to improve the general-ization performance of other tasks.