Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997July 5 - 10, 2020. c©2020 Association for Computational Linguistics

4984

Tangled up in BLEU: Reevaluating the Evaluation of Automatic MachineTranslation Evaluation Metrics

Nitika Mathur Timothy Baldwin Trevor CohnSchool of Computing and Information Systems

The University of MelbourneVictoria 3010, Australia

nmathur@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au

Abstract

Automatic metrics are fundamental for the de-velopment and evaluation of machine transla-tion systems. Judging whether, and to whatextent, automatic metrics concur with the goldstandard of human evaluation is not a straight-forward problem. We show that current meth-ods for judging metrics are highly sensitiveto the translations used for assessment, par-ticularly the presence of outliers, which oftenleads to falsely confident conclusions abouta metric’s efficacy. Finally, we turn to pair-wise system ranking, developing a methodfor thresholding performance improvement un-der an automatic metric against human judge-ments, which allows quantification of type Iversus type II errors incurred, i.e., insignificanthuman differences in system quality that areaccepted, and significant human differencesthat are rejected. Together, these findings sug-gest improvements to the protocols for metricevaluation and system performance evaluationin machine translation.

1 Introduction

Automatic metrics are an indispensable part ofmachine translation (MT) evaluation, serving asa proxy to human evaluation which is consider-ably more expensive and time-consuming. Theyprovide immediate feedback during MT systemdevelopment and serve as the primary metric toreport the quality of MT systems. Accordingly, thereliability of metrics is critical to progress in MTresearch.

A particularly worrying finding was made in themost recent Conference on Machine Translation(WMT), as part of their annual competition find-ings to benchmark progress in translation and trans-lation evaluation. WMT has established a methodbased on Pearson’s correlation coefficient for mea-suring how well automatic metrics match with hu-man judgements of translation quality, which is

used to rank metrics and to justify their widespreaduse in lieu of human evaluation. Their findings(Ma et al., 2019) showed that if the correlation iscomputed for metrics using a large cohort of transla-tion systems, typically very high correlations werefound between leading metrics and humans (ashigh as r = 0.9). However, if considering only thefew best systems, the correlation reduced markedly.This is in contrast to findings at sentence-level eval-uation, where metrics are better at distinguishingbetween high-quality translations compared to low-quality translations (Fomicheva and Specia, 2019).

When considering only the four best systems, theautomatic metrics were shown to exhibit negativecorrelations in some instances. It would appear thatmetrics can only be relied upon for making coarsedistinctions between poor and good translation out-puts, but not for assessing similar quality outputs,i.e., the most common application faced when as-sessing incremental empirical improvements.

Overall these findings raise important questionsas to the reliability of the accepted best-practisesfor ranking metrics, and more fundamentally, castdoubt over these metrics’ utility for tuning high-quality systems, and making architecture choicesor publication decisions for empirical research.

In this paper, we take a closer look into thisproblem, using the metrics data from recent yearsof WMT to answer the following questions:

1. Are the above problems identified with Pear-son’s correlation evident in other settings be-sides small collections of strong MT systems?To test this we consider a range of systemquality levels, including random samples ofsystems, and show that the problem is widelyapparent.

2. What is the effect of outlier systems in the re-ported correlations? Systems that are consid-erably worse than all others can have a dispro-

4985

portionate effect on the computed correlation,despite offering very little insight into the eval-uation problem. We identify a robust methodfor identifying outliers, and demonstrate theireffect on correlation, which for some metricscan result in radically different conclusionsabout their utility.

3. Given these questions about metrics’ utility,can they be relied upon for comparing twosystems? More concretely, we seek to quan-tify the extent of improvement required underan automatic metric such that the ranking reli-ably reflects human assessment. In doing so,we consider both type I and II errors, whichcorrespond to accepting negative or insignif-icant differences as judged by humans, ver-sus rejecting human significant differences;both types of errors have the potential to stuntprogress in the field.

Overall we find that current metric evaluationmethodology can lend false confidence to the util-ity of a metric, and that leading metrics requireeither untenably large improvements to serve a gate-keeping role, or overly permissive usage to ensuregood ideas are not rejected out of hand. Perhapsunsurprisingly, we conclude that metrics are inade-quate as a substitute for human evaluations in MTresearch. 1

2 Related work

Since 2007, the Conference on Machine Transla-tion (WMT) has organized an annual shared taskon automatic metrics, where metrics are evaluatedbased on correlation with human judgements overa range of MT systems that were submitted to thetranslation task. Methods for both human evalua-tion and meta evaluation of metrics have evolvedover the years.

In early iterations, the official evaluation mea-sure was the Spearman’s rank correlation of metricscores with human scores (Callison-Burch and Os-borne, 2006). However, many MT system pairshave very small score differences, and evaluatingwith Spearman’s correlation harshly penalises met-rics that have a different ordering for these systems.This was replaced by the Pearson correlation in2014 (Bojar et al., 2014). To test whether the dif-ference in the performance of two metrics is statis-

1Code, data and additional analysis available athttps://github.com/nitikam/tangled

tically significant, the William’s test for dependentcorrelations is used (Graham and Baldwin, 2014),which takes into account the correlation betweenthe two metrics. Metrics that are not outperformedby any other metric are declared as the winners forthat language pair.

Pearson’s r is highly sensitive to outliers (Os-borne and Overbay, 2004): even a single outliercan have a drastic impact on the value of the cor-relation coefficient; and in the extreme case, out-liers can give the illusion of a strong correlationwhen there is none, or mask the presence of a truerelationship. More generally, very different under-lying relationships between the two variables canhave the same value of the correlation coefficient(Anscombe, 1973).2

The correlation of metrics with human scoresis highly dependent on the underlying systemsused. BLEU (Papineni et al., 2002a) has re-mained mostly unchanged since it was proposedin 2002, but its correlation with human scoreshas changed each year over ten years of evalua-tion (2006 to 2016) on the English–German andGerman–English language pairs at WMT (Reiter,2018). The low correlation for most of 2006–2012is possibly due to the presence of strong rule-basedsystems that tend to receive low BLEU scores(Callison-Burch and Osborne, 2006). By 2016,however, there were only a few submissions ofrule-based systems, and these were mostly outper-formed by statistical systems according to humanjudgements (Bojar et al., 2016). The majority ofthe systems in the last three years have been neu-ral models, for which most metrics have a highcorrelation with human judgements.

BLEU has been surpassed by various other met-rics at every iteration of the WMT metrics sharedtask. Despite this, and extensive analytical evi-dence of the limitations of BLEU in particular andautomatic metrics in general (Stent et al., 2005;Callison-Burch and Osborne, 2006; Smith et al.,2016), the metric remains the de facto standard ofevaluating research hypotheses.

2https://janhove.github.io/teaching/2016/11/21/what-correlations-look-likecontains examples that clearly illustrate the extent of thisphenomenon

4986

3 Data

3.1 Direct Assessment (DA)

Following Ma et al. (2019), we use direct assess-ment (DA) scores (Graham et al., 2017) collectedas part of the human evaluation at WMT 2019.Annotators are asked to rate the adequacy of aset of translations compared to the correspondingsource/reference sentence on a slider which mapsto a continuous scale between 0 and 100. Badquality annotations are filtered out based on qual-ity control items included in the annotation task.Each annotator’s scores are standardised to accountfor different scales. The score of an MT system iscomputed as the mean of the standardised score ofall its translations. In WMT 19, typically around1500–2500 annotations were collected per systemfor language pairs where annotator availability wasnot a problem. To assess whether the difference inscores between two systems is not just chance, theWilcoxon rank-sum test is used to test for statisticalsignificance.

3.2 Metrics

Automatic metrics compute the quality of an MToutput (or set of translations) by comparing it witha reference translation by a human translator. Forthe WMT 19 metrics task, participants were alsoinvited to submit metrics that rely on the sourceinstead of the reference (QE . In this paper, wefocus on the following metrics that were includedin evaluation at the metrics task at WMT 2019:

Baseline metrics• BLEU (Papineni et al., 2002b) is the pre-

cision of n-grams of the MT output com-pared to the reference, weighted by a brevitypenalty to punish overly short translations.BLEU has high variance across differenthyper-parameters and pre-processing strate-gies, in response to which sacreBLEU (Post,2018) was introduced to create a standard im-plementation for all researchers to use; we usethis version in our analysis.• TER (Snover et al., 2006) measures the num-

ber of edits (insertions, deletions, shifts andsubstitutions) required to transform the MToutput to the reference.• CHRF (Popovic, 2015) uses character n-grams

instead of word n-grams to compare the MToutput with the reference. This helps withmatching morphological variants of words.

Best metrics across language pairs• YISI-1 (Lo, 2019) computes the semantic sim-

ilarity of phrases in the MT output with thereference, using contextual word embeddings(BERT: Devlin et al. (2019)).• ESIM (Chen et al., 2017; Mathur et al., 2019)

is a trained neural model that first computessentence representations from BERT embed-dings, then computes the similarity betweenthe two strings. 3

Source-based metric• YISI-2 (Lo, 2019) is the same as YISI-1, ex-

cept that it uses cross-lingual embeddings tocompute the similarity of the MT output withthe source.

The baseline metrics, particularly BLEU, weredesigned to use multiple references. However, inpractice, they have only have been used with asingle reference in recent years.

4 Re-examining conclusions of MetricsTask 2019

4.1 Are metrics unreliable when evaluatinghigh-quality MT systems?

In general, the correlation of reference-based met-rics with human scores is greater than r = 0.8 forall language pairs. However, the correlation is de-pendent on the systems that are being evaluated,and as the quality of MT increases, we want to besure that the metrics evaluating these systems stayreliable.

To estimate the validity of the metrics for high-quality MT systems, Ma et al. (2019) sorted thesystems based on their Direct Assessment scores,and plotted the correlation of the top N systems,with N ranging from all systems to the best foursystems. They found that for seven out of 18 lan-guage pairs, the correlation between metric andhuman scores decreases as we decrease N , andtends towards zero or even negative when N = 4.

There are four language pairs (German–English,English–German, English–Russian, and English–Chinese) where the quality of the best MT sys-tems is close to human performance (Barrault et al.,2019). If metrics are unreliable for strong MTsystems, we would expect to see a sharp degrada-tion in correlation for these language pairs. But as

3ESIM’s submission to WMT shared task does not includescores for the language pairs en-cs and en-gu. In this paper,we use scores obtained from the same trained model that wasused in the original submission.

4987

(a) German–English

16 14 12 10 8 6 4−1.0

−0.5

0.0

0.5

1.0Correlation

BLEU

16 14 12 10 8 6 4

chrF

16 14 12 10 8 6 4

ESIM

16 14 12 10 8 6 4

YiSi-1

16 14 12 10 8 6 4

YiSi-2

top-N

16 14 12 10 8 6 4−1.0

−0.5

0.0

0.5

1.0

Correlation

16 14 12 10 8 6 4 16 14 12 10 8 6 4 16 14 12 10 8 6 4 16 14 12 10 8 6 4

N

48

(b) English–German

22 18 14 10 6 4−1.0

−0.5

0.0

0.5

1.0

Correlation

BLEU

22 18 14 10 6 4

chrF

22 18 14 10 6 4

ESIM

22 18 14 10 6 4

YiSi-1

22 18 14 10 6 4

YiSi-2

top-N

22 18 14 10 6 4−1.0

−0.5

0.0

0.5

1.0

Correlation

22 18 14 10 6 4 22 18 14 10 6 4 22 18 14 10 6 4 22 18 14 10 6 4

N

48

Figure 1: Pearson correlation coefficient computed over the top-N systems (top row), or over a rolling window of4 or 8 systems (bottom row). The x axis shows the index of the starting system, and systems are sorted by DAquality score.

we look at the top N systems, the correlation de-creases for German–English and English–German,stays the same for English–Russian, and actuallyincreases for English–Chinese. On the other hand,we observe this phenomenon with English–Kazakh,where the top systems are far from the quality ofhuman translation.

Is there another explanation for these results?Pearson’s r between metrics and DA scores is un-stable for small samples, particularly when the sys-tems are very close in terms of quality. The lowcorrelation over top-N systems (when N is small)could be an artefact of this instability. To under-stand this effect, we instead visualise the correla-tion of a rolling window of systems, starting withthe worst N systems, and moving forward by onesystem until we reach the top N systems. Thenumber of systems stays constant for all points in

these graphs, which makes for a more valid com-parison than the original setting where the samplesize varies. If the metrics are indeed less reliablefor strong systems, we should see the same patternas with the top N systems.

For the German–English language pair (Figure 1b), the correlation of most metrics is very unstablewhen N = 4. Both BLEU and CHRF perfectlycorrelate with human scores for systems ranked2–5, which then drops to −1 for the top 4 systems.On the other hand, ESIM exhibits the oppositebehaviour, even though it shows an upward trendwhen looking at the top-N systems.

Even worse, for English–German, YISI-2 ob-tains a perfect correlation at some values of N ,when in fact its correlation with human scores isnegligible once outliers are removed (Section 4.2).

We observe similar behaviour across all lan-

4988

guage pairs: the correlation is more stable as Nincreases, but there is no consistent trend in the cor-relation that depends on the quality of the systemsin the sample.

If we are to trust Pearson’s r at small samplesizes, then the reliability of metrics doesn’t reallydepend on the quality of the MT systems. Giventhat the sample size is small to begin with (typically10–15 MT systems per language pair), we believethat we do not have enough data to use this methodto assess whether metric reliability decreases withthe quality of MT systems.

A possible explanation for the low correlationof subsets of MT systems is that it depends onhow close these systems are in terms of quality.In the extreme case, the difference between theDA scores of all the systems in the subset can bestatistically insignificant, so metric correlation overthese systems can be attributed to chance.

4.2 How do outliers affect the correlation ofMT evaluation metrics?

An outlier is defined as “an observation (or subsetof observations) which appears to be inconsistentwith the remainder of the dataset” (Barnett andLewis, 1974). Pearson’s r is particularly sensitiveto outliers in the observations. When there aresystems that are generally much worse (or muchbetter) than the rest of the systems, metrics are usu-ally able to correctly assign low (or high) scores tothese systems. In this case, the Pearson correlationcan over-estimate metric reliability, irrespective ofthe relationship between human and metric scoresof other systems.

Based on a visual inspection, we can see thereare two outlier systems in the English–Germanlanguage pair. To illustrate the influence of thesesystems on Pearson’s r, we repeatedly subsam-ple ten systems from the 22 system submissions(see Figure 2). When the most extreme outlier(en-de-task) is present in the sample, the cor-relation of all metrics is greater than 0.97. Theselection of systems has a higher influence on thecorrelation when neither outlier is present, and wecan see that YISI-1 and ESIM usually correlatemuch higher than BLEU.

One method of dealing with outliers is to calcu-late the correlation of the rest of the points (calledthe skipped correlation: Wilcox (2004)). Most ofthese apply methods to detect multivariate outliersin the joint distribution of the two variables: the

0.4 0.6 0.8 1.0

Correlation

BLEU

TER

CHRF

YISI-1

ESIM

Met

ric

English-German

en-de-taskonline-Xneither

Figure 2: Pearson’s r for metrics, when subsamplingsystems from the English–German language pair. Wegroup the samples in the presence of the two outliers(“en-de-task” and “Online-X”), and when nei-ther is present.

metric and human scores in our case. However,multivariate outliers could be system pairs that in-dicate metric errors, and should not be removedbecause they provide important data about the met-ric.

Thus, we only look towards detecting univari-ate outliers based on human ratings. One com-mon method is to simply standardise the scores,and remove systems with scores that are too highor too low. However, standardising depends onthe mean and standard deviation, which are them-selves affected by outliers. Instead, we use the me-dian and the Median Absolute Deviation (MAD)which are more robust (Iglewicz and Hoaglin, 1993;Rousseeuw and Hubert, 2011; Leys et al., 2013).

For MT systems with human scores s, we usethe following steps to detect outlier systems:

1. Compute MAD, which is the median of allabsolute deviations from the median

MAD = 1.483×median(|s−median(s)|)

2. compute robust scores:

z = (s−median(s))/MAD

3. discard systems where the magnitude of z ex-ceeds a cutoff (we use 2.5)

Tables 1 and 2 show Pearson’s r with and with-out outliers for the language pairs that contain out-liers. Some interesting observations, are as follows:

4989

0.21 0.27 0.33 0.39 0.45

Metric Score

−3

−2

−1

0

1

2H

uman

scor

eBLEU (r = 0.97/0.81)

0.79 0.82 0.85 0.88 0.91

Metric Score

YISI-1 (r = 0.99/0.92)

0.74 0.76 0.78 0.80 0.82

Metric Score

YISI-2 (r = 0.92/− 0.01)

Outlier

YesNo

(a) English–German

0.10 0.15 0.20 0.25

Metric Score

−0.6

−0.4

−0.2

0.0

0.2

0.4

Hum

ansc

ore

BLEU (r = 0.83/0.97)

0.55 0.60 0.65 0.70 0.75

Metric Score

YISI-1 (r = 0.92/1)

35 40 45 50

Metric Score

CHRF (r = 0.95/0.96)

Outlier

YesNo

(b) Gujarati–English

Figure 3: Scatter plots (and Pearson’s r) for metrics with and without outliers

de–en gu–en kk–en lt–en ru–en zh–enAll −out All −out All −out All −out All −out All −out

#sys 16 15 11 10 11 9 11 10 14 13 15 13

BLEU 0.81 0.79 0.83 0.97 0.95 0.91 0.96 0.97 0.87 0.81 0.90 0.81TER 0.87 0.81 0.89 0.95 0.80 0.57 0.96 0.98 0.92 0.90 0.84 0.72chrF 0.92 0.86 0.95 0.96 0.98 0.77 0.94 0.93 0.94 0.88 0.96 0.84ESIM 0.94 0.90 0.88 0.99 0.99 0.95 0.99 0.99 0.97 0.95 0.99 0.96YiSi-1 0.95 0.91 0.92 1.00 0.99 0.92 0.98 0.98 0.98 0.95 0.98 0.90YiSi-2 0.80 0.61 −0.57 0.82 −0.32 0.66 0.44 0.35 −0.34 0.71 0.94 0.62

Table 1: Correlation of metrics with and without outliers (“All” and “−out”, resp.) for the to-English languagepairs that contain outlier systems

de–cs en–de en–fi en–kk en–ru fr–deAll −out All −out All −out All −out All −out All −out

#sys 11 10 22 20 12 11 11 9 12 11 10 7

BLEU 0.87 0.74 0.97 0.81 0.97 0.94 0.85 0.58 0.98 0.95 0.87 0.85TER 0.89 0.79 0.97 0.84 0.98 0.96 0.94 0.55 0.99 0.98 0.89 0.67chrF 0.97 0.97 0.98 0.88 0.99 0.97 0.97 0.90 0.94 0.97 0.86 0.80ESIM 0.98 0.99 0.99 0.93 0.96 0.93 0.98 0.90 0.99 0.99 0.94 0.83YiSi-1 0.97 0.98 0.99 0.92 0.97 0.94 0.99 0.89 0.99 0.98 0.91 0.85YiSi-2 0.61 0.12 0.92 −0.01 0.70 0.48 0.34 0.69 −0.77 0.13 −0.53 0.07

Table 2: Correlation of metrics with and without outliers (“All” and “−out”, resp.) for the language pairs intolanguages other than English that contain outlier systems.

4990

• for language pairs like Lithuanian–Englishand English–Finnish, the correlation betweenthe reference based metrics and DA is highirrespective of the presence of the outlier;• the correlation of BLEU with DA drops

sharply from 0.85 to 0.58 for English–Kazakhwhen outliers are removed;• for English–German, the correlation of BLEU

and TER appears to be almost as high as thatof YISI-1 and ESIM. However, when we re-move the two outliers, there is a much widergap between the metrics.• if metrics wrongly assign a higher score to an

outlier (e.g. most metrics in Gujarat–English),removing these systems increases correlation,and reporting only the skipped correlation isnot ideal.

To illustrate the severity of the problem, we showexamples from the metrics task data where out-liers present the illusion of high correlation whenthe metric scores are actually independent of thehuman scores without the outlier. For English–German, the source-based metric YISI-2 correctlyassigns a low score to the outlier en-de-task.When this system is removed, the correlation isnear zero. At the other extreme, YISI-2 incorrectlyassigns a very high score to a low-quality outlierin the English–Russian language pair, resulting ina strongly negative correlation. When we removethis system, we find there is no association betweenmetric and human scores.

The results for all metrics that participated in theWMT 19 metrics task are presented in Tables 3, 4and 5 in the appendix.

5 Beyond correlation: metric decisionsfor system pairs

In practice, researchers use metric scores to com-pare pairs of MT systems, for instance when claim-ing a new state of the art, evaluating different modelarchitectures, or even in deciding whether to pub-lish. Basing these judgements on metric scorealone runs the risk of making wrong decisions withrespect to the true gold standard of human judge-ments. That is, while a change may result in asignificant improvement in BLEU, this may not bejudged to be an improvement by human assessors.

Thus, we examine whether metrics agree withDA on all the MT systems pairs across all lan-guages used in WMT 19.

Following Graham et al. (2014), we use statisti-

−0.5 0.0 0.5 1.0 1.5 2.0

NS

0.0-1.5

1.5-3.0

3.0-5.0

5.0-7.5

7.5-12.0

12.0-100.0

MetricDi�erence

BLEU

Worse

NS

Better

−0.5 0.0 0.5 1.0 1.5 2.0

NS

0.0-1.5

1.5-3.0

3.0-5.0

5.0-7.5

7.5-12.0

12.0-100.0

MetricDi�erence

TER

Worse

NS

Better

−0.5 0.0 0.5 1.0 1.5 2.0

NS

0.0-1.0

1.0-2.0

2.0-4.0

4.0-6.0

6.0-10.0

10.0-100.0

MetricDi�erence

chrF

Worse

NS

Better

0.0 0.5 1.0 1.5 2.0

NS

0.0-0.5

0.5-1.0

1.0-2.0

2.0-3.5

3.5-6.0

6.0-100.0

MetricDi�erence

YiSi-1

Worse

NS

Better

−0.5 0.0 0.5 1.0 1.5 2.0

DA Di�erence

NS

0.0-2.5

2.5-5.0

5.0-7.5

7.5-15.0

15.0-30.0

30.0-200.0

MetricDi�erence

ESIM

Worse

NS

Better

Figure 4: Pairwise differences in human DA evalua-tion (x-axis) compared to difference in metric evalua-tion (binned on y-axis; NS means insignificant metricdifference). The colours indicate pairs judged by hu-mans to be insignificantly different (cyan/light gray),significantly worse (red/dark gray on the left) and sig-nificantly better (green/dark gray on the right).

4991

cal significance tests to detect if the difference inscores (human or metric) between two systems (S1and S2) can just be attributed to chance.

For human scores, we apply the Wilcoxon rank-sum test which is used by WMT when rankingsystems. We use the bootstrap method (Koehn,2004) to test for statistical significance of the differ-ence in BLEU between two systems. YISI-1 andESIM compute the system score as the average ofsentence scores, so we use the paired t-test to com-pute significance. Although CHRF is technicallythe macro-average of n-gram statistics over the en-tire test set, we treat this as a micro-average whencomputing significance such that we can use themore powerful paired t-test over sentence scores.

Figure 4 visualises the agreement between met-ric score differences and differences in human DAscores. Ideally, only differences judged as trulysignificant would give rise to significant and largemagnitude differences under the metrics; and whenmetrics judge differences to be insignificant, ide-ally very few instances would be truly significant.However, this is not the case: there are substan-tial numbers of insignificant differences even forvery high metric differences (cyan, for higher rangebins); moreover, the “NS” category — denoting aninsignificant difference in metric score — includesmany human significant pairs (red and green, topbin).

Considering BLEU (top plot in Figure 2), forinsignificant BLEU differences, humans judge onesystem to be better than the other for half of thesesystem pairs. This corresponds to a Type I error. Itis of concern that BLEU cannot detect these differ-ences. Worse, the difference in human scores hasa very wide range. Conversely, when the BLEUscore is significant but in the range 0–3, more thanhalf of these systems are judged to be insignifi-cantly different in quality (corresponding to a TypeII error). For higher BLEU deltas, these errorsdiminish, however, even for a BLEU differencebetween 3 and 5 points, about a quarter of thesesystem pairs are of similar quality. This paints adour picture for the utility of BLEU as a tool forgatekeeping (i.e., to define a ‘minimum publish-able unit’ in deciding paper acceptance on empiri-cal grounds, through bounding the risk of Type IIerrors), as the unit would need to be whoppinglylarge to ensure only meaningful improvements areaccepted. Were we seek to minimise Type I errorsin the interests of nurturing good ideas, the thresh-

BLEU TER chrF YiSi-1 ESIM

BLEU

TER

chrF

YiSi-1

ESIM

326 59 88 99 127

68 335 89 92 125

72 64 310 63 97

63 47 43 290 74

82 71 68 65 2810

50

100

150

200

250

Figure 5: The agreement between metric errors overall 1362 system comparisons. The values in the diag-onal indicate the total number of Type 1 and Type 2errors for the metric. The off-diagonal cells show thetotal number of errors made by the row-metric wherethe column-metric is correct.

old would need to be so low as to be meaningless,effectively below the level required for acceptanceof the bootstrap significance test.

The systems evaluated consist of a mix of sys-tems submitted by researchers (mostly neural mod-els) and anonymous online systems (where the MTsystem type is unknown). Even when we restrictthe set of systems to only neural models submittedby researchers, the patterns of Type 1 and Type 2errors remain the same (figure omitted for spacereasons).

TER makes similar errors: TER scores canwrongly show that a system is much better thananother when humans have judged them similar, oreven worse, drawn the opposite conclusion.

CHRF, YISI-1 and ESIM have fewer errors com-pared to BLEU and TER. When these metrics mis-takenly fail to detect a difference between systems,the human score difference is considerably lowerthan for BLEU. Accordingly, they should be usedin place of BLEU. However the above argumentis likely to still hold true as to their utility for gate-keeping or nurturing progress, in that the thresholdswould still be particularly punitive or permissive,for the two roles, respectively.

Finally, Figure 5 looks at agreement betweenmetric decisions when comparing MT systems.As expected, when BLEU or TER disagree withCHRF, ESIM, or YISI-1, the former are morelikely to be wrong. BLEU and TER have an 80%overlap in errors. The decisions of ESIM, a trained

4992

neural model, diverge a little more from the othermetrics. Overall, despite the variety of approachestowards the task, all five metrics have common bi-ases: over half of all erroneous decisions made bya particular metric are made in common with allother metrics.

6 Conclusion

In this paper, we revisited the findings of the met-rics task at WMT 2019, which flagged potentialproblems in the current best practises for assess-ment of evaluation metrics.

Pearson’s correlation coefficient is known to beunstable for small sample sizes, particularly whenthe systems in consideration are very close in qual-ity. This goes some way to explaining the findingswhereby strong correlations between metric scoresand human judgements evaporate when consider-ing small numbers of strong systems. We showthat the same can be true for any small set of simi-lar quality systems, not just the top systems. Thiseffect can partly be attributed to noise due to thesmall sample size, rather than true shortcomingsin the metrics themselves. We need better meth-ods to empirically test whether our metrics are lessreliable when evaluating high quality MT systems.

A more serious problem, however, is outlier sys-tems, i.e. those systems whose quality is muchhigher or lower than the rest of the systems. Wefound that such systems can have a disproportion-ate effect on the computed correlation of metrics.The resulting high values of correlation can thenlead to to false confidence in the reliability of met-rics. Once the outliers are removed, the gap be-tween correlation of BLEU and other metrics (e.g.CHRF, YISI-1 and ESIM) becomes wider. In theworst case scenario, outliers introduce a high corre-lation when there is no association between metricand human scores for the rest of the systems. Thus,future evaluations should also measure correlationsafter removing outlier systems.

Finally, the same value of correlation coefficientcan describe different patterns of errors. Any singlenumber is not adequate to describe the data, andvisualising metric scores against human scores isthe best way to gain insights into metric reliabil-ity. This could be done with scatter plots (e.g. Fig-ure 3a) for each language pair, or Figure 5, whichcompresses this information into one graph.

Metrics are commonly used to compare two sys-tems, and accordingly we have also investigated

the real meaning encoded by a difference in metricscore, in terms of what this indicates about humanjudgements of the two systems. Most publishedwork report BLEU differences of 1-2 points, how-ever at this level we show this magnitude of dif-ference only corresponds to true improvements inquality as judged by humans about half the time.Although our analysis assumes the Direct Assess-ment human evaluation method to be a gold stan-dard despite its shortcomings, our analysis doessuggest that the current rule of thumb for publish-ing empirical improvements based on small BLEUdifferences has little meaning.

Overall, this paper adds to the case for retir-ing BLEU as the de facto standard metric, andinstead using other metrics such as CHRF, YISI-1,or ESIM in its place. They are more powerful in as-sessing empirical improvements. However, humanevaluation must always be the gold standard, andfor continuing improvement in translation, to estab-lish significant improvements over prior work, allautomatic metrics make for inadequate substitutes.

To summarise, our key recommendations are:• When evaluating metrics, use the technique

outlined in Section 4.2 to remove outliers be-fore computing Pearson’s r.• When evaluating MT systems, stop using

BLEU or TER for evaluation of MT, andinstead use CHRF, YISI-1, or ESIM;• Stop using small changes in evaluation met-

rics as the sole basis to draw important em-pirical conclusions, and make sure these aresupported by manual evaluation.

Acknowledgements

We are grateful to the anonymous reviewers fortheir comments and valuable suggestions. Thiswork was supported in part by the Australian Re-search Council.

ReferencesFrancis J Anscombe. 1973. Graphs in statistical analy-

sis. The American Statistician, 27(1):17–21.

Vic Barnett and Toby Lewis. 1974. Outliers in Statisti-cal Data. Wiley.

Loïc Barrault, Ondrej Bojar, Marta R. Costa-jussà,Christian Federmann, Mark Fishel, Yvette Gra-ham, Barry Haddow, Matthias Huck, Philipp Koehn,Shervin Malmasi, Christof Monz, Mathias Müller,Santanu Pal, Matt Post, and Marcos Zampieri. 2019.

4993

Findings of the 2019 conference on machine transla-tion (WMT19). In Proceedings of the Fourth Con-ference on Machine Translation (Volume 2: SharedTask Papers, Day 1), pages 1–61, Florence, Italy. As-sociation for Computational Linguistics.

Ondrej Bojar, Christian Buck, Christian Federmann,Barry Haddow, Philipp Koehn, Johannes Leveling,Christof Monz, Pavel Pecina, Matt Post, HerveSaint-Amand, Radu Soricut, Lucia Specia, and AlešTamchyna. 2014. Findings of the 2014 workshop onstatistical machine translation. In Proceedings of theNinth Workshop on Statistical Machine Translation,pages 12–58, Baltimore, Maryland, USA. Associa-tion for Computational Linguistics.

Ondrej Bojar, Rajen Chatterjee, Christian Federmann,Yvette Graham, Barry Haddow, Matthias Huck, An-tonio Jimeno Yepes, Philipp Koehn, Varvara Lo-gacheva, Christof Monz, Matteo Negri, AurélieNévéol, Mariana Neves, Martin Popel, Matt Post,Raphael Rubino, Carolina Scarton, Lucia Spe-cia, Marco Turchi, Karin Verspoor, and MarcosZampieri. 2016. Findings of the 2016 conferenceon machine translation. In Proceedings of theFirst Conference on Machine Translation: Volume2, Shared Task Papers, pages 131–198, Berlin, Ger-many. Association for Computational Linguistics.

Chris Callison-Burch, Miles Osborne, and PhilippKoehn. 2006. Re-evaluating the role of Bleu in ma-chine translation research. In 11th Conference ofthe European Chapter of the Association for Com-putational Linguistics, Trento, Italy. Association forComputational Linguistics.

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, HuiJiang, and Diana Inkpen. 2017. Enhanced LSTMfor natural language inference. In Proceedings ofthe 55th Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers),pages 1657–1668, Vancouver, Canada. Associationfor Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186, Minneapolis, USA.

Marina Fomicheva and Lucia Specia. 2019. Taking MTevaluation metrics to extremes: Beyond correlationwith human judgments. Computational Linguistics,45(3):515–558.

Yvette Graham and Timothy Baldwin. 2014. Testingfor significance of increased correlation with humanjudgment. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural Language Process-ing (EMNLP), pages 172–176, Doha, Qatar.

Yvette Graham, Timothy Baldwin, Alistair Moffat, andJustin Zobel. 2017. Can machine translation sys-tems be evaluated by the crowd alone? Natural Lan-guage Engineering, 23(1):3–30.

Yvette Graham, Nitika Mathur, and Timothy Baldwin.2014. Randomized significance tests in machinetranslation. In Proceedings of the Ninth Workshopon Statistical Machine Translation, pages 266–274,Baltimore, Maryland, USA. Association for Compu-tational Linguistics.

Boris Iglewicz and David Caster Hoaglin. 1993. Howto detect and handle outliers, volume 16. Asq Press.

Philipp Koehn. 2004. Statistical significance testsfor machine translation evaluation. In Proceed-ings of the 2004 Conference on Empirical Methodsin Natural Language Processing, pages 388–395,Barcelona, Spain.

Christophe Leys, Christophe Ley, Olivier Klein,Philippe Bernard, and Laurent Licata. 2013. Detect-ing outliers: Do not use standard deviation aroundthe mean, use absolute deviation around the me-dian. Journal of Experimental Social Psychology,49(4):764–766.

Chi-kiu Lo. 2019. YiSi — a unified semantic MTquality evaluation and estimation metric for lan-guages with different levels of available resources.In Proceedings of the Fourth Conference on Ma-chine Translation (Volume 2: Shared Task Papers,Day 1), pages 507–513, Florence, Italy.

Qingsong Ma, Johnny Wei, Ondrej Bojar, and YvetteGraham. 2019. Results of the WMT19 metricsshared task: Segment-level and strong MT sys-tems pose big challenges. In Proceedings of theFourth Conference on Machine Translation (Volume2: Shared Task Papers, Day 1), pages 62–90, Flo-rence, Italy.

Nitika Mathur, Timothy Baldwin, and Trevor Cohn.2019. Putting evaluation in context: Contextualembeddings improve machine translation evaluation.In Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages2799–2808, Florence, Italy.

Jason W Osborne and Amy Overbay. 2004. Thepower of outliers (and why researchers should al-ways check for them). Practical Assessment, Re-search & Evaluation, 9(6):1–12.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Com-putational Linguistics, pages 311–318, Philadelphia,USA.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic eval-uation of machine translation. In Proceedings of

4994

the 40th Annual Meeting of the Association for Com-putational Linguistics, pages 311–318, Philadelphia,USA.

Maja Popovic. 2015. chrF: character n-gram f-scorefor automatic MT evaluation. In Proceedings of theTenth Workshop on Statistical Machine Translation,pages 392–395, Lisbon, Portugal. Association forComputational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEUscores. In Proceedings of the Third Conference onMachine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computa-tional Linguistics.

Ehud Reiter. 2018. A structured review of the validityof BLEU. Computational Linguistics, 44(3):393–401.

Peter J Rousseeuw and Mia Hubert. 2011. Robuststatistics for outlier detection. Wiley Interdisci-plinary Reviews: Data Mining and Knowledge Dis-covery, 1(1):73–79.

Aaron Smith, Christian Hardmeier, and Joerg Tiede-mann. 2016. Climbing mont BLEU: The strangeworld of reachable high-BLEU translations. In Pro-ceedings of the 19th Annual Conference of the Eu-ropean Association for Machine Translation, pages269–281.

Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin-nea Micciulla, and John Makhoul. 2006. A studyof translation edit rate with targeted human annota-tion. In Proceedings of the Association for MachineTransaltion in the Americas, pages 223–231.

Amanda Stent, Matthew Marge, and Mohit Singhai.2005. Evaluating evaluation methods for generationin the presence of variation. In Computational Lin-guistics and Intelligent Text Processing, pages 341–351, Berlin, Heidelberg. Springer Berlin Heidelberg.

Rand Wilcox. 2004. Inferences based on a skippedcorrelation coefficient. Journal of Applied Statistics,31(2):131–143.

4995

A The effect of removing outlier systems on the results of the WMT 19 metrics task

de–cs de–fr fr–deAll −out All All −out

n 11 10 11 10 7

BEER 0.978 0.976 0.941 0.848 0.794BLEU 0.941 0.922 0.891 0.864 0.821CDER 0.864 0.734 0.949 0.852 0.794CHARACTER 0.965 0.959 0.928 0.849 0.848CHRF 0.974 0.970 0.931 0.864 0.796CHRF+ 0.972 0.967 0.936 0.848 0.785EED 0.982 0.984 0.940 0.851 0.792ESIM 0.980 0.986 0.950 0.942 0.825HLEPORA_BASELINE 0.941 0.903 0.814 − −HLEPORB_BASELINE 0.959 0.951 0.814 − −NIST 0.954 0.944 0.916 0.862 0.800PER 0.875 0.757 0.857 0.899 0.427SACREBLE-BLEU 0.869 0.742 0.891 0.869 0.846SACREBLE-CHRF 0.975 0.980 0.952 0.882 0.815TER 0.890 0.787 0.956 0.895 0.673WER 0.872 0.749 0.956 0.894 0.657YISI-0 0.978 0.972 0.952 0.820 0.836YISI-1 0.973 0.980 0.969 0.908 0.846YISI-1_SRL − − − 0.912 0.814Source-based metrics:IBM1-MORPHEME 0.355 0.009 0.509 0.625 0.357IBM1-POS4GRAM − − 0.085 0.478 0.719YISI-2 0.606 0.122 0.721 0.530 0.066

Table 3: Pearson correlation of metrics for the language pairs that do not involve English. For language pairsthat contain outlier systems, we also show correlation after removing outlier systems. Correlations of metrics notsignificantly outperformed by any other for that language pair are highlighted in bold.

4996

de–e

nfi–

engu

–en

kk–e

nlt–

enru

–en

zh–e

nA

ll−

out

All

All

−ou

tA

ll−

out

All

−ou

tA

ll−

out

All

−ou

tn

1615

1211

1011

911

1014

1315

13

BE

ER

0.90

60.

852

0.99

30.

952

0.98

20.

986

0.93

00.

947

0.94

80.

915

0.81

90.

942

0.80

6B

ER

TR

0.92

60.

897

0.98

40.

938

0.99

50.

990

0.82

90.

948

0.95

90.

971

0.93

30.

974

0.91

1B

LE

U0.

849

0.77

00.

982

0.83

40.

975

0.94

60.

912

0.96

10.

980

0.87

90.

830

0.89

90.

807

CD

ER

0.89

00.

827

0.98

80.

876

0.97

50.

967

0.84

30.

975

0.98

10.

892

0.87

50.

917

0.84

7C

HA

RA

CT

ER

0.89

80.

852

0.99

00.

922

0.97

80.

953

0.83

30.

955

0.96

30.

923

0.82

80.

943

0.84

5C

HR

F0.

917

0.86

20.

992

0.95

50.

962

0.97

80.

775

0.94

00.

933

0.94

50.

876

0.95

60.

841

CH

RF

+0.

916

0.86

00.

992

0.94

70.

961

0.97

60.

769

0.94

00.

934

0.94

50.

878

0.95

60.

851

EE

D0.

903

0.85

30.

994

0.97

60.

988

0.98

00.

779

0.92

90.

930

0.95

00.

872

0.94

90.

840

ES

IM0.

941

0.89

60.

971

0.88

50.

986

0.98

60.

945

0.98

90.

990

0.96

80.

946

0.98

80.

961

HL

EP

OR

A_B

AS

EL

INE

−−

−0.

975

0.85

5−

−0.

947

0.87

9H

LE

PO

RB

_BA

SE

LIN

E−

−−

0.97

50.

855

0.90

60.

930

−0.

947

0.87

9M

ET

EO

R+

+_2

.0(S

YN

TAX

)0.

887

0.84

40.

995

0.90

90.

939

0.97

40.

859

0.92

80.

935

0.95

00.

878

0.94

80.

836

ME

TE

OR

++

_2.0

(SY

NTA

X+

CO

PY

)0.

896

0.85

00.

995

0.90

00.

930

0.97

10.

871

0.92

70.

931

0.95

20.

890

0.95

20.

841

NIS

T0.

813

0.70

50.

986

0.93

00.

985

0.94

20.

837

0.94

40.

963

0.92

50.

878

0.92

10.

722

PE

R0.

883

0.80

80.

991

0.91

00.

948

0.73

70.

533

0.94

70.

933

0.92

20.

880

0.95

20.

884

PR

EP

0.57

50.

452

0.61

40.

773

0.96

70.

776

0.81

70.

494

0.39

70.

782

0.68

50.

592

0.11

1S

AC

RE

BL

E-B

LE

U0.

813

0.79

40.

985

0.83

40.

975

0.94

60.

912

0.95

50.

967

0.87

30.

813

0.90

30.

807

SA

CR

EB

LE

-CH

RF

0.91

00.

852

0.99

00.

952

0.93

70.

969

0.75

00.

935

0.92

30.

919

0.87

40.

955

0.84

6T

ER

0.87

40.

812

0.98

40.

890

0.94

70.

799

0.56

60.

960

0.97

50.

917

0.89

60.

840

0.71

7W

ER

0.86

30.

803

0.98

30.

861

0.92

60.

793

0.57

90.

961

0.98

10.

911

0.88

50.

820

0.71

6W

MD

O0.

872

0.85

70.

987

0.98

30.

981

0.99

80.

953

0.90

00.

923

0.94

20.

844

0.94

30.

851

YIS

I-0

0.90

20.

847

0.99

30.

993

0.99

00.

991

0.87

60.

927

0.93

30.

958

0.88

90.

937

0.78

2Y

ISI-

10.

949

0.91

40.

989

0.92

40.

997

0.99

40.

920

0.98

10.

978

0.97

90.

947

0.97

90.

899

YIS

I-1_

SR

L0.

950

0.91

60.

989

0.91

80.

998

0.99

40.

917

0.98

30.

981

0.97

80.

943

0.97

70.

897

Sour

ce-b

ased

met

rics

:IB

M1-

MO

RP

HE

ME

0.34

50.

223

0.74

0−

−0.

487

0.63

8−

−IB

M1-

PO

S4G

RA

M0.

339

0.13

7−

−−

−−

−L

AS

IM0.

247

0.33

4−

−−

−0.

310

0.26

0−

LP

0.47

40.

279

−−

−−

0.48

80.

168

−U

NI

0.84

60.

809

0.93

0−

−−

0.80

50.

666

−U

NI+

0.85

00.

805

0.92

4−

−−

0.80

80.

669

−Y

ISI-

20.

796

0.61

20.

642

0.56

60.

820

0.32

40.

662

0.44

20.

346

0.33

90.

708

0.94

00.

622

YIS

I-2_

SR

L0.

804

0.63

0−

−−

−−

0.94

70.

675

Tabl

e4:

Pear

son

corr

elat

ion

ofm

etri

csfo

rthe

to-E

nglis

hla

ngua

gepa

irs.

Forl

angu

age

pair

sth

atco

ntai

nou

tlier

syst

ems,

we

also

show

corr

elat

ion

afte

rrem

ovin

gou

tlier

syst

ems.

Cor

rela

tions

ofm

etri

csno

tsig

nific

antly

outp

erfo

rmed

byan

yot

herf

orth

atla

ngua

gepa

irar

ehi

ghlig

hted

inbo

ld.

4997

en–c

sen

–de

en–fi

en–g

uen

–kk

en–l

ten

–ru

en–z

hA

llA

ll−

out

All

−ou

tA

llA

ll−

out

All

All

−ou

tA

lln

1122

2012

1111

119

1212

1112

BE

ER

0.99

00.

983

0.86

90.

989

0.97

80.

829

0.97

10.

826

0.98

20.

977

0.94

70.

803

BL

EU

0.89

70.

921

0.41

90.

969

0.94

30.

737

0.85

20.

576

0.98

90.

986

0.96

70.

901

CD

ER

0.98

50.

973

0.84

90.

978

0.95

70.

840

0.92

70.

668

0.98

50.

993

0.98

10.

905

CH

AR

AC

TE

R0.

994

0.98

60.

886

0.96

80.

939

0.91

00.

936

0.89

50.

954

0.98

50.

982

0.86

2C

HR

F0.

990

0.97

90.

881

0.98

60.

972

0.84

10.

972

0.90

00.

981

0.94

30.

968

0.88

0C

HR

F+

0.99

10.

981

0.88

30.

986

0.97

00.

848

0.97

40.

907

0.98

20.

950

0.97

30.

879

EE

D0.

993

0.98

50.

894

0.98

70.

978

0.89

70.

979

0.88

30.

975

0.96

70.

984

0.85

6E

SIM

−0.

991

0.92

80.

957

0.92

6−

0.98

00.

900

0.98

90.

989

0.98

60.

931

HL

EP

OR

A_B

AS

EL

INE

−−

−0.

841

0.96

80.

852

−−

−H

LE

PO

RB

_BA

SE

LIN

E−

−−

0.84

10.

968

0.85

20.

980

−−

NIS

T0.

896

0.32

10.

246

0.97

10.

936

0.78

60.

930

0.61

10.

993

0.98

80.

973

0.88

4P

ER

0.97

60.

970

0.81

50.

982

0.96

10.

839

0.92

10.

545

0.98

50.

981

0.95

50.

895

SA

CR

EB

LE

-BL

EU

0.99

40.

969

0.80

60.

966

0.93

90.

736

0.85

20.

576

0.98

60.

977

0.94

60.

801

SA

CR

EB

LE

-CH

RF

0.98

30.

976

0.87

40.

980

0.95

80.

841

0.96

70.

840

0.96

60.

985

0.98

80.

796

TE

R0.

980

0.96

90.

841

0.98

10.

960

0.86

50.

940

0.54

70.

994

0.99

50.

985

0.85

6W

ER

0.98

20.

966

0.83

10.

980

0.95

80.

861

0.93

90.

525

0.99

10.

994

0.98

30.

875

YIS

I-0

0.99

20.

985

0.86

90.

987

0.97

70.

863

0.97

40.

840

0.97

40.

953

0.96

70.

861

YIS

I-1

0.96

20.

991

0.91

70.

971

0.93

70.

909

0.98

50.

892

0.96

30.

992

0.97

80.

951

YIS

I-1_

SR

L−

0.99

10.

917

−−

−−

−0.

948

Sour

ce-b

ased

met

rics

:IB

M1-

MO

RP

HE

ME

0.87

10.

870

0.19

80.

084

0.25

4−

−0.

810

−−

IBM

1-P

OS4G

RA

M−

0.39

30.

449

−−

−−

−−

LA

SIM

−0.

871

0.00

7−

−−

−0.

823

0.33

6−

LP

−0.

569

0.55

8−

−−

−0.

661

0.17

8−

UN

I0.

028

0.84

10.

251

0.90

70.

808

−−

−0.

919

0.76

0−

UN

I+−

−−

−−

−0.

918

0.74

6−

US

FD

−0.

224

0.30

1−

−−

−0.

857

0.51

4−

US

FD

-TL

−0.

091

0.21

2−

−−

−0.

771

0.17

7−

YIS

I-2

0.32

40.

924

0.01

40.

696

0.47

80.

314

0.33

90.

685

0.05

50.

766

0.13

40.

097

YIS

I-2_

SR

L−

0.93

60.

155

−−

−−

−0.

118

Tabl

e5:

Cor

rela

tion

ofm

etri

csfo

rth

efr

om-E

nglis

hla

ngua

gepa

irs.

For

lang

uage

pair

sth

atco

ntai

nou

tlier

syst

ems,

we

also

show

corr

elat

ion

afte

rre

mov

ing

outli

ersy

stem

s.V

alue

sin

bold

indi

cate

that

the

met

ric

isno

tsig

nific

antly

outp

erfo

rmed

byan

yot

herm

etri

cun

dert

heW

illia

ms

Test

.