Uniformity of some recent color metrics testedwith an accurate color-difference tolerance dataset

M. Melgosa, J. J. Quesada, and E. Hita

The Rochester Institute of Technology-Dupont dataset [Color Res. Appl. 16, 297-316 (1991)] has beenused to analyze the uniformity of seven color metrics, developed after CIELUV and CIELAB, withmethods similar to those previously applied to several other classical datasets [J. Opt. Soc. Am. A 9,1247-1253 (1992)]. Significant performance improvements over CIELAB were found with severalCIELAB-based metrics, mainly with the model recently proposed by Commission Internationale deL'Eclairage Technical Committee 1-29 [Color Res. Appl. 18, 137-139 (1993)]. Several significantdifferences found between some pairs of metrics became insignificant when we selected from theRochester Institute of Technology-Dupont dataset pairs of samples with only chromaticity differences.

Key words: Color differences, color metrics.

1. IntroductionThe search for better correlation between visuallyperceived and instrumentally measured color differ-ences can be considered to be one of the mostimportant and unresolved problems in practical appli-cations of colorimetry. It is desirable to have anavailable color space (with its associated metric orcolor-difference formula) where any pair of sampleswith a constantly perceived color difference is repre-sented by a pair of points between which the mea-sured color difference is also constant. Such a space,where, for example, all the color-discriminationthresholds are represented by spheres of equal ra-dius, can be called uniform; unfortunately it is not yetavailable because of, among other things, our incom-plete knowledge of color-vision mechanisms.

In 1976 the Commission Internationale deL'Eclairage (CIE), with the primary goal of promot-ing uniformity of practice between users,1 2 recom-mended the use of CIELUV and CIELAB as approxi-mately uniform color spaces. Since then CIELABhas been widely accepted in industry and research, asone can conclude from recent surveys in the U.S.A.and other countries. 3 4

The authors are with the Universidad de Granada, 18071,Granada, Spain; M. Melgosa and E. Hita are with the Departa-mento de Optica, Facultad de Ciencias; and J. J. Quesada is withthe Departamento de Matematica Aplicada.

Received 28 January 1994; revised manuscript received 19 May1994.

0003-6935/94/348069-09$06.00/0.© 1994 Optical Society of America.

Several authors5-7 have shown the lack of unifor-mity of CIELAB, but this point is not surprising if onebears in mind the CIE assertion that with CIELAB itmay be necessary to use different weightings for thelightness, chroma, and hue differences in differentpractical applications (see Ref. 1, p. 33, note 9). Thisbasic idea of using CIELAB with appropriate weight-ing functions for each of its three color-differencecomponents has led to several interesting CIELAB-based models, 89 which have also been successfullyapplied in some textile industries in the last fewyears. More recently, a tentative recommendationof CIE Technical Committee 1-29 based on CIELABhas been proposed for study. 10

As is well known 1 color-difference perception de-pends also on the experimental conditions in whichthe visual task is developed (for example, the textureof the samples, illumination or luminance levels, andthe color of the background). The influence of someof these conditions, often called parametric effects,was analyzed recently by CIE Technical Committee1-28,12 and it seems clear that these conditions havestrongly contributed to the wide spread of previousexperimental results. Undoubtedly this spread, in-cluding inconsistencies between different experimen-tal results, has been a major problem in the develop-ment of color metrics that give a well-correlatedresult for all color-perception experimental measure-ments.

This last point must be emphasized. To developnew, successful, and uniform metrics, accurate andwide datasets of visual color-difference evaluationsare needed.13"14 The CIE has also suggested 5 estab-

1 December 1994 / Vol. 33, No. 34 / APPLIED OPTICS 8069

lishing a comprehensive set of data that describes theperceptibility of color differences and gives prioritywhen necessary to the results obtained under theparametric factors that are most common in indus-trial practice. Along this line, among the severalmore recent and worthy experimental datasets re-ported,16-18 we refer to one known as the RIT-Dupontdataset' 920 [part of a joint research program of theMunsell Color Science Laboratory at the RochesterInstitute of Technology and Dupont Automotive Prod-ucts (see Ref. 20)], which was accurately and specifi-cally developed for fitting and testing the perfor-mance of new color metrics, starting fromexperimental conditions that are typical of commer-cial color decisions.

In this paper we contribute to the analysis of theperformance of some of the most important colormetrics proposed after CIELUV and CIELAB fromthe point of view of the uniformity of such metricswith respect to the RIT-Dupont dataset. In essencethe analysis method employed here is analogous tothat of some previous studies carried out from classi-cal color-threshold experimental results2122 and isuseful for checking some of our earlier conclusions.However, in this paper, by using the RIT-Dupontdataset, we can manage a greater number of colormetrics and give special attention to the statisticalsignificance of the differences found between themetrics, because revisions of the current CIE metrics(CIELUV and CIELAB) should reasonably occur onlywhen a real improvement in performance is observedfrom accurate experimental datasets.

The statistical analysis for testing the significanceof the differences between the different color metricsbeing studied has been made with nonparametrictesting methods because of the abnormally distrib-uted nature of the data. Thus we have applied twodistribution-free procedures for the comparison ofindependent samples, namely, the U test of Mann andWhitney23 for comparing two independent samplesand the H test of Kruskal and Wallis24 for comparingseveral independent samples.

Moreover, as we explain in Section 4, to analyze thehomogeneity of the color metrics being studied, weapplied the test proposed by Cochran25 to check theequality/inequality of several variances.

A more detailed description of the parameters formeasuring the uniformity of the color metrics, to-gether with the set of nine color metrics employed byus and the use of the RIT-Dupont dataset, is inSection 2. Section 3 contains the results and thefirst preliminary conclusions. They are completedin Section 4 with detailed explanations of the applica-tion of the nonparametric statistical tests to theRIT-Dupont dataset. Finally, in Section 5 are themain conclusions of our research.

2. Procedure

2.A. Color MetricsIn this study we considered first, as an obviousreference, the currently recommended CIE metrics:

CIELUV and CIELAB.1 Next, we considered threecolor metrics based on CIELAB that showed subse-quent improvements in this space and have beenadopted with good results by several textile industriesmainly in the UK. These color metrics are JPC79,26

CMC(l:c), 8 and BFD(l:c).9 In the last two metrics theparameters related to parametric factors are kept as1 = c = 1 for this study.

The current interest in retaining the strong fea-tures and international acceptance of CIELAB hasled CIE Technical Committee 1-29 to propose thefollowing generalized color-difference equation1 0 27

based also on CIELAB:

(AL* 2

AETCl-29 = LkLS

( AC* 2 (AH* 2

+kcSC kHSHI

I 1/2X (1)

where kL, kc, and kH are parametric factors thatdepend on experimental conditions and have assignedvalues of unity in a given set of basis conditions.10

However, SL, SC, and SH are correction factors, calledweighting functions, that appear to improve theperceptual uniformity of the CIELAB space and aregiven as

SL = 1, SC = 1 + 0.045Cab*,

SH = 1 + 0.015Cb*. (2)

The previous color metric with the weighting func-tions just indicated has been used in our currentanalysis and is denoted as TC1-29. The same metricbut with slightly different weighting functions shownbelow has also been used here and is denoted TC1-29*:

SL = 1, SC = 1 + 0.039Cab*,

SH = 1 + 0.009Cab* (3)

These last weighting functions were those obtainedwhen the TC1-29 final recommendation10 was beingoptimized,27 but only one of the experimental datasetsemployed was used, the RIT-Dupont dataset. Thusthe final results in this paper with TC1-29* can beconsidered as a check of our methodology, because agood performance can be expected with this metric,although the methods presented in this paper andthose used for optimizing the TC1-29 models10 27 aredifferent.

Finally, two more color metrics were added to thisstudy28 29; they are denoted cdf-G* and cdf-G**.These recently proposed color metrics are based onearlier results of MacAdam,30 not CIELAB, and thesemetrics showed satisfactory results when they wereapplied to some classical color-threshold experimen-tal datasets.

More detailed descriptions of the previous colormetrics can be found in the original references givenabove and also in some recent reviews on color-difference metrics.3

8070 APPLIED OPTICS / Vol. 33, No. 34 / 1 December 1994

2.B. Experimental DatasetAs we mentioned above, this study is based on theRIT-Dupont experimental results. 19 20 This datasethas unusually high precision and accuracy comparedwith most of the previous datasets (e.g., color-threshold ellipsoids or ellipses). The experimentaldesign, development of visual tasks, and data analysisof RIT-Dupont research can be considered optimal forsubsequent development and/or testing of successfulcolor metrics in the field of industrial colorimetry.

In the RIT-Dupont research 19 color positions wereanalyzed; the gamut of real surface colors with appro-priate variations in the color space along severalvector directions was sampled for each center. Thefinal results were published in Table IV of Ref. 20(Appendix) and have been directly used as the mainsource of information for this paper. (Some slightdifferences between Table IV in Ref. 20 and theoriginal revised database are not significant for thisstudy.)

We numbered the 19 color centers in the sameorder as appeared in the original Table IV.20 Toapply the cdf-G* and cdf-G** groups of formulas, it isnecessary to assign each center to a given region.We used Fig. 1 in Ref. 29 for this, although in a fewcases (centers 1, 4, 11, and 14), for pairs placed in theboundaries of two regions, some doubt arose aboutthe appropriate assignment. In these cases we didthe assignment by trial and error, looking at the bestperformance of these metrics from the point of viewof our next analysis of uniformity. The final assign-ments, names, and numbers of each of the 19 colorcenters are given in Table 1, where the five colorcenters recommended by the CIE for study15 areidentified. Figure 1 shows the approximate loca-tions in the a*, b* CIELAB diagram of these color

Table 1. Assignment of Numbers, Names, and Zones Selected for thecdf-G* and cdf-G** Sets of Formulas for Each of the 19 Color Centers of

the RIT-Dupont Dataseta

Center Name Zone

1 Moderate Blueb Blue2 Moderate greenish blue Achromatic3 Medium graya Achromatic4 Moderate bluish greenb Green5 Light brown Achromatic6 Grayish purple Achromatic7 Dark reddish orangeb Red8 Moderate yellowb Achromatic9 Grayish yellow green Achromatic

10 Black Achromatic11 Light bluish green Green12 Moderate reddish brown Red13 Dark bluish green Green14 Brilliant greenish blue Achromatic15 Very dark red Red16 Moderate purplish pink Red17 Dark blue Blue18 Light gray Achromatic19 Strong orange yellow Orange

aRef. 20.bCIE centers. 15

75-

50-

b* 25-

0-

-25-

-50 -25 0 25 50

a*Fig. 1. Approximate positions in the CIELAB diagram a*, b* forthe 19 color centers analyzed in the RIT-Dupont 20 dataset.

centers. It is well understood that with RIT-Dupontresearch a unique standard is not associated witheach vector, and consequently there is not, strictlyspeaking, a color center, reference, or standard samplefor all the pairs obtained in each of the 19 positionsstudied.

Different pairs of samples can be calculated fromthe RIT-Dupont data, as suggested in Ref. 19, pp.145-146, or Ref. 20, p. 306, by using a tolerance of50% rejection probability or median tolerance de-signed as T50. For each tolerance the 95% upperfiducial limit (UFL) and lower fiducial limit (LFL) arealso available for 152 of the 156 tolerances reported.As suggested, 20 a weight could be calculated andassigned to each tolerance, indicating different preci-sions in the estimation of that tolerance. For thenext calculations these weighting factors are of thefollowing form:

weight = [T5 0 /(UFL - LFL)]. (4)

However, the most significant thing that must bekept in mind for our next analysis is that all thesepairs have visual tolerances perceptually equivalentto a unique near-gray anchor pair of 1.02 CIELABcolor-difference units. That is, these pairs have anidentical visual color difference AV.

We start our calculations for analyzing the unifor-mity with the 152 tolerances where UFL and LFL areknown by computing the a*, b, L* coordinates forthe 152 x 2 = 304 pairs of samples. The correspond-ing color differences for these 304 pairs, with each ofthe nine color metrics mentioned above, are used toevaluate the uniformity of the metrics, as describedbelow.

2.C. Parameters of UniformityFor each color metric the weighted arithmetic meanAE of the color-difference pairs, in each of the 19 color

1 December 1994 / Vol. 33, No. 34 / APPLIED OPTICS 8071

19U

5 1 2 7

9 U U U

154 310 18 16

*1 2 6

1 7U m* 1

U

centers, together with its corresponding coefficient ofvariation u% (the ratio of the standard deviation tothe mean, given in percentages) has been computed.The weights for these computations were indicated inEq. (4). It seems to be convenient to use a weightedmean AE instead of the simple arithmetric meanbecause of the different precisions when estimatingeach tolerance in the RIT-Dupont database.

From the AE and a% values in the color centers thefollowing two parameters were calculated for eachcolor metric: (a) Parameter S1%, the standard devia-tion of the 19 values of AE (normalized to their

average, denoted as (E)); (b) parameter S2%, theaverage of the u% values of the 19 centers.

The physical meaning of parameters S1% and S2%can be easily understood. Because all the samplepairs considered here have a perceptually identicalcolor difference (AV = constant), for an ideal colormetric the parameter a% should be zero for each colorcenter and consequently the parameter S2% shouldalso be zero. On the other hand, parameter S1%indicates the potential differences between severalregions of the color space and should also approachzero for an ideal metric. Finally parameter (E)

Table 2. Weighted Arithmetic Mean AE and Coefficients of Variation a% of the Color-Difference Pairs Corresponding to Each of the 19 Color CentersaCalculated for Each of the Nine Color Metrics

Center CIELUV CIELAB JPC79 CMC(1:1) BFD(1:1) TC1-29 TC1-29* CDF-G* CDF-G**

1 AE 2.10acr% 60.08

2 AE 1.62(d% 27.48

3 AE 1.32d o% 21.94

4 AE 1.92o 27.81

5 AE 1.84o 29.26

6 AE 1.98do 40.42

7 AE 2.30d o 33.22

8 AE 1.95d o% 26.31

9 AE 1.65d o% 21.82

10 AE 0.94do 20.35

11 AE 2.41(7% 16.52

12 AE 1.32d o% 18.73

13 AE 1.81d o 32.25

14 AE 2.76u%0o 25.21

15 AE 1.71d o 22.10

16 AE 2.61d o 33.14

17 AE 1.43cr7% 27.54

18 AE 1.81d7% 20.87

19 AE 2.67d o 20.94

1.5552.87

0.9945.97

1.28 0.8221.28 31.11

1.0015.08

1.5828.90

1.3218.01

1.3433.39

0.8929.21

1.1729.65

1.51 1.0941.02 35.05

1.6716.19

1.4721.53

1.2619.55

1.0614.27

1.7719.12

1.1818.48

1.8646.64

0.9621.56

0.8521.47

0.9224.07

1.4825.29

0.9516.59

0.9527.61

0.9936.96

1.92 1.1416.22 20.49

1.5915.32

1.1710.89

1.81 0.9719.73 25.64

1.3225.39

1.4218.43

2.2636.72

0.8818.64

2.14 1.6951.06 26.44

1.0227.50

aRef. 20.

8072 APPLIED OPTICS / Vol. 33, No. 34 / 1 December 1994

1.0836.95

0.9415.88

1.2422.79

0.9914.97

1.2421.56

1.1924.44

1.079.44

0.958.46

0.9815.83

1.7919.01

1.058.81

1.1915.50

1.2423.85

1.2720.98

1.5619.70

1.206.68

1.2320.24

1.3021.41

1.2826.42

1.5326.20

1.4220.16

1.3517.31

1.6428.21

1.2614.11

1.3417.36

1.3519.61

1.7114.57

1.5814.20

1.2911.56

1.6219.70

1.5717.64

1.629.95

1.5415.99

1.289.27

2.0130.09

1.3716.29

0.9336.67

0.9012.49

0.9713.74

0.988.44

0.886.92

1.0622.17

0.898.87

0.9312.23

0.929.96

1.0514.44

1.098.15

0.8615.93

1.1019.89

1.2624.76

1.108.86

1.2411.04

0.9920.83

1.4018.16

1.1816.14

1.0137.52

0.9515.09

0.9813.89

1.069.30

0.955.80

1.1222.79

0.998.32

1.009.10

0.9710.84

1.0514.42

1.188.27

0.9214.93

1.1820.24

1.3624.39

1.179.48

1.3110.87

1.0419.03

1.4118.28

1.2911.24

3.6832.29

2.5246.91

1.3842.89

2.0144.83

2.1136.46

2.8939.77

3.3929.97

1.6335.99

1.6827.01

3.2035.65

2.1033.33

2.6939.39

3.7234.11

2.8645.74

6.1641.30

3.0954.83

2.7132.38

1.1939.16

2.4730.28

4.7733.54

2.3046.50

1.2942.26

2.1144.62

2.0636.17

2.7139.10

3.4229.41

1.6936.64

1.6327.28

3.0335.23

2.1834.07

2.7140.75

3.8634.29

2.6738.45

6.2040.18

3.2654.32

3.1834.53

1.1239.57

2.4530.33

1.2212.33

Table 3. Parameters Indicating the Size of the Color Differences (AE), the Homogeneity S1%, and the Isotropy S2% for each Color Metric and theEntire RIT-Dupont Databaseab

Parameter CIELUV CIELAB JPC79 CMC(1:1) BFD(1:1) TC1-29 TC1-29* CDF-G* CDF-G**

(AE) 1.90 1.52 1.09 1.22 1.48 1.04 1.10 2.71 2.77S1% 24.99 20.29 27.28 18.87 12.88 13.91 13.40 40.10 42.92S2% 27.68 24.46 28.01 18.10 18.42 15.25 14.94 38.02 37.75

aRef. 20.bSmall values of S1% and especially of S2% indicate a good color-metric performance.

could be useful in ascertaining the relative sizes of thecolor-difference units in our metrics.

In summary, parameter S2% could be seen as ameasure of the isotropy and S1% as a measure of thehomogeneity of the color metrics, both characterizinguniformity. From a practical point of view param-eter S2% is more interesting than S1% for checkingthe performance of the color metrics, because compari-sons between two different regions of the color spaceare not as common as comparisons around a givencolor center. Note that the smaller values obtainedfor S2% with a color metric correspond to a betterisotropy of this metric and consequently to a bettercorrelation with the visually perceived color differ-ence in a given region.

In previous studies21 22 we used the same param-eters defined above to check the uniformity of somecolor metrics, although with different experimentaldatasets. Other parameters similar to those re-ferred to above have also been proposed by otherauthors.32,33

3. Results and DiscussionOur analysis concerning the uniformity of severalcolor metrics, through the parameters indicated above,was first carried out for all 304 color-difference pairsavailable from the RIT-Dupont dataset.19 ,20 The sec-ond step was to repeat the same calculations but now

20

15

0.4

use only some of these pairs, the 120 pairs for whichthere were only chromaticity differences (pairs ob-tained from the vectors designed by B, C, D, E inTable IV of Ref. 20).

The main reasons for these new calculations wereto check for possible improvement in each colormetric and in particular improvements in the cdf-G*and cdf-G** metrics. These two metrics were ob-tained28 29 from the metric coefficients proposed byMacAdam in a chromaticity diagram.30 We expectthat they will be properly applied to calculating onlychromaticity differences.

The values computed for parameters AE and u%, byuse of all 304 color-difference pairs, are shown inTable 2, for each of the nine color metrics and each ofthe 19 centers. The results in Table 2 are summa-rized in Table 3, showing the values of parameters(AE), S%, and S2%, for each color metric. Forbrevity the equivalent to Table 2 when only 120 pairswith a chromaticity difference were used has not beengiven; however, in Table 4 the summarized resultsare shown analogously as in Table 3.

Although Table 2 can be considered cumbersome,we should obtain some useful information from itthat is not available from Table 3. (Note, for ex-ample, the small values-the good performance- ofBFD for centers 1 and 17, placed in the blue region.)Slight discrepancies can be found between the data in

S2°/

20

15

0.6 0.8 1 1.2 1.4 1.6 0.003 0.012 0.021 0.03 0.039 0.048 0.057 0.066

Ki (solid) K3 (dashed) K2 (solid), K4 (dashed)Fig. 2. Relationship between the values of our parameter S2% with the RIT-Dupont dataset and the values of the four constants involvedin the weightingfunctions ofthe TC1-29* modell': Sc = K1 + K2 C*; SH = K3 + K4 C*, where the values proposed by TC1-29* wereKl =K3 = 1, K2 = 0.039, and K4 = 0.009.

1 December 1994 / Vol. 33, No. 34 / APPLIED OPTICS 8073

I,

\" ,,'~~~~~~~~~~.

I ,

2b,

I'll

Table 4. Parameters Indicating the Size of the Color Differences (.E), the Homogeneity S1%, and the Isotropy S2% for each Color Metric and Part ofthe RIT-Dupont Database,ab the Pairs Having only a Chromaticity Difference

Parameter CIELUV CIELAB JPC79 CMC(1:1) BFD(1:1) TC1-29 TC1-29* CDF-G* CDF-G**

(AE) 2.20 1.77 1.34 1.28 1.67 1.01 1.11 3.42 3.50S1% 27.56 31.85 39.81 22.08 18.22 16.76 14.77 38.45 41.34

S2% 19.74 19.83 16.87 16.42 12.64 14.59 14.98 13.93 13.72

aRef. 20.bSmall values of Sl% and especially S2% indicate a good color-metric performance.

Tables 2 and 3 because for simplicity Table 2 showsrounded values of AE and u%, whereas the values inTable 3 are those obtained from all the significantfigures.

An initial inspection of the values of parametersS2% and S1% in Tables 3 and 4 shows clear improve-ment in several metrics, e.g., CMC, BFD, or TC1-29,with respect to the CIE-recommended metrics (CIE-LUV and CIELAB). Although this point is analyzedthoroughly below, from the point of view of thestatistical significance, we note that this improve-ment is an important result that is in agreement withthat of some other researchers. 9 34

Looking at S2% as the main parameter to beconsidered, we receive the best results for the wholeRIT-Dupont dataset from TC1-29* followed by TC1-29(see Table 3). As we mentioned in Subsection 2.B,the good performance of TC1-29* was in part ex-pected because of the origin of this color metric. Atthis point we have checked whether the values for theconstants in Eqs. (2) and (3) for Sc and SH alsooptimized the results for our parameters S1% andS2%. As an example, Fig. 2 shows for S2% that theparameters selected for TC1-29* are also accuratelyoptimized for our analysis of the uniformity.

When the chromaticity differences are analyzed(Table 4), the smallest value of S2% is obtained byBFD, followed by cdf-G**, cdf-G*, and TC1-29 in thisorder. Note, however, that it seems that the BFDformula strongly improves the remaining metrics forthe centers in the blue region, as was looked forspecifically in the derivation of such a metric.9Thus, if the blue centers, 1 and 17, are eliminated, thevalue of S2% given by BFD increases, while it de-creases for all the remaining metrics, both in Tables 3and 4. (For example, without these two centers theTC1-29 metric improves BFD, by a smaller S2%value, for chromaticity differences.)

In Table 4 the values of parameter S2% are smallerthan in Table 3 for most metrics. The improvementis highly significant for cdf-G* and cdf-G** for thereasons given at the beginning of this section.

Several additional and interesting comments shouldbe made from Tables 2-4. Note, for example, that inTables 3 and 4, for both S1% and S2%, the CMCmetric always improves CIELAB and TC1-29 alwaysimproves CMC. However, we think that an interest-ing question would arise when these improvementsare significant. Our main goal in Section 4 is theanalysis of the significance of the differences betweenthe metrics from a statistical point of view.

4. Significance of the Differences between the Metrics

To determine when the main parameter S2%, ob-tained for each color metric, has significantly differ-ent values, we applied some nonparametric tests (alsocalled rank tests or free-distribution procedures) withgood asymptotic efficiency. 35 This type of analysiswas preferred because only weak assumptions aboutthe underlying distributions, which are essentiallyunknown in the present case, are necessary.

First, we tested the hypothesis of homogeneity orthe equivalence of a few selected sets of four metricsby the H test,23 starting from the values of a% givenin Table 2. The results are in Table 5. At a 95%confidence level, if the value of the statistic calculatedH is greater than 7.815, the hypothesis of equivalencebetween the metrics must be rejected. In our case,even at a 99.5% confidence level all four sets ofmetrics indicated in Table 5 could be considered asnonequivalent, which confirms that some of the newmetrics developed after CIELUV and CIELAB couldgive a significant improvement.

For a better analysis of the results from the H test,we compared by pairs all nine of our metrics. Forthis purpose we applied the U test,24 starting withTable 2 (derived from the entire RIT-Dupont dataset),and the results are in the upper right half of Table 6.The results in the lower left half of Table 6 in boldnumbers show the values obtained when the U test isapplied to all possible pairs of metrics, but now with apart of the RIT-Dupont dataset: the 120 pairs withonly chromaticity differences (from a table, analogousto Table 2, from which Table 4 was derived).

The values in Table 6 must be compared with thecritical values tabulated for this test at several confi-dence levels. For example, at a confidence level of95% this critical value is 123 and at a 90% confidencelevel it is 135. If the value in Table 6 is greater thanthe critical value, the two metrics being studiedcannot be considered as significantly different at this

Table 5. Results of the Nonparametric H test, Applied to Test whetherSeveral Different Sets of Color Metrics are Significantly Different

Color Metrics i Values

CIELUV, CIELAB, CMC(1:1), BFD(1:1) 16.22CIELUV, CIELAB, CMC(1:1), TC1-29 22.70CIELAB, JPC79, CMC(1:1), BFD(1:1) 19.55CIELAB, CMC(1:1), BFD(1:1), TC1-29 15.63

aRef. 23.

8074 APPLIED OPTICS / Vol. 33, No. 34 / 1 December 1994

Table 6. Results of the Nonparametric U Test Applied to all Color-Metric Pairs Possible to Test Whether They Can be ConsideredSignificantly Different

Color Metrics CIELUV CIELAB JPC79 CMC(1:1) BFD(1:1) TC1-29 TC1-29* CDF-G* CDF-G**

CIELUV 112 169 67 60 39 38 49 46CIELAB 174 118 132 124 76 71 59 62JPC79 146 156 67.5 67 45 43 89 63CMC(1:1) 145 153 180.5 175.5 135 125 10 10BFD(1:1) 106 117 152 144 118 113 3 3TC1-29 131 130 161 163 167 176 9 10TC1-29* 130 144 161 165 164 175.5 10 11CDF-G* 122 132 158 156 175 178 180 181CDF-G** 121 132.5 158 157 175 174 177 180

aRef. 24.bThe boldface numbers indicate the results from using the RIT-Dupont color pairs with only a chromaticity difference.

level and the hypothesis of homogeneity of thesemetrics can be accepted.

One could draw an interesting conclusion fromTable 6 by noting that the values in the upper righthalf are in general smaller than those in the lower lefthalf; that is, most of the metrics are not significantlydifferent when only chromaticity differences are ana-lyzed, but they are significantly different when theyare applied to the entire dataset. This is the case, forexample, of the CIELUV/CIELAB pair, at a 95%confidence level and also of the CMC/TC1-29 pair atjust a 90% confidence level.

This fact does not necessarily imply that the morefrequently observed significant pairwise comparisonsfor the color-difference data are due to differences inthe lightness sensitivity of the metrics alone. Wemust remember that the structure of the RIT-Dupont dataset is such that when a subset of chroma-ticity-only differences is selected, the remaining differ-ences are not lightness-only differences. Moreoverwe can also observe from Table 6 that, for example, bycomparison CIELAB/TC1-29 has U values that arevery different (76 and 130) for color and chromaticitydifferences, but we know that both metrics have thesame lightness sensitivity. This is not a contradic-tion; actually when considering the whole dataset, wesee that the mutual influence of chromaticity/ light-ness causes a significant difference between the twocolor metrics CIELAB and TC1-29.

In conclusion, we cannot consider the differences inTable 6 between the color differences and chromatic-ity-only differences to be a result of the inclusion/exclusion of lightness differences alone, although weshould emphasize that a good balanced sampling oflightness and chromaticity differences is needed, asthe RIT-Dupont database has, for significant testingof the color-metrics performance.

From Table 6 we can also see, among other things,that for the whole dataset (the upper right half), thedifferences among CMC/BFD, TC1-29/TC1-29*, andcdf-G*/cdf-G** are not significant. Also, theCIELAB/CMC difference is not significant at a 95%confidence level whereas the CIELAB/TC1-29 differ-ence is highly significant. The CMC/TC1-29 pair

appears to be just as significantly different at the 90%confidence level.

For chromaticity differences (Table 6, lower lefthalf) the best results obtained by BFD are notsignificantly different from those obtained by eithercdf-G* or TC1-29 at a 95% confidence level.

With Cochran's test25 we check the equality ofseveral variances in two conditions: with equal-sized groups and variance of one group is substan-tially larger than that of the others. We appliedCochran's test to the study of the homogeneity of thenine color metrics measured by parameter S1%.With the values of S1% from both Tables 3 and 4, weused Cochran's test to compute the ratio between themaximum variance and the sum of all variances.This ratio is compared with the tabulated value forthe statistic. With Cochran's test, values are ob-tained for color differences and chromaticity differ-ences of 0.49 and 0.42, respectively. In our case, at a99% confidence level, the critical value is 0.25, andthen we conclude that in both cases there is noequality of variances, so there is not the same degreeof homogeneity in the various color metrics beingstudied.

5. ConclusionsAn accurate color-difference tolerance dataset19 20 hasbeen used for testing the uniformity of several colormetrics proposed recently. To obtain a measure ofthe homogeneity and isotropy of each color metric,two appropriate parameters previously proposed 1

were employed.Among the metrics considered, several CIELAB-

based models showed significant improvements withrespect to CIELAB and/or CIELUV, according to theH tests carried out. The best results from the pointof view of isotropy came from the TC1-29 metric. 10

Good results were also found for CMC(1: 1) andBFD(1:1), which were also significantly better thanCIELAB, at a 90% confidence level, with the U testsapplied. The differences between CIELAB and CIE-LUV, and also between TC1-29 and CMC(1:1), weresignificant from U tests at a 90% confidence level witha superior performance from the former in both

1 December 1994 / Vol. 33, No. 34 / APPLIED OPTICS 8075

cases. The improvements obtained with BFD(1:1)are particularly evident for the centers in the blueregion. All these results are considered in general tobe in good agreement with those we reported previ-ously2l 22 when analyzing several threshold color dif-ferences.

When the same previous analysis was repeated byusing only the pairs with a chromaticity difference inthe RIT-Dupont dataset, several of the differencesfound previously between the metrics became insig-nificant. This result should be connected with previ-ous studies in which differences between severalmetrics were not found to be significant,6 36 and itshould be kept in mind in future developments ofuseful experimental datasets.

Although the RIT-Dupont dataset can be consid-ered accurate and important, it is desirable to confirmthese good results achieved by the TC1-29 model withother datasets. Moreover from our point of viewother research on a connection between either thesemetrics or other CIELAB-based metrics and advancesin the knowledge of color-vision mechanisms shouldalso be interesting.

It is a pleasure to thank D. H. Alman for providingthe original RIT-Dupont dataset and for commentsand suggestions on the first version of this paper.

This research was supported in part by Direcci6nGeneral de Investigaci6n Cientifica y T6cnica, Minis-terio de Educaci6n y Ciencia (Spain), under ResearchProject PB91-0717 and Comisi6n Interministerial deCiencia y Tecnologia, Ministerio de Educaci6n yCiencia (Spain), under Research Project TIC91-0646.The authors are grateful for this support.

References1. Colorimetry, CIE Publ. 15.2 (Central Bureau of the CIE,

Vienna, 1986).2. A. R. Robertson, "Historical development of CIE recom-

mended color difference equations," Color Res. Appl. 15,167-170 (1990).

3. R. G. Kuehni, "Industrial color difference: progress andproblems," Color Res. Appl. 15, 261-265 (1990).

4. H. Komatsubara and G. Kawakami, "Research on the applica-tion of color-difference formulas to industry," Stud. Color 35,6-20 (1988).

5. A. R. Robertson, "The CIE 1976 color-difference formulae,"Color Res. Appl. 2, 7-11 (1977).

6. M. R. Pointer, "A comparison of the CIE 1976 colour spaces,"Color Res. Appl. 6, 108-118 (1981).

7. E. Hita, M. Melgosa, J. Romero, and L. Jimenez del Barco,"Uniformity study of color representation systems ATD andATDN," Opt. Pura Apl. 20, 39-47 (1987).

8. F. J. J. Clarke, R. McDonald, and B. Rigg, "Modification to theJPC79 colour-difference formula," J. Soc. Dyers Colour. 100,128-132 (1984).

9. M. R. Luo and B. Rigg, "BFD(I:c) colour-difference formula.Part 1-Development of the formula," J. Soc. Dyers Colour.103, 86-94 (1987).

10. D. H. Alman, "CIE Technical Committee 1-29 industrialcolor-difference evaluation progress report," Color Res. Appl.18, 137-139 (1993).

11. A. Robertson, "Colour differences," Die Farbe 29, 273-298(1981).

12. Parametric Effects in Colour-Difference Evaluation, CIE Publ.101 (Central Bureau of the CIE, Vienna, 1993).

13. R. G. Kuehni, "Advances in color-difference formulas," ColorRes. Appl. 7, 19-23 (1982).

14. R. S. Berns, "Synopsis of round table discussion on colorim-etry in industry," in Proceedings of the Seventh Congress ofthe International Colour Association, A. Nemsics and J.Schanda, eds. (Technical University of Budapest and Hungar-ian National Colour Committee, Budapest, Hungary, 1993),Vol. A, pp. 185-187.

15. A. R. Robertson, "CIE Guidelines for coordinated research oncolour-difference evaluation," Color Res. Appl. 3, 149-151(1978).

16. M. R. Luo and B. Rigg, "Chromaticity-discrimination ellipsesfor surface colours," Color Res. Appl. 11, 25-42 (1986).

17. K. Witt, "Three-dimensional threshold of color-differenceperceptibility in painted samples: variability of observers infour CIE color regions," Color Res. Appl. 12, 128-134 (1987).

18. T. Indow, A. R. Robertson, M. von Grunau, and G. H. Fielder,"Discrimination ellipsoids of aperture and simulated surfacecolors by matching and paired comparison," Color Res. Appl.17, 6-23 (1992).

19. D. H. Alman, R. S. Berns, G. D. Snyder, and W. A. Larsen,"Performance testing of color-difference metrics using a colortolerance dataset," Color Res. Appl. 14, 139-151 (1989).

20. R. S. Berns, D. H. Alman, L. Reniff, G. D. Snyder, and M. R.Balonon-Rosen, "Visual determination of suprathreshold color-difference tolerances using probit analysis," Color Res. Appl.16, 297-316 (1991).

21. M. Melgosa, E. Hita, J. Romero, and L. Jimenez del Barco,"Some classical color differences calculated with new formu-las," J. Opt. Soc. Am. A 9, 1247-1253 (1992).

22. M. Melgosa, M. M. P6rez, L. Jimenez del Barco, and E. Hita,"A test of TC 1-29 color-difference formula using some classi-cal datasets (preliminary report)," in Proceedings of the Euro-pean Optical Society Annual Meeting (EOSAM 93) (Departa-mento de Fisica Aplicada, Universidad de Zaragoza, Zaragoza,Spain, 1993), Vol. 3, pp. 37-38.

23. H. B. Mann and D. R. Whitney, "On a test of whether one oftwo random variables is stochastically larger than the other,"Ann. Math. Stat. 18,50-60 (1947).

24. W. H. Kruskal and W. A. Wallis, "Use of ranks in one-criterionvariance analysis," J. Am. Stat. Assoc. 47, 583-621 (1952).

25. W. G. Cochran, "The distribution of the largest of a set ofestimated variances as a fraction of their total," Ann. Eng.(London) 11, 47-61 (1941).

26. R. McDonald, "Industrial pass/fail colour matching, Part 3:colour-matching formulas," J. Soc. Dyers Colour. 96, 486-497(1980).

27. R. S. Berns, "The mathematical development of CIE TC 1-29proposed color difference equation: CIELCH," in Proceed-ings of the Seventh Congress of the International ColourAssociation, A. Nemsics and J. Schanda, eds. (TechnicalUniversity of Budapest and Hungarian National Colour Com-mittee, Budapest, Hungary, 1993), Vol. B, pp. 189-192.

28. J. Romero, J. A. Garcia, L. Jimenez del Barco, and E. Hita,"Color difference formulas tested by use with chromaticdiscrimination data and new formulas," Appl. Opt. 28, 4697-4702 (1989).

29. J. A. Garcia, J. Romero, L. Jim6nez del Barco, and E. Hita,"Improved formula for evaluating color-differential thresh-olds," Appl. Opt. 31, 6292-6298 (1992).

30. D. L. MacAdam, "Specification of small chromaticity differ-ences in daylight," J. Opt. Soc. Am. 33, 18-26 (1943).

31. M. Melgosa, E. Hita, J. Romero, and L. Jim6nez del Barco,"Color-difference formulae," Opt. Pura Apl. 23, 137-145(1990).

8076 APPLIED OPTICS / Vol. 33, No. 34 / 1 December 1994

32. E. Coates, R. C. Kiszka, J. R. Provost, and B. Rigg, "Theaccuracy of colour-difference equations in relation to perceivedcolour differences," Proc. Colour 73, 300-302 (1973).

33. C. Alder, K. P. Chaing, T. F. Chong, E. Coates, A. A. Khalili,and B. Rigg, "Uniform chromaticity scales-new experimentaldata," J. Soc. Dyers Colour. 98, 14-20 (1982).

34. M. R. Luo and B. Rigg, "BFD(I:c) colour-difference formula.

Part 2-Performance of the formula," J. Soc. Dyers Colour.103, 126-132 (1987).

35. L. Sachs, Applied Statistics, Springer Series in Statistics(Springer-Verlag, New York, 1982).

36. R. McDonald, "A review of the relationship between visual andinstrumental assessment of colour difference, part 2," J. OilColour Chem. Assoc. 65, 93-106 (1982).

1 December 1994 / Vol. 33, No. 34 / APPLIED OPTICS 8077