comparing images using color coherence vectorsrdz/papers/pzm-mm96.pdfreview of color histograms. we...

14
Comparing Images Using Color Coherence Vectors Greg Pass Ramin Zabih * Justin Miller Computer Science Department Cornell University Ithaca, NY 14853 gregpass,rdz,[email protected] http://www.cs.cornell.edu/home/rdz/ccv.html Abstract Color histograms are used to compare images in many applications. Their advantages are ef- ficiency, and insensitivity to small changes in camera viewpoint. However, color histograms lack spatial information, so images with very different appearances can have similar his- tograms. For example, a picture of fall fo- liage might contain a large number of scattered red pixels; this could have a similar color his- togram to a picture with a single large red ob- ject. We describe a histogram-based method for comparing images that incorporates spa- tial information. We classify each pixel in a given color bucket as either coherent or inco- herent, based on whether or not it is part of a large similarly-colored region. A color coher- ence vector (CCV) stores the number of coher- ent versus incoherent pixels with each color. By separating coherent pixels from incoherent pixels, CCV’s provide finer distinctions than color histograms. CCV’s can be computed at over 5 images per second on a standard work- station. A database with 15,000 images can be queried for the images with the most sim- ilar CCV’s in under 2 seconds. We show that CCV’s can give superior results to color his- * To whom correspondence should be addressed tograms for image retrieval. KEYWORDS: Content-based Image Re- trieval, Processing, Color Histograms INTRODUCTION Many applications require simple methods for comparing pairs of images based on their over- all appearance. For example, a user may wish to retrieve all images similar to a given im- age from a large database of images. Color histograms are a popular solution to this prob- lem, and are used in systems like QBIC [4] and Chabot [11]. Color histograms are computa- tionally efficient, and generally insensitive to small changes in camera position. Color histograms also have some limita- tions. A color histogram provides no spatial information; it merely describes which colors are present in the image, and in what quanti- ties. In addition, color histograms are sensitive to both compression artifacts and changes in overall image brightness. In this paper we describe a color-based method for comparing images which is similar to color histograms, but which also takes spa- tial information into account. We begin with a 1

Upload: others

Post on 11-May-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Comparing Images Using Color Coherence Vectors

Greg Pass Ramin Zabih∗ Justin MillerComputer Science Department

Cornell UniversityIthaca, NY 14853

gregpass,rdz,[email protected]://www.cs.cornell.edu/home/rdz/ccv.html

Abstract

Color histograms are used to compare imagesin many applications. Their advantages are ef-ficiency, and insensitivity to small changes incamera viewpoint. However, color histogramslack spatial information, so images with verydifferent appearances can have similar his-tograms. For example, a picture of fall fo-liage might contain a large number of scatteredred pixels; this could have a similar color his-togram to a picture with a single large red ob-ject. We describe a histogram-based methodfor comparing images that incorporates spa-tial information. We classify each pixel in agiven color bucket as either coherent or inco-herent, based on whether or not it is part of alarge similarly-colored region. A color coher-ence vector (CCV) stores the number of coher-ent versus incoherent pixels with each color.By separating coherent pixels from incoherentpixels, CCV’s provide finer distinctions thancolor histograms. CCV’s can be computed atover 5 images per second on a standard work-station. A database with 15,000 images canbe queried for the images with the most sim-ilar CCV’s in under 2 seconds. We show thatCCV’s can give superior results to color his-

∗To whom correspondence should be addressed

tograms for image retrieval.

KEYWORDS: Content-based Image Re-trieval, Processing, Color Histograms

INTRODUCTION

Many applications require simple methods forcomparing pairs of images based on their over-all appearance. For example, a user may wishto retrieve all images similar to a given im-age from a large database of images. Colorhistograms are a popular solution to this prob-lem, and are used in systems like QBIC [4] andChabot [11]. Color histograms are computa-tionally efficient, and generally insensitive tosmall changes in camera position.

Color histograms also have some limita-tions. A color histogram provides no spatialinformation; it merely describes which colorsare present in the image, and in what quanti-ties. In addition, color histograms are sensitiveto both compression artifacts and changes inoverall image brightness.

In this paper we describe a color-basedmethod for comparing images which is similarto color histograms, but which also takes spa-tial information into account. We begin with a

1

Page 2: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

review of color histograms. We then describecolor coherence vectors (CCV’s) and how tocompare them. Examples of CCV-based im-age queries demonstrate that they can givesuperior results to color histograms. We con-trast our method with some recent algorithms[8, 14, 15, 17] that also combine spatial in-formation with color histograms. Finally, wepresent some possible extensions to CCV’s.

COLOR HISTOGRAMS

Color histograms are frequently used to com-pare images. Examples of their use in multi-media applications include scene break detec-tion [1, 7, 10, 12, 22] and querying a databaseof images [3, 4, 11, 13]. Their popularity stemsfrom several factors.

• Color histograms are computationallytrivial to compute.

• Small changes in camera viewpoint tendnot to effect color histograms.

• Different objects often have distinctivecolor histograms.

Researchers in computer vision have alsoinvestigated color histograms. For exam-ple, Swain and Ballard [19] describe the useof color histograms for identifying objects.Hafner et al. [6] provide an efficient methodfor weighted-distance indexing of color his-tograms. Stricker and Swain [18] analyze theinformation capacity of color histograms, aswell as their sensitivity.

Definitions

We will assume that all images are scaled tocontain the same number of pixels M . We dis-cretize the colorspace of the image such thatthere are n distinct (discretized) colors. A

color histogram H is a vector 〈h1, h2, . . . , hn〉,in which each bucket hj contains the numberof pixels of color j in the image. Typically im-ages are represented in the RGB colorspace,and a few of the most significant bits are usedfrom each color channel. For example, Zhang[22] uses the 2 most significant bits of eachcolor channel, for a total of n = 64 buckets inthe histogram.

For a given image I, the color histogramHI is a compact summary of the image. Adatabase of images can be queried to findthe most similar image to I, and can returnthe image I ′ with the most similar color his-togram HI′ . Typically color histograms arecompared using the sum of squared differences(L2-distance) or the sum of absolute value ofdifferences (L1-distance). So the most similarimage to I would be the image I ′ minimizing

‖HI − HI′‖ =n∑

j=1

(HI [j] − HI′ [j])2,

for the L2-distance, or

|HI − HI′ | =n∑

j=1

|HI [j] − HI′ [j]|,

for the L1-distance. Note that we are assum-ing that differences are weighted evenly acrossdifferent color buckets for simplicity.

COLOR COHERENCEVECTORS

Intuitively, we define a color’s coherence as thedegree to which pixels of that color are mem-bers of large similarly-colored regions. We re-fer to these significant regions as coherent re-gions, and observe that they are of significantimportance in characterizing images.

2

Page 3: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Figure 1: Two images with similar color histograms

For example, the images shown in figure 1have similar color histograms, despite theirrather different appearances.1 The color redappears in both images in approximately thesame quantities. In the left image the redpixels (from the flowers) are widely scattered,while in the right image the red pixels (fromthe golfer’s shirt) form a single coherent re-gion.

Our coherence measure classifies pixels aseither coherent or incoherent. Coherent pixelsare a part of some sizable contiguous region,while incoherent pixels are not. A color co-herence vector represents this classification foreach color in the image. CCV’s prevent co-herent pixels in one image from matching in-coherent pixels in another. This allows finedistinctions that cannot be made with colorhistograms.

Computing CCV’s

The initial stage in computing a CCV is sim-ilar to the computation of a color histogram.We first blur the image slightly by replacing

1The color images used in this paper can be foundat http://www.cs.cornell.edu/home/rdz/ccv.html.

pixel values with the average value in a smalllocal neighborhood (currently including the 8adjacent pixels). This eliminates small vari-ations between neighboring pixels. We thendiscretize the colorspace, such that there areonly n distinct colors in the image.

The next step is to classify the pixels withina given color bucket as either coherent or in-coherent. A coherent pixel is part of a largegroup of pixels of the same color, while an in-coherent pixel is not. We determine the pixelgroups by computing connected components.A connected component C is a maximal set ofpixels such that for any two pixels p, p′ ∈ C,there is a path in C between p and p′. (For-mally, a path in C is a sequence of pixelsp = p1, p2, . . . , pn = p′ such that each pixel pi

is in C and any two sequential pixels pi, pi+1are adjacent to each other. We consider twopixels to be adjacent if one pixel is among theeight closest neighbors of the other; in otherwords, we include diagonal neighbors.) Notethat we only compute connected componentswithin a given discretized color bucket. Thiseffectively segments the image based on thediscretized colorspace.

Connected components can be computed in

3

Page 4: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

linear time (see, for example, [20]). When thisis complete, each pixel will belong to exactlyone connected component. We classify pixelsas either coherent or incoherent depending onthe size in pixels of its connected component.A pixel is coherent if the size of its connectedcomponent exceeds a fixed value τ ; otherwise,the pixel is incoherent.

For a given discretized color, some of thepixels with that color will be coherent andsome will be incoherent. Let us call the num-ber of coherent pixels of the j’th discretizedcolor αj and the number of incoherent pixelsβj . Clearly, the total number of pixels withthat color is αj + βj , and so a color histogramwould summarize an image as

〈α1 + β1, . . . , αn + βn〉 .

Instead, for each color we compute the pair

(αj , βj)

which we will call the coherence pair for thej’th color. The color coherence vector for theimage consists of

〈(α1, β1) , . . . , (αn, βn)〉 .

This is a vector of coherence pairs, one for eachdiscretized color.

In our experiments, all images were scaledto contain M = 38, 976 pixels, and we haveused τ = 300 pixels (so a region is classifiedas coherent if its area is about 1% of the im-age). With this value of τ , an average imagein our database consists of approximately 75%coherent pixels.

An example CCV

We next demonstrate the computation of aCCV. To keep our example small, we will letτ = 4 and assume that we are dealing with animage in which all 3 color components have

the same value at every pixel (in the RGB col-orspace this would represent a grayscale im-age). This allows us to represent a pixel’scolor with a single number (i.e., the pixel withR/G/B values 12/12/12 will be written as 12).

Suppose that after we slightly blur the inputimage, the resulting intensities are as follows.

22 10 21 22 15 1624 21 13 20 14 1723 17 38 23 17 1625 25 22 14 15 2127 22 12 11 21 2024 21 10 12 22 23

Let us discretize the colorspace so that bucket1 contains intensities 10 through 19, bucket 2contains 20 through 29, etc. Then after dis-cretization we obtain

2 1 2 2 1 12 2 1 2 1 12 1 3 2 1 12 2 2 1 1 22 2 1 1 2 22 2 1 1 2 2

The next step is to compute the connectedcomponents. Individual components will belabeled with letters (A, B, . . .) and we willneed to keep a table which maintains the dis-cretized color associated with each label, alongwith the number of pixels with that label. Ofcourse, the same discretized color can be asso-ciated with different labels if multiple contigu-ous regions of the same color exist. The imagemay then become

B C B B A AB B C B A AB C D B A AB B B A A EB B A A E EB B A A E E

and the connected components table will be

4

Page 5: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Label A B C D E

Color 1 2 1 3 1Size 12 15 3 1 5

The components A, B, and E have more thanτ pixels, and the components C and D lessthan τ pixels. Therefore the pixels in A, B andE are classified as coherent, while the pixels inC and D are classified as incoherent.

The CCV for this image will be

Color 1 2 3α 17 15 0β 3 0 1

A given color bucket may thus contain only co-herent pixels (as does 2), only incoherent pix-els (as does 3), or a mixture of coherent andincoherent pixels (as does 1). If we assumethere are only 3 possible discretized colors, theCCV can also be written

〈(17, 3) , (15, 0) , (0, 1)〉 .

Comparing CCV’s

Consider two images I and I ′, together withtheir CCV’s GI and GI′ , and let the numberof coherent pixels in color bucket j be αj (forI) and α′

j (for I ′). Similarly, let the numberof incoherent pixels be βj and β′

j . So

GI = 〈(α1, β1) , . . . , (αn, βn)〉and

GI′ = 〈(α′1, β

′1) , . . . , (α′

n, β′n)〉

Color histograms will compute the differencebetween I and I ′ as

∆H =n∑

j=1

∣∣(αj + βj) − (α′j + β′

j)∣∣ . (1)

Our method for comparing is based on thequantity

∆G =n∑

j=1

∣∣(αj − α′j)

∣∣ +∣∣(βj − β′

j)∣∣ . (2)

From equations 1 and 2, it follows thatCCV’s create a finer distinction than color his-tograms. A given color bucket j can containthe same number of pixels in I as in I ′, i.e.

αj + βj = α′j + β′

j ,

but these pixels may be entirely coherent inI and entirely incoherent in I ′. In this caseβj = α′

j = 0, and while ∆H = 0, ∆G will belarge.

In general, ∆H ≤ ∆G. This is true evenif we use squared differences instead of abso-lute differences in the definitions of ∆H and∆G. This is because both d(x) = |x| andd(x) = x2 are metrics, so they satisfy the tri-angle inequality

d(x + y) ≤ d(x) + d(y). (3)

If we rearrange the terms in equation 1 we get

∆H =n∑

j=1

∣∣(αj − α′j) + (βj − β′

j)∣∣ .

Applying the triangle inequality we have

∆H ≤n∑

j=1

∣∣(αj − α′j)

∣∣+∣∣(βj − β′j)

∣∣ = ∆G.

EXPERIMENTALRESULTS

We have implemented color coherence vectors,and have used them for image retrieval froma large database. Our database consists of14,554 images, which are drawn from a vari-ety of sources. Our largest sources include the11,667 images used in Chabot [11], the 1,440images used in QBIC [4], and a 1,005 imagedatabase available from Corel. In addition, weincluded a few groups of images in PhotoCDformat. Finally, we have taken a number ofMPEG videos from the Web and segmented

5

Page 6: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

them into scenes using the method describedin [21]. We have added one or two imagesfrom each scene to the database, totaling 349images. The image database thus contains awide variety of imagery. In addition, some ofthe imagery has undergone substantial lossycompression via JPEG or MPEG.

We have compared our results with a num-ber of color histogram variations. These in-clude the L1 and L2 distances, with both 64and 512 color buckets. In addition, we haveimplemented Swain’s opponent axis colorspace[19], and used the color discretization schemehe describes. In each case we include a smallamount of smoothing as it improves the perfor-mance of color histograms. On our database,the L1 distance with the 64-bucket RGB col-orspace gave the best results, and is used as abenchmark against CCV’s.

Hand examination of our database re-vealed 52 pairs of images which contain dif-ferent views of the same scene. Examplesare shown in figures 3 and 4. One im-age is selected as a query image, and theother represents a “correct” answer. Ineach case, we have shown where the sec-ond image ranks, when similarity is com-puted using color histograms versus CCV’s.The color images shown are available athttp://www.cs.cornell.edu/home/rdz/ccv.html.

In 46 of the 52 cases, CCV’s produced betterresults, while in 6 cases they produced worseresults. The average change in rank due toCCV’s was an improvement of just over 75positions (note that this included the 6 caseswhere CCV’s do worse). The average percent-age change in rank was an improvement of30%. In the 46 cases where CCV’s performedbetter than color histograms, the average im-provement in rank was 88 positions, and theaverage percentage improvement was 50%. Inthe 6 cases where color histograms performedbetter than CCV’s, the average rank improve-ment was 21 positions, and the average per-

centage improvement was 48%. A histogramof the change in rank obtained by using CCV’sis shown in figure 2.

When CCV’s produced worse results, it wasalways due to a change in overall image bright-ness (i.e., the two images were almost iden-tical, except that one was brighter than theother). Because CCV’s use discretized colorbuckets for segmentation, they are more sen-sitive to changes in overall image brightnessthan color histograms. We believe that thisdifficulty can be overcome by using a bettercolorspace than RGB, as we discuss in the ex-tensions section of this paper.

Efficiency

There are two phases to the computation in-volved in querying an image database. First,when an image is inserted into the database,a CCV must be computed. Second, when thedatabase is queried, some number of the mostsimilar images must be retrieved. Most meth-ods for content-based indexing include thesedistinct phases. For both color histograms andCCV’s, these phases can be implemented inlinear time.

We ran our experiments on a 50 MHzSPARCstation 20, and provide the resultsfrom color histogramming for comparison.Color histograms can be computed at 67 im-ages per second, while CCV’s can be com-puted at 5 images per second. Using colorhistograms, 21,940 comparisons can be per-formed per second, while with CCV’s 7,746can be performed per second. The imagesused for benchmarking are 232 × 168. Bothimplementations are preliminary, and the per-formance can definitely be improved.

6

Page 7: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Figure 2: Change in rank due to CCV’s. Positive numbers indicate improved performance.

RELATED WORK

Recently, several authors have proposed al-gorithms for comparing images that com-bine spatial information with color histograms.Hsu et al. [8] attempts to capture the spa-tial arrangement of the different colors in theimage, in order to perform more accuratecontent-based image retrieval. Rickman andStonham [14] randomly sample the endpointsof small triangles and compare the distribu-tions of these triplets. Smith and Chang [15]concentrate on queries that combine spatial in-formation with color. Stricker and Dimai [17]divide the image into five partially overlappingregions and compute the first three momentsof the color distributions in each image. Wewill discuss each approach in turn.

Hsu [8] begins by selecting a set of repre-sentative colors from the image. Next, theimage is partitioned into rectangular regions,where each region is predominantly a single

color. The partitioning algorithm makes useof maximum entropy. The similarity betweentwo images is the degree of overlap betweenregions of the same color. Hsu presents re-sults from querying a database with 260 im-ages, which show that the integrated approachcan give better results than color histograms.

While the authors do not report runningtimes, it appears that Hsu’s method requiressubstantially more computation than the ap-proach we describe. A CCV can be com-puted in a single pass over the image, with asmall number of operations per pixel. Hsu’spartitioning algorithm in particular appearsmuch more computationally intensive than ourmethod. Hsu’s approach can be extended tobe independent of orientation and position,but the computation involved is quite substan-tial. In contrast, our method is naturally in-variant to orientation and position.

Rickman and Stonham [14] randomly sam-ple pixel triples arranged in an equilateral tri-

7

Page 8: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Histogram rank: 50. CCV rank: 26.

Histogram rank: 35. CCV rank: 9.

Histogram rank: 367. CCV rank: 244.

Histogram rank: 128. CCV rank: 32.

Figure 3: Example queries with their partner images

8

Page 9: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Histogram rank: 310. CCV rank: 205.

Histogram rank: 88. CCV rank: 38.

Histogram rank: 13. CCV rank: 5.

Histogram rank: 119. CCV rank: 43.

Figure 4: Additional examples

9

Page 10: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

angle with a fixed side length. They use 16levels of color hue, with non-uniform quanti-zation. Approximately a quarter of the pixelsare selected for sampling, and their methodstores 372 bits per image. They report resultsfrom a database of 100 images.

Smith and Chang’s algorithm also partitionsthe image into regions, but their approach ismore elaborate than Hsu’s. They allow a re-gion to contain multiple different colors, andpermit a given pixel to belong to several differ-ent regions. Their computation makes use ofhistogram back-projection [19] to back-projectsets of colors onto the image. They then iden-tify color sets with large connected compo-nents.

Smith and Chang’s image database contains3,100 images. Again, running times are notreported, although their algorithm does speedup back-projection queries by pre-computingthe back-projections of certain color sets.Their algorithm can also handle certain kindsof queries that our work does not address; forexample, they can find all the images wherethe sun is setting in the upper left part of theimage.

Stricker and Dimai [17] compute momentsfor each channel in the HSV colorspace, wherepixels close to the border have less weight.They store 45 floating point numbers per im-age. Their distance measure for two regionsis a weighted sum of the differences in eachof the three moments. The distance mea-sure for a pair of images is the sum of thedistance between the center regions, plus (foreach of the 4 side regions) the minimum dis-tance of that region to the corresponding re-gion in the other image, when rotated by 0, 90,180 or 270 degrees. Because the regions over-lap, their method is insensitive to small rota-tions or translations. Because they explicitlyhandle rotations of 0, 90, 180 or 270 degrees,their method is not affected by these particularrotations. Their database contains over 11,000

images, but the performance of their methodis only illustrated on 3 example queries. LikeSmith and Chang, their method is designed tohandle certain kinds of more complex queriesthat we do not consider.

EXTENSIONS

There are a number of ways in which our algo-rithm could be extended and improved. Oneextension involves generalizing CCV’s, whileanother centers on improving the choice of col-orspace.

Histogram refinement and effec-tive feature extraction

Our approach can be generalized to featuresother than color and coherence. We are in-vestigating histogram refinement, in which thepixels of a given color bucket are subdividedbased on particular features (coherence, tex-ture, etc.). CCV’s can be viewed as a simpleform of histogram refinement, in which his-togram buckets are split in two based on co-herence.

Histogram refinement also permits furtherdistinctions to be imposed upon a CCV. In thesame way that we distinguish between pixels ofsimilar color by coherence, for example, we candistinguish between pixels of similar coherenceby some additional feature. We can apply thismethod repeatedly; each split imposes an ad-ditional constraint on what it means for twopixels to be similar. If the initial histogram isa color histogram, and it is only refined oncebased on coherence, then the resulting refinedhistogram is a CCV. But there is no require-ment that the initial histogram be based oncolor, or that the initial split be based on co-herence.

Consider the example shown in figure 5. Webegin with a color histogram and divide the

10

Page 11: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

Edge Information

Red Blue Green

Incoherent Coherent

Red Blue GreenEdge Not an edge

BlueRed Green

Adjacency

Adjacency

Coherence

Color

Figure 5: An example of histogram refinement.

11

Page 12: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

pixels in each bucket based on coherence (i.e.,generate a CCV). The incoherent pixels canbe refined by tallying the colors of pixels ad-jacent to the incoherent pixels in a particularbin. The coherent pixels, in turn, can be fur-ther refined into those that lie on edges andthose which do not. The coherent pixels onedges can then be further refined by recordingwhat colors the coherent regions are adjacentto. The leaves of the tree, therefore, repre-sent those images that satisfy these successiveconstraints. This set of refinements defines acompact summary of the image, and remainsan efficient structure to compute and compare.

The best system of constraints to imposeon the image is an open issue. Any combi-nation of features might give effective results,and there are many possible features to choosefrom. However, it is possible to take advantageof the temporal structure of a successively re-fined histogram. One feature might serve asa filter for another feature, by ensuring thatthe second feature is only computed on pixelswhich already possess the first feature.

For example, the perimeter-to-area ratio canbe used to classify the relative shapes of colorregions. If we used this ratio as an initial re-finement on color histograms, incoherent pix-els would result in statistical outliers, and thusgive questionable results. This feature is bet-ter employed after the coherent pixels havebeen segregated. We call the choice of a sensi-ble structure of constraints effective feature ex-traction. Refining a histogram not only makesfiner distinctions between pixels, but functionsas a statistical filter for successive refinements.

Choice of colorspace

Many systems based on color histogramsspend considerable effort on selecting a goodset of colors. Hsu [8], for example, assumesthat the colors in the center of the image aremore important than those at the periphery,

while Smith and Chang [15] use several differ-ent thresholds to extract colors and regions. Awide variety of different colorspaces have alsobeen investigated. For example, Swain andBallard [19] make use of the opponent-axis col-orspace, while QBIC [4] uses the Munsell col-orspace.

The choice of colorspace is a particularly sig-nificant issue for CCV’s, since they use the dis-cretized color buckets to segment the image. Aperceptually uniform colorspace, such as CIELab, should result in better segmentations andimprove the performance of CCV’s. A relatedissue is the color constancy problem [9], whichcauses objects of the same color to appearrather differently depending upon the lightingconditions. The simplest effect of color con-stancy is a change in overall image brightness,which is responsible for the negative exam-ples obtained in our experiments. Standardhistogramming methods are sensitive to im-age gain. More sophisticated methods, suchas cumulative histograms [16] or color ratiohistograms [5], might alleviate this problem.

In fact, most proposed improvements tocolor histograms can also be applied to CCV’s.This includes improvements beyond the selec-tion of better colorspaces. For example, someauthors [17] suggest that color moments beused in lieu of histograms. Color momentscould be computed separately for coherent andincoherent pixels.

CONCLUSIONS

We have described a new method for compar-ing pairs of images that combines color his-tograms with spatial information. Most re-search in content-based image retrieval has fo-cused on query by example (where the systemautomatically finds images similar to an inputimage). However, other types of queries arealso important. For example, it is often useful

12

Page 13: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

to search for images in which a subset of an-other image (e.g. a particular object) appears.This would be particularly useful for querieson a database of videos. One approach tothis problem might be to generalize histogramback-projection [19] to separate pixels basedon spatial coherence.

It is clear that larger and larger imagedatabases will demand more complex similar-ity measures. This added time complexity canbe offset by using efficient, coarse measuresthat prune the search space by removing im-ages which are clearly not the desired answer.Measures which are less efficient but more ef-fective can then be applied to the remainingimages. Baker and Nayar [2] have begun toinvestigate similar ideas for pattern recogni-tion problems. To effectively handle large im-age databases will require a balance betweenincreasingly fine measures (such as histogramrefinement) and efficient coarse measures.

Acknowledgements

We wish to thank Virginia Ogle for giv-ing us access to the Chabot imagery, andThorsten von Eicken for supplying additionalimages. Greg Pass has been supported by Cor-nell’s Alumni-Sponsored Undergraduate Re-search Program, and Justin Miller has beensupported by a grant from the GTE ResearchFoundation.

References

[1] Farshid Arman, Arding Hsu, and Ming-Yee Chiu. Image processing on com-pressed data for large video databases. InACM Multimedia Conference, pages 267–272, 1993.

[2] Simon Baker and Shree Nayar. Patternrejection. In Proceedings of IEEE Con-

ference on Computer Vision and PatternRecognition, pages 544–549, 1996.

[3] M. G. Brown, J. T. Foote, G. J. F. Jones,K. Sparck Jones, and S. J. Young. Au-tomatic content-based retrieval of broad-cast news. In ACM Multimedia Confer-ence, 1995.

[4] M. Flickner et al. Query by image andvideo content: The QBIC system. IEEEComputer, 28(9):23–32, September 1995.

[5] Brian V. Funt and Graham D. Finlayson.Color constant color indexing. IEEETransactions on Pattern Analysis andMachine Intelligence, 17(5):522–529, May1995.

[6] J. Hafner, H. Sawhney, W. Equitz,M. Flickner, and W. Niblack. Efficientcolor histogram indexing for quadraticform distance functions. IEEE Transac-tions on Pattern Analysis and MachineIntelligence, 17(7):729–736, July 1995.

[7] Arun Hampapur, Ramesh Jain, and TerryWeymouth. Production model based dig-ital video segmentation. Journal of Mul-timedia Tools and Applications, 1:1–38,March 1995.

[8] Wynne Hsu, T. S. Chua, and H. K. Pung.An integrated color-spatial approach tocontent-based image retrieval. In ACMMultimedia Conference, pages 305–313,1995.

[9] E. H. Land and J. J. McCann. Lightnessand Retinex theory. Journal of the Opti-cal Society of America, 61(1):1–11, 1971.

[10] Akio Nagasaka and Yuzuru Tanaka. Au-tomatic video indexing and full-videosearch for object appearances. In 2ndWorking Conference on Visual DatabaseSystems, October 1991.

13

Page 14: Comparing Images Using Color Coherence Vectorsrdz/Papers/PZM-MM96.pdfreview of color histograms. We then describe color coherence vectors (CCV’s) and how to compare them. Examples

[11] Virginia Ogle and Michael Stonebraker.Chabot: Retrieval from a relationaldatabase of images. IEEE Computer,28(9):40–48, September 1995.

[12] K. Otsuji and Y. Tonomura. Projection-detecting filter for video cut detection.Multimedia Systems, 1:205–210, 1994.

[13] Alex Pentland, Rosalind Picard, and StanSclaroff. Photobook: Content-based ma-nipulation of image databases. Inter-national Journal of Computer Vision,18(3):233–254, June 1996.

[14] Rick Rickman and John Ston-ham. Content-based image retrieval usingcolor tuple histograms. SPIE proceedings,2670:2–7, February 1996.

[15] John Smith and Shih-Fu Chang. Toolsand techniques for color image re-trieval. SPIE proceedings, 2670:1630–1639, February 1996.

[16] Thomas M. Strat. Natural Object Recog-nition. Springer-Verlag, 1992.

[17] Markus Stricker and Alexander Dimai.Color indexing with weak spatial con-straints. SPIE proceedings, 2670:29–40,February 1996.

[18] Markus Stricker and Michael Swain. Thecapacity of color histogram indexing. InProceedings of IEEE Conference on Com-puter Vision and Pattern Recognition,pages 704–708, 1994.

[19] Michael Swain and Dana Ballard. Colorindexing. International Journal of Com-puter Vision, 7(1):11–32, 1991.

[20] Patrick Winston and Berthold Horn.Lisp. Addison-Wesley, second edition,1984.

[21] Ramin Zabih, Justin Miller, and KevinMai. A feature-based algorithm for de-tecting and classifying scene breaks. InACM Multimedia Conference, pages 189–200, November 1995.

[22] HongJiang Zhang, Atreyi Kankanhalli,and Stephen William Smoliar. Automaticpartitioning of full-motion video. Multi-media Systems, 1:10–28, 1993.

14