One of the most common tasks in imageprocessing is to alter an image’s color.

Often this means removing a dominant and undesirablecolor cast, such as the yellow in photos taken underincandescent illumination. This article describes amethod for a more general form of color correction thatborrows one image’s color characteristics from anoth-er. Figure 1 shows an example of this process, where weapplied the colors of a sunset photograph to a daytimecomputer graphics rendering.

We can imagine many methodsfor applying the colors of one imageto another. Our goal is to do so witha simple algorithm, and our corestrategy is to choose a suitable colorspace and then to apply simple oper-ations there. When a typical threechannel image is represented in anyof the most well-known colorspaces, there will be correlationsbetween the different channels’ val-ues. For example, in RGB space,most pixels will have large values forthe red and green channel if the bluechannel is large. This implies that ifwe want to change the appearanceof a pixel’s color in a coherent way,we must modify all color channels

in tandem. This complicates any color modificationprocess. What we want is an orthogonal color spacewithout correlations between the axes.

Recently, Ruderman et al. developed a color space,called lαβ, which minimizes correlation between chan-nels for many natural scenes.2 This space is based ondata-driven human perception research that assumesthe human visual system is ideally suited for processingnatural scenes. The authors discovered lαβ color spacein the context of understanding the human visual sys-tem, and to our knowledge, lαβ space has never beenapplied otherwise or compared to other color spaces.There’s little correlation between the axes in lαβ space,

which lets us apply different operations in different colorchannels with some confidence that undesirable cross-channel artifacts won’t occur. Additionally, this colorspace is logarithmic, which means to a first approxima-tion that uniform changes in channel intensity tend to beequally detectable.3

Decorrelated color spaceBecause our goal is to manipulate RGB images, which

are often of unknown phosphor chromaticity, we firstshow a reasonable method of converting RGB signalsto Ruderman et al.’s perception-based color space lαβ.Because lαβ is a transform of LMS cone space, we firstconvert the image to LMS space in two steps. The firstis a conversion from RGB to XYZ tristimulus values. Thisconversion depends on the phosphors of the monitorthat the image was originally intended to be displayedon. Because that information is rarely available, we usea device-independent conversion that maps white inthe chromaticity diagram (CIE xy) to white in RGBspace and vice versa. Because we define white in thechromaticity diagram as x = X/(X + Y + Z) = 0.333, y= Y/(X + Y + Z) = 0.333, we need a transform thatmaps X = Y = Z = 1 to R = G = B = 1. To achieve this,we modified the XYZitu601-1 (D65) standard conver-sion matrix to have rows that add up to 1. The Interna-tional Telecommunications Union standard matrix is

(1)

By letting Mitux = (111)T and solving for x, we obtain avector x that we can use to multiply the columns ofmatrix Mitu, yielding the desired RGB to XYZ conversion:

(2)

Compared with Mitu, this normalization procedure con-

XYZ

RGB

=

0 5141 0 3239 0 16040 2651 0 6702 0 06410 0241 0 1228 0 8444

. . .

. . .

. . .

Mitu =

0 4306 0 3415 0 17840 2220 0 7067 0 07130 0202 0 1295 0 9394

. . .

. . .

. . .

0272-1716/01/$10.00 © 2001 IEEE

Applied Perception

34 September/October 2001

We use a simple statistical

analysis to impose one

image’s color characteristics

on another. We can achieve

color correction by choosing

an appropriate source image

and apply its characteristic

to another image.

Erik Reinhard, Michael Ashikhmin, Bruce Gooch,and Peter Shirley University of Utah

Color Transferbetween Images

stitutes a small adjustment to the matrix’s values.Once in device-independent XYZ space, we can con-

vert the image to LMS space using the followingconversion:4

(3)

Combining these two matrices gives the following trans-formation between RGB and LMS cone space:

(4)

The data in this color space shows a great deal of skew,which we can largely eliminate by converting the datato logarithmic space:2

L = log LM = log MS = log S (5)

Using an ensemble of spectral images that representsa good cross-section of naturally occurring images, Rud-erman et al. proceed to decorrelate these axes. Theirmotivation was to better understand the human visualsystem, which they assumed would attempt to processinput signals similarly. We can compute maximal decor-relation between the three axes using principal compo-nents analysis (PCA), which effectively rotates them.The three resulting orthogonal principal axes have sim-ple forms and are close to having integer coefficients.Moving to those nearby integer coefficients, Rudermanet al. suggest the following transform:

(6)

If we think of the L channel as red, the M as green, andthe S as blue, we can see that this is a variant of manyopponent-color models:4

Achromatic ∝ r + g + bYellow–blue ∝ r + g – bRed–green ∝ r – g (7)

Thus the l axis represents an achromatic channel,while the α and β channels are chromatic yellow–blueand red–green opponent channels. The data in thisspace are symmetrical and compact, while we achievedecorrelation to higher than second order for the set ofnatural images tested.2 Flanagan et al.5 mentioned thiscolor space earlier because, in this color space, theachromatic axis is orthogonal to the equiluminant

plane. Our color-correction method operates in this lαβspace because decorrelation lets us treat the three colorchannels separately, simplifying the method.

After color processing, which we explain in the nextsection, we must transfer the result back to RGB to dis-play it. For convenience, here are the inverse opera-tions. We convert from lαβ to LMS using this matrixmultiplication:

(8)

L

M

S

= −−

1 1 11 1 11 2 0

3

30 0

06

60

0 02

2

lαβ

lαβ

=

−−

1

30 0

01

60

0 01

2

1 1 11 1 21 1 0

L

M

S

LMS

RGB

=

0 3811 0 5783 0 04020 1967 0 7244 0 07820 0241 0 1288 0 8444

. . .

. . .

. . .

LMS

XYZ

=−

−

0 3897 0 6890 0 07870 2298 1 1834 0 0464

0 0000 0 0000 1 0000

. . .. . .

. . .

IEEE Computer Graphics and Applications 35

(a)

(b)

(c)

1 Applying asunset to anocean view. (a) Renderedimage1 (imagecourtesy ofSimon Pre-moze), (b) photograph(image courtesyof the NationalOceanic andAtmosphericAdministrationPhoto Library),and (c) color-processedrendering.

Then, after raising the pixel values to the power ten togo back to linear space, we can convert the data fromLMS to RGB using

(9)

Statistics and color correctionThe goal of our work is to make a synthetic image take

on another image’s look and feel. More formally thismeans that we would like some aspects of the distribu-tion of data points in lαβ space to transfer betweenimages. For our purposes, the mean and standard devi-ations along each of the three axes suffice. Thus, wecompute these measures for both the source and targetimages. Note that we compute the means and standarddeviations for each axis separately in lαβ space.

First, we subtract the mean from the data points:

(10)

Then, we scale the data points comprising the syn-thetic image by factors determined by the respectivestandard deviations:

(11)

After this transformation, the resulting data points havestandard deviations that conform to the photograph.Next, instead of adding the averages that we previous-ly subtracted, we add the averages computed for thephotograph. Finally, we convert the result back to RGBvia log LMS, LMS, and XYZ color spaces using Equations8 and 9.

Because we assume that we want to transfer oneimage’s appearance to another, it’s possible to selectsource and target images that don’t work well together.The result’s quality depends on the images’ similarity incomposition. For example, if the synthetic image con-tains much grass and the photograph has more sky in it,then we can expect the transfer of statistics to fail.

We can easily remedy this issue. First, in this exam-ple, we can select separate swatches of grass and skyand compute their statistics, leading to two pairs of clus-ters in lαβ space (one pair for the grass and sky swatch-es). Then, we convert the whole rendering to lαβ space.We scale and shift each pixel in the input image accord-ing to the statistics associated with each of the clusterpairs. Then, we compute the distance to the center ofeach of the source clusters and divide it by the cluster’sstandard deviation σc,s. This division is required to com-

′ =

′ =

′ =

l ltl

sl

t

s

t

s

σσ

ασσ

α

βσ

σβ

α

α

β

β

*

*

*

l l l*

*

*

= −

= −

= −

α α α

β β β

RGB

LMS

=−

− −−

4 4679 3 5873 0 11931 2186 2 3809 0 1624

0 0497 0 2439 1 2045

. . .. . .

. . .

Applied Perception

36 September/October 2001

2 Color correc-tion in differentcolor spaces.From top tobottom, theoriginal sourceand targetimages, fol-lowed by thecorrectedimages usingRGB, lαβ, andCIECAM97scolor spaces.(Source imagecourtesy ofOliver Deussen.)

pensate for different cluster sizes. We blend the scaledand shifted pixels with weights inversely proportionalto these normalized distances, yielding the final color.This approach naturally extends to images with morethan two clusters. We can devise other metrics to weightrelative contributions of each cluster, but weightingbased on scaled inverse distances is simple and workedreasonably well in our experiments.

Another possible extension would be to computehigher moments such as skew and kurtosis, which arerespective measures of the lopsidedness of a distribu-tion and of the thickness of a distribution’s tails. Impos-ing such higher moments on a second image wouldshape its distribution of pixel values along each axis tomore closely resemble the corresponding distributionin the first image. While it appears that the mean andstandard deviation alone suffice to produce practicalresults, the effect of including higher moments remainsan interesting question.

ResultsFigures 1 and 2 showcase the main reason for devel-

oping this technique. The synthetic image and the pho-tograph have similar compositions, so we can transferthe photograph’s appearance to the synthetic image.Figures 3 and 4 give other examples where we show thatfairly dramatic transfers are possible and still producebelievable results. Figure 4 also demonstrates the effec-tiveness of using small swatches to account for the dis-similarity in image composition.

Figure 5 (next page) shows that nudging some of thestatistics in the right direction can sometimes make theresult more visually appealing. Directly applying themethod resulted in a corrected image with too much redin it. Reducing the standard deviation in the red–greenchannel by a factor of 10 produced the image in Figure5c, which more closely resembles the old photograph’sappearance.

IEEE Computer Graphics and Applications 37

3 Differenttimes of day. (a) Renderedimage6 (imagecourtesy ofSimon Pre-moze), (b) photograph(courtesy of theNOAA PhotoLibrary), and (c) correctedrendering.

(a)

(b)

(c)

4 Using swatches. We applied (a) the atmosphere of Vincent van Gogh’s Cafe Terrace on the Place du Forum, Arles, atNight (Arles, September 1888, oil on canvas; image from the Vincent van Gogh Gallery, http://www.vangoghgallery.com) to (b) a photograph of Lednice Castle near Brno in the Czech Republic. (c) We matched the blues of the sky inboth images, the yellows of the cafe and the castle, and the browns of the tables at the cafe and the people at thecastle separately.

(a) (b) (c)

We show that we can use the lαβ color space for huecorrection using the gray world assumption. The ideabehind this is that if we multiply the three RGB chan-nels by unknown constants cr, cg, and cb, we won’tchange the lαβ channels’ variances, but will change theirmeans. Thus, manipulating the mean of the two chro-matic channels achieves hue correction. White is spec-ified in LMS space as (1, 1, 1), which converts to (0, 0,0) in lαβ space. Hence, in lαβ space, shifting the aver-age of the chromatic channels α and β to zero achieveshue correction. We leave the average for the achromat-ic channel unchanged, because this would affect theoverall luminance level. The standard deviations shouldalso remain unaltered.

Figure 6 shows results for a scene rendered usingthree different light sources. While this hue correctionmethod overshoots for the image with the red lightsource (because the gray world assumption doesn’t holdfor this image), on the whole it does a credible job onremoving the light source’s influence. Changing the grayworld assumption implies the chromatic α and β aver-ages should be moved to a location other than (0, 0). Bymaking that a 2D choice problem, it’s easy to browsepossibilities starting at (0, 0), and Figure 6’s third col-umn shows the results of this exercise. We hand cali-brated the images in this column to resemble the resultsof Pattanaik et al.’s chromatic adaption algorithm,7

which for the purposes of demonstrating our approachto hue correction we took as ground truth.

Finally, we showcase a different use of the lαβ spacein the sidebar “Automatically Constructing NPRShaders.”

Gamma correctionNote that we haven’t mentioned gamma correction

for any of the images. Because the mean and standarddeviations are manipulated in lαβ space, followed by atransform back to LMS space, we can obtain the sameresults whether or not the LMS colors are first gammacorrected. This is because log x

γ= γ log x, so gamma cor-

Applied Perception

38 September/October 2001

(a)

(b)

(c)

5 (a) Renderedforest (imagecourtesy ofOliver Deussen);(b) old photo-graph of a tunaprocessingplant at SanctiPetri, referredto as El Bosque,or the forest(image courtesyof the NOAAPhoto Library);and (c) correct-ed rendering.

6 Color correction. Top to bottom:renderings using red, tungsten, andblue light sources. Left to right:original image, corrected using grayworld assumption in lαβ space. Theaverages were browsed to moreclosely resemble Pattanaik’s colorcorrection results.7 (Images cour-tesy of Mark Fairchild.)

rection is invariant to shifing and scaling in logarithmiclαβ space.

Although gamma correction isn’t invariant to a trans-form from RGB to LMS, a partial effect occurs becausethe LMS axes aren’t far from the RGB axes. In our expe-rience, failing to linearize the RGB images results inapproximately a 1 or 2 percent difference in result. Thus,in practice, we can often ignore gamma correction,which is beneficial because source images tend to be ofunknown gamma value.

Color spacesTo show the significance of choosing the right color

space, we compared three different color spaces. Thethree color spaces are RGB as used by most graphicsalgorithms, CIECAM97s, and lαβ. We chose theCIECAM97s color space because it closely relates to thelαβ color space. Its transformation matrix to convertfrom LMS to CIECAM97s7 is

This equation shows that the two chromatic channelsC1 and C2 resemble the chromatic channels in lαβ space,

bar a scaling of the axes. The achromatic channel is dif-ferent (see Equation 6). Another difference is thatCIECAM97s operates in linear space, and lαβ is definedin log space.

Using Figure 2’s target image, we produced scatterplots of 2,000 randomly chosen data points in lαβ, RGB,and CIECAM97s. Figure 7 (next page) depicts these scat-ter plots that show three pairs of axes plotted against eachother. The data points are decorrelated if the data are axisaligned, which is the case for all three pairs of axes in bothlαβ and CIECAM97s spaces. The RGB color space showsalmost complete correlation between all pairs of axesbecause the data cluster around a line with a 45-degreeslope. The amount of correlation and decorrelation inFigure 7 is characteristic for all the images that we tried.This provides some validation for Ruderman et al.’s colorspace because we’re using different input data.

Although we obtained the results in this article in lαβcolor space, we can also assess the choice of color spaceon our color correction algorithm. We expect that colorspaces similar to lαβ space result in similar color cor-rections. The CIECAM97s space, which has similar chro-matic channels, but with a different definition of theluminance channel, should especially result in similarimages (perhaps with the exception of overall lumi-nance). The most important difference is that

ACC

LMS

1

2

2 00 1 00 0 051 00 1 09 0 090 11 0 11 0 22

= −−

. . .

. . .

. . .

IEEE Computer Graphics and Applications 39

To approximate an object’s material properties,we can also apply gleaning color statistics fromimage swatches to automatically generatingnonphotorealistic shading (NPR) models in thestyle of Gooch et al.1 Figure A shows an exampleof a shading model that approximates artisticallyrendered human skin.

Gooch’s method uses a base color for eachseparate region of the model and a global cooland warm color. Based on the surface normal,they implement shaders by interpolating betweenthe cool and the warm color with the regions’base colors as intermediate control points. Theoriginal method requires users to select base,warm, and cool colors, which they perform in YIQspace. Unfortunately, it’s easy to select acombination of colors that yields poor results suchas objects that appear to be lit by a myriad ofdifferently colored lights.

Rademacher2 extended this method by shadingdifferent regions of an object using different sets ofbase, cool, and warm colors. This solution canproduce more pleasing results at the cost of evenmore user intervention.

Using our color statistics method, it’s possible toautomate the process of choosing base colors. InFigure A, we selected a swatch containing anappropriate gradient of skin tones from Lena’sback.3 Based on the swatch’s color statistics, weimplemented a shader that takes the average of

the swatch as base color andinterpolates along the α axis inlαβ space between +σα

t and −σαt.

We chose interpolation along theα axis because a yellow–bluegradient occurs often in naturedue to the sun and sky. Welimited the range to one standarddeviation to avoid excessive blueand yellow shifts.

Because user intervention isnow limited to choosing anappropriate swatch, creatingproper NPR shading models isnow straightforward andproduces credible results, asFigure A shows.

References1. A. Gooch et al., “A Non-Photore-

alistic Lighting Model for Automat-ic Technical Illustration,” ComputerGraphics (Proc. Siggraph 98), ACM Press, New York,1998, pp. 447-452.

2. P. Rademacher, “View-Dependent Geometry,” Proc. Sig-graph 99, ACM Press, New York, 1999, pp. 439-446.

3. L. Soderberg, centerfold, Playboy, vol. 19, no. 11, Nov.1972.

A An example of an automaticallygenerated NPR shader. An imageof a 3D model of Michaelanglo’sDavid shaded with an automatical-ly generated skin shader generat-ed from the full-size Lena image.3

Automatically Constructing NPR Shaders

CIECAM97s isn’t logarithmic. Figure 2 shows the results.Color correction in lαβ space produces a plausible resultfor the given source and target images. UsingCIECAM97s space also produces reasonable results, butthe image is more strongly desaturated than in lαβspace. It doesn’t preserve the flowers’ colors in the field

very well. Using RGB color space doesn’t improve theimage at all. This result reinforces the notion that theproper choice of color space is important.

ConclusionsThis article demonstrates that a color space with

decorrelated axes is a useful tool for manipulating colorimages. Imposing mean and standard deviation onto thedata points is a simple operation, which produces believ-able output images given suitable input images. Appli-cations of this work range from subtle postprocessingon images to improve their appearance to more dramaticalterations, such as converting a daylight image into anight scene. We can use the range of colors measured inphotographs to restrict the color selection to ones thatare likely to occur in nature. This method’s simplicitylets us implement it as plug-ins for various commercialgraphics packages. Finally, we foresee that researcherscan successfully use lαβ color space for other tasks suchas color quantization. �

AcknowledgmentsWe thank everyone who contributed images to this

article. We also thank the National Oceanic and Atmos-pheric Administration for putting a large photographcollection in the public domain. The National ScienceFoundation’s grants 97-96136, 97-31859, 98-18344, 99-77218, 99-78099 and the US Department of EnergyAdvanced Visualization Technology Center (AVTC)/Views supported this work.

References1. S. Premoze and M. Ashikhmin, “Rendering Natural

Waters,” Proc. Pacific Graphics, IEEE CS Press, Los Alami-tos, Calif., 2000, pp. 23-30.

2. D.L. Ruderman, T.W. Cronin, and C.C. Chiao, “Statisticsof Cone Responses to Natural Images: Implications forVisual Coding,” J. Optical Soc. of America, vol. 15, no. 8,1998, pp. 2036-2045.

3. D.R.J. Laming, Sensory Analysis, Academic Press, London,1986.

4. G. Wyszecki and W.S. Stiles, Color Science: Concepts andMethods, Quantitative Data and Formulae, 2nd ed., JohnWiley & Sons, New York, 1982.

5. P. Flanagan, P. Cavanagh, and O.E. Favreau, “IndependentOrientation-Selective Mechanism for The Cardinal Direc-tions of Colour Space,” Vision Research, vol. 30, no. 5, 1990,pp. 769-778.

6. S. Premoze, W. Thompson, and P. Shirley, “GeospecificRendering of Alpine Terrain,” Proc. 10th EurographicsWorkshop on Rendering, Springer-Verlag, Wien, Austria,1999, pp. 107-118.

7. S.N. Pattanaik et al., “A Multiscale Model of Adaptationand Spatial Vision for Realistic Image Display,” ComputerGraphics (Proc. Siggraph 98), ACM Press, New York, 1998,pp. 287-298.

Applied Perception

40 September/October 2001

7 Scatter plots made in (a) RGB, (b) lαβ, and (c)CIECAM97s color spaces. The image we used here is thetarget image of Figure 2. Note that the degree of corre-lation in these plots is defined by the angle of rotationof the mean axis of the point clouds, rotations ofaround 0 or 90 degrees indicate uncorrelated data, andin between values indicate various degrees of correla-tion. The data in lαβ space is more compressed than inthe other color spaces because it’s defined in log space.

(a)

(b)

(c)

Erik Reinhard is a postdoctoralfellow at the University of Utah in thevisual-perception and parallel-graphics fields. He has a BS and aTWAIO diploma in computer sciencefrom Delft University of Technologyand a PhD in computer science from

the University of Bristol.

Michael Ashikhmin is an assis-tant professor at State University ofNew York (SUNY), Stony Brook. Heis interested in developing simple andpractical algorithms for computergraphics. He has an MS in physicsfrom the Moscow Institute of Physics

and Technology, an MS in chemistry from the University ofCalifornia at Berkeley, and a PhD in computer science fromthe University of Utah.

Bruce Gooch is a graduate studentat the University of Utah. His researchinterest is in nonphotorealistic ren-dering. He has a BS in mathematicsand an MS in computer science fromthe University of Utah. He’s an NSFgraduate fellowship awardee.

Peter Shirley is an associate pro-fessor at the University of Utah. Hisresearch interests are in realistic ren-dering and visualization. He has aBA in physics from Reed College,Portland, Oregon, and a PhD in com-puter science from the University of

Illinois at Urbana-Champaign.

Readers may contract Reinhard directly at the Dept. ofComputer Science, Univ. of Utah, 50 S. Central CampusDr. RM 3190, Salt Lake City, UT 84112–9205, emailreinhard@cs.utah.edu.

For further information on this or any other computingtopic, please visit our Digital Library at http://computer.org/publications/dlib.

IEEE Computer Graphics and Applications 41

������������� ������ ����������������� ����������������

������ �����������������

A K Peters Computer Graphicsand Applications . . . .

� ������ ���� ����������������

������ ������������� ���

�� ��� ����������������������������� ����!��������"��#�������$��%�� &������'���

� ������� ����������������������� ���������%�(����)���������� ����&��������'���

��������������� ������������������������� ������ �!� �""���#�$�� �!� � !%���������&��'��������(��)))���'��������(

��!������ ����������������*������+,��*���������� ����& �������'���