Color Harmonization for Videos

Nikhil Sawant Niloy J. MitraDept. of CSE, Indian Institute of Technology, Delhi

{nikhilus85,niloym}@gmail.com

Abstract

Color harmonization is an artistic technique to adjustthe colors of a given image in order to enhance their visualharmony. In this paper, we present a method to automati-cally improve the color harmony of images. Harmonizationis performed using a carefully designed optimization in thehue space, while keeping the saturation and intensity com-ponents unchanged. Finally, for videos, we pose the prob-lem as an efficient joint optimization in space and time, thusminimizing flickering or visual artifacts in the harmonizedoutput video. We report the performance of our algorithmon a variety of test images and video sequences.

1. Introduction

We are all aware that not all colors match or look nicewith every other colors. Although the notion of harmonyof color can depend on place, time, context, etc., there aresome color arrangements specially those motivated by co-existing colors in nature, which are universally consideredto be harmonious. Color harmonization refers to takingan input image, specially of a scene made of juxtapositionof various colored objects, and enhancing the harmony ofthe scene by minimally modifying or shifting the colors inthe image. The situation is similar for videos where un-harmonized scenes are common due to rapidly changingenvironment. The problem is compounded since in everyframe the scene may be changing, along with fluctuations inillumination. Further, we may have indoor-outdoor scenesback to back which can be very disturbing if colors are notharmonized. We propose a novel method which harmonizesthe given video by solving a joint space-time optimization.Our procedure tries to retain the original colors as far as pos-sible, while minimally shifting the colors which are disturb-ing harmony. Our algorithm is simple and efficient, suitablefor realtime use, and outputs a harmonized video while min-imizing any flicker due to discontinuity in time and motion.

Study of color and light has been an active topic of in-terest since early days. As early as in the seventeenth cen-

tury, Newton gave us the first color wheel. Subsequently,Maxwell came up with important contributions in the fieldof colors [3, 4]. Itten [2] was the first to introduce the colorwheel based on hue information. He proposed a schemeof color harmony based on relative positions of colors on acolor wheel. He introduced two (complementary colors),three (equilateral triangle), four (square), six (hexagon)color harmony schemes. Tokumaru [5] extended this workfor harmony evaluation. He also introduced template-basedharmonization schemes. Such templates attempt to quantifyour perception and understanding of matching colors, al-lowing us to solve the color harmonization problem, whosegoal is to improve the visual appeal of an image, in an opti-mization framework.

Our work is inspired by the recent work on image colorharmonization by Cohen-Or and colleagues [1]. They usedthe templates proposed by Tokumaru et al. [5] to harmonizethe images along with a graph cut method to ensure contigu-ous coloring. Such research efforts have led to the develop-ment of few professional tools for color harmonization forimages. However, to our knowledge, no such system existsfor color harmonizing video sequences.

In a conceivable approach for video color harmonization,one can take each individual frame and independently per-form color harmonization. Such a technique is slow, im-

Figure 1: (Left) Input image. (Right) The color harmonizedimage is visually more pleasing. The output depends on thetype of hue-template [5] used, template X in this case.

practical, and does not exploit the coherence between ad-jacent frames. More importantly, treating each frame inde-pendently results in artifacts in the forms of flickering asshown in Figure 6.

Producing harmonized videos is more involved com-pared to harmonized images for number of reasons like con-stant change in colors, light, scene, etc. Given a video se-quence our method harmonizes the video so that viewer willhave a pleasant experience. We also propose a novel methodto address the segmentation problem, thus helping to main-tain contiguous coloring. In order to maintain contiguouscoloring in time, we automatically detect and make use ofI-frames, commonly used in MPEG-styled video codecs.

The paper is organized as follows: In Section 2, we dis-cuss image color harmonization and propose important im-provements over existing methods. In Section 3, we in-troduce color harmonization for videos, and describe ourmethod for video color harmonization addressing the asso-ciated challenges. We present various results in Section 4.

2. Color harmonization for images

Harmony is a pleasant combination of different things,encountered in various contexts — music, poetry, colors,etc. In absence of harmony, things seems to stand apartleaving an uncomfortable feeling. Lack of color harmony isoften the reason behind boring or chaotic looking images.Historically color harmony often draws inspiration from na-ture which is mostly harmonized. In terms of a color wheel,analogous colors (adjacent colors) and complementary col-ors (opposite colors) leave a harmonized effect on the vi-sual system. In our work, the color harmony is based on thetemplates defined by Takumaru et al. [5]. Figure 2 showsthe various harmonic templates on the basis of hue infor-mation.

There are eight templates {i, V , L, I , T , Y , X , N} de-fined in HSV space. We denote a template by t and its orien-tation by θ. The templates can have any orientation between[0− 2π], i.e., the shaded area within each template t can berotated over the range [0 − 2π]. We call each intermediate

template i template V template L template I

template T template Y template X template N

Figure 2: Color harmonization can be seen as fitting or ap-proximating hue information using harmonizing templates,which were originally proposed by Tokumaru et al. [5].

position as an orientation of t and denote it by θ. The rangeof θ is [0 − 2π] which has been discretized into 360 ori-entations. We did not observe significant benefits for finerdiscretization levels. Notice that in the color wheel, valuesare wrapped around going from 0 to 2π, i.e., the left andright end of the bars in Figure 2 are identified. An image issaid to be harmonized under a particular template t only ifthe entire hue histogram of the image falls within the shadedregion for a particular orientation θ. Combination of tem-plate t and orientation θ is denoted by θt, where θ ∈ [0−2π]and t ∈ {i, V , L, I , T , Y , X , N}. We leave out templateN from further discussions since it is defined only for grayscale images.

To harmonize an image we adjust or shift the image huehistogram so that it falls under the gray region for some θt.However, colors may become undesirable to the viewer e.g.if sky is not blue or if grass is not green and so on. This canbe avoided if minimum adjustment to the hue is performedduring the process of color harmonization. The problem isto select t and its corresponding θ which requires minimumcolor adjustment — we denote such an optimal combinationof θ and t by θH

1.Given an image I , we define a potential function P (θt,

I). The function measures the potential of the image I foran orientation θ under a template t, i.e., θt:

P (θt, I) = [# image pixels in the shaded region].

This simply returns the number of pixels under θt. Wedo not take saturation [1] into consideration because it wasfound that results are not only comparable, but also this po-tential function is more computationally efficient, and henceis better suited for video processing.

Our first task is to find out an optimal orientation of θt

for a fixed template t, which we call θot2. The potential

is calculated by considering all the values θ for the giventemplate t. We select optimal θ, i.e., θot such that:

θot = max(P (θt, I)), θ ∈ [0, 2π].

We maximize the value of potential function because ithelps in minimizing the hue adjustment while harmoniza-tion. Likewise we calculate θot for each template. Tochoose harmonizing template (θH ) we choose θot with max-imum potential among all θot:

θH = max(P (θot, I)), T ∈ {i, V, L, I, T, Y,X}.

Therefore θH gives us the harmonizing template. P (θH , I)is the maximum possible value of the potential function fora given image I over all choices of t and θ.

1θH : Both θ and t are variable, we are not only required to find optimalθ but also optimal template t, which together forms the optimal solution.

2θot : θ is variable but t is fixed i.e., θ corresponds to the fixed t, thisgives the optimal solution for given template t.

Figure 3: Representative hue histogram before (left) andafter (right) color adjustment. Hue values are adjusted suchthat the non-zero parts of the hue-histogram lie inside theshaded (in green) region.

Since perception of colors additionally depends on manyfactor such as context, emotion, culture, likings, etc., weprovide an additional option in our harmonization schemeto allow the user to choose a template t and a desired orien-tation θ.

Color adjustment. Once we find out the θH we knowboth template t and θ for which image potential is the high-est. In order to make the given image optimally harmo-nized we need to adjust color such that entire hue histogramof the given image falls under θH . Simple linear contrac-tion method has been used to adjust the hue values fallingoutside the current template to fall within the template asshown in Figure 3. We need to split the hue space in orderto make image hue to fall under the shaded region. Split-ting points are selected such that linear contraction neededto adjust the hue of the image is constant. After applyingthe contraction, the entire hue histogram lies within the θH

producing a harmonized image. This simple method usedfor color adjustment helps in further speeding up the pro-cess which is essential when we work on videos.

Segmentation problem. The segmentation problem isdue to splitting of colors present within the same region inan image. A continuous portion in the image can containdifferent shades of a color. These shades are marginally sep-

Figure 4: The red bar shows the splitting position accordingto a given template and orientation. The green bar is theposition according to our histogram splitting method for re-ducing segmentation artifacts. Corresponding resultant im-ages are shown in Figure 5.

arated along the hue scale. During color adjustment it mayhappen that different shades of a color may get mapped tocolors with larger distances on the hue scale. This makesdual colors to appear in the same region as shown in Fig-ure 5 leading to visually objectionable artifacts.

In Figure 4, the red bar shows the original splitting po-sition according to a selected template and orientation. Ifcolors are adjusted according to the original splitting posi-tion, we might get a segmentation problem. We propose asimple technique to prevent such an artifact. A new splittingposition is chosen as follows:

for i = 0 to IH(S) dofor j = 0 to δ do

if IH(S + j) = i thenreturn (S + j)

elseif IH(S − j) = i then

return (S − j)end if

end ifend for

end forreturn S

Here S is the original splitting position, IH(S) is thevalue of hue histogram of image I at S. We define a pa-rameter δ which is maximum possible variation of splittingposition from original splitting position S. This algorithmreturns a new splitting position. The method considers δlocations from original splitting position in both directionand finds out the position with the minimum number of pix-els present (see Figure 4). We expect a splitting positionto be present in an empty region in the hue histogram, but

Figure 5: (Left) Original image, (middle) color harmonizedresult with segmentation artifacts, (right) final image usingour splitting position refinement. Our method significantlyreduces color bleeding.

Figure 6: A video sequence, when color harmonized by simply processing each frame individually, results in flickering.

this is not always possible. Figure 5 shows the improve-ment achieved when colors are adjusted according to thenew splitting position.

The splitting position with zero value on hue histogrammost of the times than not removes segmentation problem.But one can never guarantee that finding such a position willremove every occurrence of segmentation problem. The pa-rameter δ should be chosen in such a way that it is suffi-ciently large to find a suitable position. Also care should betaken that this position does not come under shaded regionfor the given template and orientation.

Using this method it is possible to remove segmentationproblem for most of the images. While graph cut basedmethods [1] give better results, they are usually slow. Forreal-time processing of video frames, we found that ourmethod is more suitable without noticeable compromise invisual quality.

Two parameters variation. We further modify templateX with two parameters variation. During color adjustmentit is expected that image does not lose its original colorsas far as possible and our method of two parameter varia-tion extends this idea. We call this method two parametersvariation because here we not only allow variation in θ butalso allow variation in distance between shaded region also.We allow template to adjust itself so that it fits accordingto image hue histogram. Hence there is minimum color ad-justment in the harmonized image. We limit the variationin template to keep the final image harmonized. Figure 7shows an example of image harmonization using two pa-rameter variation approach.

3. Color harmonization for videos

Often we encounter mismatched colors leading to unnat-ural appearances in videos. Such video sequences can bepainful to the eyes. We extend the the basic scheme for im-age color harmonization to videos.

Videos are nothing but a sequence of image frames. Eachframe differs slightly from the previous one to create theillusion of motion. Video is said to be harmonized if all the

frames carry harmony not only individually, but also withrespect to groups of adjacent frames. Such a video sequenceis pleasant. A sequence of disturbing frames or even a singledisturbing frame can affect the harmony of the entire video.We propose a technique which produces a video sequencewhich is harmonized and pleasant for the viewer.

Approach. We start with a simple approach, where weextract out the frames of a video and individually harmo-nize each frame. This is done by harmonizing all frames ofa given video using the above stated method, i.e., we calcu-late θH . All the remaining steps like adjusting colors andhandling segmentation problem are done similarly.

We follow this procedure for all the video frames oneby one. The frames are stitched back to produce the outputvideo.

Shortcomings. This simple approach makes each frameharmonized individually. But we observed that the obtainedvideo contains some flickering artifacts. This flickering isdue to noticeable change in the colors between the adjacentframes belonging to the same scene, as shown in Figure 6.

In the above approach each frame is harmonized irre-spective of its adjacent frames. Since a frame shows a slightchange from its previous frame, it is possible that this maycause a change in θ. If the change is small then differencebetween adjacent frames is not noticeable. However, if the

Figure 7: Original image (left), harmonized image usingstandard template X (middle), and the result after two pa-rameter variation (right). The final result is not only colorharmonized, but also exhibits less color variation as com-pared to the middle image.

Figure 8: Our joint space-time optimization approach results in a flicker-free color harmonized video. A naive approach onthe same sequence results in artifacts as seen in Figure 6.

change is large, as in Figure 6, then it becomes noticeable.In order to eliminate this artifact, adjacent frames need to becorrelated during the harmonization process. As a solutionto this problem, we propose an improved method for videocolor harmonization.

Note that flickering artifacts are not always due to changein θ, it is also possible that the template t may change be-tween adjacent frames. During the calculation of θH thereis scope for varying θ as well as template t. Such changes intemplate never go unnoticed in video unless change of sceneis present at the same junction. As a solution we select atemplate t initially, and subsequently only allow variationsin θ, i.e., selecting θot instead of θH . For the remainingdiscussion we fix the value of t and only consider changein θ. We have chosen template X as it harmonizes imageswith least change in color thus preserving the natural colorcontent of images and producing more realistic images.

Improvement. As we discussed in the earlier sections,flickering artifacts are observed when each frame is har-monized individually. Hence we modify our approach toavoid such artifacts by coupling or correlating the adjacentframes.

Grouping. In order to remove the flickering artifacts weconcentrate on processing of the frames in groups ratherthan individually. Certain number of subsequent frames aregrouped together as shown in Figure 9. The mean of huehistogram of all the images belonging to a group is calcu-lated. The orientation θ for a group is obtained by consid-ering the mean hue values calculated for that group. All theframes belonging to that particular group are harmonizedwith this calculated θ. Here θ is the resultant of all theframes, hence it harmonizes image with acceptable (close tothe minimum) color adjustment if not the minimum. Choiceof number of frames per group depends on the video se-quence and needs to be adjusted accordingly.

As a result no artifacts for the frames within a group areobserved. But still with this approach there might be visi-ble flickering at group boundaries. This indicates grouping

removes intra-group flickering but can lead to inter-groupartifacts. Such an inter-group flickers is a result of a largedifference between mean hue values between two groups.

Overlapping. Overlapping method is slight improvementover grouping with a small variation. Instead of makingdisjoint groups of frames, we overlap frames between twoadjacent groups as shown in Figure 9. This helps in reduc-ing the inter-group flickering. The common frames betweentwo groups help to change the mean hue values graduallyrather than drastically. This method gives pleasing resultswith reduced artifacts at the cost of increased computingtime.

We observed that inter-group flickering is due to changeof scene. When the two adjacent groups contain the framesfrom same scene, flickering was often not noticeable. Butwhen a group contains frames from two different scenesthen the adjacent groups, both preceding and succeeding,show flickering. This suggests that the group containingframes from two different scenes causes the flickering. Ifwe can avoid such groups then inter-group flickering can becompletely removed. One method to remove such groups isby disallowing frames from two different scene in the same

Figure 9: (Left) Non-overlapping grouping of frames forspace-time color harmonization. Such a scheme can still re-sult in minor flickers across group boundaries. A slightlymore expensive alternative, which uses a sliding win-dow (right) for space-time color harmonization, producesin smoother and more pleasing videos.

group. For this purpose we need to identify scene changesin the video sequence.

I-frame detection. We consider I-frame as the first frameof the scene, each scene always starts with I-frame. Wedetect I-frame so that we can avoid grouping the frames be-longing to different scenes under the same group. Furthersometimes there is an abrupt change between frames of thesame scene. We would like to identify such frames and treatthem as I-frames.

Our next job is to detect such frames. To detect I-frameswe use the square of absolute difference for each pixel be-tween adjacent frames. This error measure is commonlyreferred to as the mean square error (MSE). An appropri-ate threshold value is set: If the MSE is greater than thethreshold value, then that particular frame is classified as anI-frame (see Figure 10).

Using this approach we find two consecutive I-frames.The frames between two I-frames are harmonized us-ing overlapping approach with I-frame as a start of sub-sequence. Since between two I-frames there is little changein hue values of the frame no flickering artifact is seen, alsochange of θ is normally observed at I-frames which goes un-noticed because of change in scene. Hence this is a methodto remove flickering artifacts, the result of this techniquecan be seen in Figure 8.

4. Results and applications

Color harmonization helps to harmonize the image, andimprove the balance among the colors present in the image.Our software automatically calculates the best suited har-

0 500 1000 1500 2000 2500 3000 35000

5

10

15

20

25

Frame Number

Mea

n S

qu

are

Erro

r (M

SE)

Figure 10: Mean square errors between two successiveframes are thresholded using a suitable threshold, indicatedin red, to identify I-frames.

Figure 11: Input image (top left) and the color harmonizedresults using template T (top right), template V (bottom left)and template X (bottom right), respectively.

monic template and its orientation for a given image. Theuser can also change both the template and its orientationaccording to her preference. This allows the user to recolorthe image according to her color preference. The techniquecan be used to build tools to help users choose harmoniouscolor for webpages, presentations, wrappers for differentproducts, posters, etc.

On an average, harmonizing images of resolution 640×480 took around 3.8 seconds. When a fixed template t is se-lected for harmonization the same set of images, the averagetime was around 0.67 seconds. This time remains compara-ble even if we chose different templates from the given setof templates. We worked on test set of 15 images. Whenboth the template t and the θ were fixed for the same set of

Figure 12: Adjusting the background color according to aninput flag helps to maintain the harmony of the image evenif the flag is added to the original image.

images, the average time went down to 0.14 seconds.This technique can also used to improve the balance be-

tween background and foreground colors. As shown in theFigure 12, we adjust the color of a background image ac-cording to the foreground image, different flags in this ex-ample. One can also map colors between two images, asshown in Figure 13. We can transfer colors between twoflowers, likewise we can use this technique for other objectstoo.

When it comes to videos, if we color harmonize eachframe of the videos individually, we might introduce flick-ering (see Figure 6). This is caused by large differences inorientations or changes in the template across consecutiveframes. Depending on the video sequence, such flickeringmay or may not be visible.

We found that our improved methods provide very goodresults for almost all videos. The resulting video frames arenot only harmonized individually but consecutive framesare harmonized coherently so as to avoid flickering (Fig-ure 8). It may happen that for some subset of sequences har-monization might create a bad result which can be correctedby manually selecting desired template and orientation forthe given subset of frames. One can make an applicationfor automatically harmonizing videos, while providing anintervention mode by which the user can manually tune pa-rameters.

5. Conclusion

We presented a simple and fast technique for color har-monization of images building on state-of-the-art methods.

Figure 13: The flowers on the right derive its colors fromthe images on the left.

Figure 14: An original video frame (left), and the processedvideo frame after color harmonization (right).

For mid-sized images, our method gives close to real-timeperformance.

We also extended our technique to color harmonizevideos. Video sequences can be more chaotic or disturbingsince a number of image frames contribute to the impact ofthe video. Even a small subset of disturbing frames maycreate chaotic effects. Our technique is designed to reducesuch effects. Using a joint space-time optimization, we har-monize each and every frame while maintaining coherencyacross successive frames.

In future work, we would like to give the user additionalcontrol to select subsections of videos in which she seeksharmonization. We also plan to incorporate user guidancefor choosing a template and the corresponding θ-range forselected sections of the input video. Often the same ob-ject has slightly different colors in different scenes. If theuser identifies such objects across frames, then we can alsomaintain color coherency in the object space.

References

[1] Daniel Cohen-Or, Olga Sorkine, Ran Gal, TommerLeyvand, and Ying-Qing Xu. Color harmonization.ACM Transactions on Graphics, 25(3):624–630, 2006.

[2] Johannes Itten. The Art of Color: the subjective expe-rience and objective rationale of color. Van NostrandReinhold, 1973.

[3] J. C. Maxwell. On the theory of compound colours.Philosophical Transactions, 150:57–84, 1860.

[4] J. C. Maxwell. Experiments on colour. Transactions ofthe Royal Society Edinburgh, 21:275–298, 1885.

[5] M. Tokumaru, N. Muranaka, and S. Imanishi. Color de-sign support system considering color harmony. Proc.of IEEE Fuzzy Systems, 1:378–383, 2002.