a gentle introduction to color harmonization techniques

Universidade Federal do Rio de Janeiro - UFRJ - Campus Cidade Universitária - Rio de Janeiro - Ilha do Fundão, CEP: 21941-972 - COPPE/PESC/LCG

Color Harmonization Survey :: A Gentle Introduction to Color Harmonization Techniques :: Seminários de Computação Gráfica II :: September 12, 2013

Color Harmonization SurveyA Gentle Introduction to Color Harmonization Techniques

Michel Alves dos Santos

Pós-Graduação em Engenharia de Sistemas e ComputaçãoUniversidade Federal do Rio de Janeiro - UFRJ - COPPECidade Universitária - Rio de Janeiro - CEP: 21941-972

Docentes Responsáveis: Prof. Dsc. Ricardo Marroquim & Prof. PhD. Cláudio Esperança

{michel.mas, michel.santos.al}@gmail.com

September 12, 2013

Michel Alves dos Santos: Laboratório de Computação Gráfica - LCG Pós-Graduação em Engenharia de Sistemas e Computação - PESC



Introduction

Yellow-orange

Yellow-green

Red-orange

Red-violet

Blue-violet

Blue-green

Red

Green

Violet

Blue

Orange

Yellow

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

x

y

D65

500

490

480

470460 380

700

620

600

580

560

540

520




Introduction

Yellow-orange

Yellow-green

Red-orange

Red-violet

Blue-violet

Blue-green

Red

Green

Violet

Blue

Orange

Yellow

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

x

y

D65

500

490

480

470460 380

700

620

600

580

560

540

520


In this presentation wewill talk about ...



Introduction

ColorHarmonization

Yellow-orange

Yellow-green

Red-orange

Red-violet

Blue-violet

Blue-green

Red

Green

Violet

Blue

Orange

Yellow

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

x

y

D65

500

490

480

470460 380

700

620

600

580

560

540

520





Introduction

ColorHarmonization

Yellow-orange

Yellow-green

Red-orange

Red-violet

Blue-violet

Blue-green

Red

Green

Violet

Blue

Orange

Yellow

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

x

y

D65

500

490

480

470460 380

700

620

600

580

560

540

520



But first, what is the concept of Harmony? And Color Harmony?



What’s Harmony?

The Concept of Harmony

[Form + Content] :: [Syntax + Semantics]

Harmony can be defined as a pleasing arrangement of parts,whether it be music, poetry, color, gastronomy, etc.

Appropriate structural relation to sensory perception!




What’s Color Harmony?

The Concept of Color Harmony

Itten [1960]: ‘Color Harmony means relationships on the hue wheel’




What´s the Importance of Color Harmony?

Importance of Colors and Color Harmony

Figure: The Birth of Venus (Sandro Botticelli) and Composition VII (Wassily Kandinsky). Worksthat enchant with their mastery of combining tones, refined aesthetic and unique harmonic sense.

◮ Highlight information and attract attention;

◮ Increase cognitive ability;

◮ Associate syntax to semantics.




Color Harmony and Color Harmonization

Harmony .VS. Harmonization

Harmony =⇒ Property :: Harmonization =⇒ Proccess

Harmonic colors are color sets which have special internalrelationships that are aesthetically pleasing to the human eye.

Color Harmonization is the process to find the best sets of colorswhich will make the image more comfortable to human visions.

Why use Color Harmonization?




Why use Color Harmonization?

Reasons for Using the Color Harmonization

◮ Because harmonic colors are pleasing to the eye;

◮ And because harmonic sets involve the human observer and provide a sense oforder and balance in the visual experience.

How can we find the set of colors more harmonious?




Harmonic Sets

Harmonic Sets and the Colorization Process

Obtaining these harmonic sets can be boring due to the tedious work of colorization.

At this point it is necessary to use smarter approaches!




Harmonic Sets

Harmonic Sets and the Colorization Process

Obtaining these harmonic sets can be boring due to the tedious work of colorization.

Such as the Color Harmonization Techniques!




Color Harmonization Techniques

Papers on Color Harmonization

Before talking about the selected works we will perform a shortexplanation of the first contributions.




The First Contributions

Background of Color Harmonization

◮ First considerations were made by Pythagoras [≈ 500 BC], Aristotle[≈ 320 BC], Plato [≈ 340 BC];

◮ System suitable for the mixing of colours: Leon Battista Alberti [≈1430], Leonardo da Vinci [≈ 1490];

◮ Discoveries in the field of the theory of harmonization: Newton,Goethe, Young and Maxwell [≈ 1703 to 1860];

◮ Introduction of a quantitative representation of harmony: Moon &Spencer [1944], Granville & Jacobson [1944];

◮ Modern Theory of Colours: Munsell [1969], Ostwald & Birren [1969]and Itten [1960];

◮ Introducing a new color circle where harmony is emphasized by thehue component: Itten [1960];

◮ Introduction of 80 harmonic schemes based on studies of Itten:Matsuda [1995];

◮ Harmonic schemes of Tokumaru [2002] to the present day...




Chosen Works ::

1 Geometric Formulation of Classical Color Harmony, 19442 Color Design Support System Considering Color Harmony, 20023 Computational Color Harmony Based on Coloroid System, 20054 Color Harmonization, 2006⋆

5 Color Conceptualization, 20076 Color Harmonization for Videos, 20087 Color Design for Illustrative Visualization, 20088 Harmonic Colormaps for Volume Visualization, 20089 An Improved Method for Color Harmonization, 200910 Image Appearance Exploration by Model-Based Navigation, 200911 Multi-scale Image Harmonization, 201012 Color Harmonization for Augmented Reality, 201013 Data-Driven Image Color Theme Enhancement, 201014 Optimizing Photo Composition, 201015 Cost-effective Feature Enhancement for Volume Datasets, 201016 Online Video Stream Abstraction and Stylization, 201117 ColourVis: Exploring Colour Usage in Paintings Over Time, 201118 Example-Based Image Color and Tone Style Enhancement, 201119 Color Compatibility from Large Datasets, 201220 Improving Photo Composition Elegantly, 201221 Image Composition With Color Harmonization, 201222 Affective Image Colorization, 201223 Understanding and Improving the Realism of Image Composites, 201224 Color Harmonization Enhancement, 201225 Saliency-Guided Consistent Color Harmonization, 2013


1944 20132002 2006 20122009 20112005 2007 20102008



Chosen Works ::




1944 20132002 2006 20122009 20112005 2007 20102008

We will do a quickanalysis of these

works!



Chosen Works ::




1944 20132002 2006 20122009 20112005 2007 20102008


works!

Considering themain contributions!



Chosen Works ::




1944 20132002 2006 20122009 20112005 2007 20102008


works!

Considering themain contributions!

Go get it!



Chosen Works :: #1 :: Moon & Spencer, 1944

•••••••••••••••••••••••••••••••••••••




Chosen Works :: #1 :: Moon & Spencer, 1944

PARRY MOON AND D. E. SPENCER

The importance of area in color harmony wasfirst considered quantitatively in 1845 by Field.7

He said that the areas must be adjusted so thatthe mixture of the colors on a rotating disk givesa neutral gray. The same idea was used subse-quently by Munsell.8 On the other hand, Bezold,9

BrUckel0 and others disagreed with Field, partlybecause of their inability to obtain the identicalcolors used by Field, and partly for more funda-mental reasons. Ostwald" also makes brief men-tion of area and gives the qualitative rule thatthe greater the purity the smaller should be thearea. The general idea of area balance seems tobe a valid one, though Field grasped only thespecial case of balance about an achromatic color.There is no reason why the balance point shouldbe limited to any particular class of colors.Birren12 mentions applications in which a chro-matic balance point is chosen to obtain a par-ticular psychological effect.

The further development of color harmony re-

N'

'4

a '-CYLINDER OF CONSTANT CHROMA

FIG. 1. The metric colorspace, showing the artesiancoordinate system (, 2, w3) and the cylindrical system(r, 0, z).

I George Field, Chromatics (London, 1845).8 A. H. Munsell, A Color Notation (Munsell Color Com-

pany, Baltimore, 1941).D W. v. Bezold, Te Theory of Color (American edition,

Boston, 1876).10 E. BrUcke, Die Physiologie der Farben (Leipzig, 1866);

Les Couleurs (Paris, 1866)." W. Ostwald, Die Harnionie der Farben (Leipzig, 1922);

Farbkunde (Leipzig, 1923); Color Science, translated intoEnglish by J. S. Taylor (Vol. 1, London, 1931; Vol. II,London, 1933).

12 F. Birren, Color Dimensions (Crimson Press, Chicago,1934).

quired a systematic method of arranging colors.The color solids of Munsell and Ostwald weredeveloped independently in America and in Ger-many. Ostwald's basic idea in color harmony isthat any orderly arrangement is harmonious.Thus any orderly geometric arrangement in thecolor solid should constitute a harmony. Ostwaldconsiders the simplest arrangements such as defi-nite spacings on the neutral axis or parallel to it,or simple spacings parallel to the surface of hiscolor solid. From these simplest harmonies, theartist can progress to an almost unlimited numberof more complicated arrangements, none of whichOstwald prohibits. He considers, however, thatthe sense of color harmony is so poorly developedthat one must educate himself to appreciate themore complicated color arrangements. The Ost-wald ideas of harmony can be applied equallywell to the Munsell color solid,3 though this factdoes not seem to be generally realized. Munsellhimself gave very little specific information onharmony, though he did mention the balance ofcolors about a neutral or other point and theorderly arrangement in the color solid.

Thus technical developments in the productionof colorants and progress in color theory haveboth helped in the advancement of color theory.There has been a real advance in the theory ofcolor harmony, but this advance has been almostentirely qualitative. The Ostwald color solid stillrests on a very insecure foundation, and it is onlyduring the past five years that the Munsell sys-tem has been placed on a satisfactory scientificbasis2 with the aid of the C.I.E. system. So it isnot strange that the vague ideas of the artist havenot been translated into more scientific terms.

3. THE METRIC COLORSPACE

To make the theory of color harmony a branchof geometry, one must have first a metric color-space. Such a space was developed for this pur-pose, as noted in a previous paper.' The C.I.E.specification of color gives an affine colorspace,where angles in general do not have any meaningand where distances in different directions cannotbe compared. Such a space cannot be used for thegeometrical formulation of color harmony untila metric is introduced. A Euclidean metric was

13 M. E. Bond and Dorothy Nickerson, J. Opt. Soc. Am.32, 709 (1942).

48

PARRY MOON AND D. E. SPENCER

CHOSENCOLOR

1228I

FIG. 2. Regions of similarity and contrast in a planez = const. (constant Munsell value).

worked out in Munsell notation without directuse of the C.I.E. specification.

4. POSTULATES

Before attempting any new work in this field,one may ask himself, "What is color harmony?"The assumption, which is nowhere stated butwhich seems to be taken for granted throughoutthe literature, is that all color combinations canbe divided into two distinct classes. These twoclasses are harmonies and disharmonies. There issupposed to be a perfectly sharp dividing lineand every color combination must lie in one classor the other, all harmonies being equally pleasingand all disharmonies equally displeasing.

Probably the foregoing is an extreme statementof the usual tacit assumption, though it seems tobe implied in the customary separation of a classof combinations called harmonies. Obviously thesharp division into two classes is not in accord-ance with fact. It seems likely that an artist canuse any combination whatever, depending on theeffect he wants to create: There are no forbiddencombinations, there is no division line.

However, even after omitting such considera-tions as color associations, individual colorpreferences, and suitability to the application,'one finds that color combinations are by nomeans equally pleasing. In fact, they can bearranged in order of aesthetic merit, fromcombinations which expert and layman alike

' E. C. Allen and J. P. Guilford, Am. J. Psychol. 48,643 (1936).

find very beautiful to combinations that no onelikes. The purpose of color harmony theory is toprovide a method of picking out color combina-tions that tend to rank high on this scale ofaesthetic merit. These combinations may becalled "harmonies," in. deference to tradition,though they are not of equal merit and they donot constitute a definite class.

The theory of color harmony developed in thisand subsequent papers rests on the followingtwo postulates:

Pleasing combinations are obtained when(1) the interval between any two colors is

unambiguous, and when(2) colors are so chosen that the points

representing them in co space are related in asimple geometric manner.

These postulates will now be explained. It is afundamental principle of aesthetics that theobserver should not be confused by the stimuli:there should never be a feeling of uncertainty.Applied to a variation in hue (or 0), the principlemeans that the hues should not be so closetogether that there is doubt as to whether theywere intended to be identical or only similar.Another ambiguous region occurs between thosehues which are definitely similar and those whichare definitely dissimilar or complementary.

12 8

Az

CONTRASTI Z,, Ai

/,F A41j;'

12Ar

FIG. 3. Regions of similarity and contrast in a planeO=const. (constant hue).

50

Metric Colorspace and Regions of Similarity

Roots of Mathematical Foundation of Color Harmony




Chosen Works :: #2 :: Tokumaru et al., 2002

•••••••••••••••••••••••••••••••••••••

Abstract - Color design is very important for a product design.In this paper, we propose a system which aims to support such acolor design. Proposed system is composed of 5 parts, such as thepart which evaluates the harmony of colors, the color combiningpart, color scheme image judging part, image word output partand lastly image comparison part. First, the system requires theuser to input a color and his preferring image of color schemeincluding his inputting color with image keyword. Next, the systemselects colors from the Munsell color database, which are inharmony with the color inputted into the system . Then, the systembuilds color schemes to combine the color inputted by the userwith the colors selected from the database by the system. Finally,images of the color schemes are evaluated and outputted the colorcombinations whose images accord with the image keyword whichthe user inputs into the system. Experimental result shows thateffective judgments of color harmony and color image are executedand we can get some good color schemes by the system.

Color Design Support System Considering Color Harmony

Masataka Tokumaru

Faculty of Engineering

Kansai University

[email protected]

Noriaki Muranaka


Kansai University

[email protected]

Shigeru Imanishi


Kansai University

[email protected]

words) by neural networks[2][3]. This method is convenientbecause the system does not require any rules for the relationshipbetween the input and the output, but it is difficult to correctparts of the system and to introduce technical knowledge intothe system because it is difficult to grasp the internal state ofthe system.

On the other hand, many researchers who study color scienceproposed methods and logics to analyze color schemes and colorimages. However a lot of them are not a computer system andthey require estimation and interpolation by user about the partwhere investigation isn't accomplished. Then we compose acomputer system which automatically designs color schemewhose harmony is well and whose image corresponds withuser’s preferring color image[6]~[8].

This paper is comprised of 6 chapters. We show the outlineof the system in the next chapter. The system adopts following




Chosen Works :: #2 :: Tokumaru et al., 2002

with fuzzy membership functions. Our system judges a color

Fig. 1 Flowchart of the proposed system.

Harmonized ColorCombinations User wants

Fuzzy Rulesand

Membership Functions

Harmonized ColorCombinations

Color ImageScale

select the image of the color combination

in the Color Image Scale

KeyWord Image Scale

Matchingselect the image word of the color combination in the KeyWord Image Scale

Image

Munsell Color Database

evaluate harmony of the color combination

Fuzzy Rulesand

Membership Functions

Hue Components Tone Components

Color

User

compare the images

Image Word

Fig. 2 Transform The Munsell Color Space into The Hue and Tone Distribution.

Value

Chroma

Hue

Hue

Value

Chroma

Type i Type V Type L Type I

Type T Type Y Type X Type N

Definition of Tokumaru Templates




Chosen Works :: #3 :: Neumann et al., 2005

•••••••••••••••••••••••••••••••••••••

Computational Color Harmony based on Coloroid System

László Neumann†, Antal Nemcsics

‡, and Attila Neumann

§

†Grup de Gràfics de Girona, Universitat de Girona, and Institució Catalana de Recerca i Estudis Avançats, ICREA, Barcelona, Spain‡Budapest University of Technology and Economics, Hungary

§Institute of Computer Graphics and Algorithms, Vienna University of Technology, Austria†[email protected], ‡[email protected], §[email protected]

(a) (b)

Figure 1: (a) visualization of the overall appearance of a dichromatic color set with ‘caleidoscope’ option of the Color Plan

Designer software and (b) interactive color selection of a dichromatic color set in multi-layer mode, applying rotated regular





Figure 3: A cylindrical projection of the continuous limit-

color curve of the Coloroid

Definition of Coloroid System

Circle of 48 Limit-colors





Figure 14: A dichromatic scene, where colors of diffuse

parts build a harmonic set. It was used in a BRDF study




Chosen Works :: #4 :: Cohen-Or et al., 2006

•••••••••••••••••••••••••••••••••••••

Color Harmonization

Daniel Cohen-Or Olga Sorkine Ran Gal Tommer Leyvand Ying-Qing Xu

Tel Aviv University∗ Microsoft Research Asia†

original image harmonized image

Figure 1: Harmonization in action. Our algorithm changes the colors of the background image to harmonize them with the foreground.

Abstract

Harmonic colors are sets of colors that are aesthetically pleasingin terms of human visual perception. In this paper, we present amethod that enhances the harmony among the colors of a givenphotograph or of a general image, while remaining faithful, as muchas possible, to the original colors. Given a color image, our methodfinds the best harmonic scheme for the image colors. It then allowsa graceful shifting of hue values so as to fit the harmonic schemewhile considering spatial coherence among colors of neighboring

colors are sets of colors that hold some special internal relation-ship that provides a pleasant visual perception. Harmony amongcolors is not determined by specific colors, but rather by their rel-ative position in color space. Generating harmonic colors has beenan open problem among artists and scientists [Holtzschue 2002].Munsell [1969] and Goethe [1971] have defined color harmony asbalance, in an effort to transfer the concept of color harmony froma subjective perspective to an objective one. Although currentlythere is no formulation that defines a harmonic set, there is a con-sensus among artists that defines when a set is harmonic, and there





i type V type L type I type

T type Y type X type N type

Harmonic Templates on the Hue Wheel

The templates may be rotated by an arbitrary angle





(a) (b) (c) (d) (e)

Figure 3: Overview of the color harmonization process. (a) The original image. (b) The hue histogram of the image before and afterharmonization. The top histogram refers to the original image, with best-fitting I-type template superimposed. The bottom histogram showsthe hues shifted to match the template sectors. (c) The resulting harmonized image. Note that the harmonization tried to preserve the originalcolors as much as possible. (d) The user manually rotates the template (top), and the hues are shifted accordingly (bottom). (e) The result ofthe manual choice of template orientation.

Figure 4: Manual choice of harmonic schemes. The original image and its hue histogram are displayed in the left column. Harmonic templateswith various orientations result in different palettes.





1

2

(a) input (b) nearest sector (c) optimized sector (d) hue histogram

Figure 5: A naive implementation associates colors with their nearest sector, yielding the artifacts in the middle image (b). Using optimizedgraph-cut labeling alleviates the problem, producing a more coherent result (c). The hue histogram and the harmonic scheme are shown in (d),to visualize the source of the problem: in the naive implementation, hues which are nearly equally close to both sectors of the template may“choose” their sector arbitrarily, and this causes color discontinuities in the resulting image. Note that when the optimization (c) is applied,two pixels with exactly the same color are not necessarily shifted to the same sectors, since we take into account the spatial relation amongpixels.

F(X ,(m,α)) = ∑p∈X

∥

∥

∥H(p)−ETm(α)(p)

∥

∥

∥·S(p) ,

1

2

H ′(p) =C(p)+w

2(1−Gσ (‖H(p)−C(p)‖)) ,





original image harmonization (mirror-L template) harmonization (V template) harmonization (Y template)

(a) original (b) harmonization result (X template) (c) harmonization result (mirror-L template)

Some Applications of Color Harmonization Technique

Useful tool for designing posters, presentations, web sites and other kinds of combined imagery




Chosen Works :: #5 :: Hou & Zhang, 2007

•••••••••••••••••••••••••••••••••••••

Color Conceptualization

Xiaodi HouDepartment of Computer Science and

Engineering, Shanghai Jiao Tong UniversityNo.800, Dongchuan RoadShanghai, China, 200240

http://bcmi.sjtu.edu.cn/~houxiaodi

Liqing ZhangDepartment of Computer Science and

Engineering, Shanghai Jiao Tong UniversityNo.800, Dongchuan RoadShanghai, China, 200240

[email protected]

ABSTRACT

In this paper, we propose a method to manipulate colors ofan image. Based on a library of natural color images, oursystem evolves several prototypes of color distribution of thelibrary, which we call “color concepts”. By applying thesecolor concepts on an input image, a user can easily changethe mood of image colors in a global manner. Our resultsof photographs and paintings indicate that this method iscapable of high-quality color manipulations.

Categories and Subject Descriptors

I.4.8 [IMAGE PROCESSING AND COMPUTER VI-

SION]: Scene Analysis - Color

General Terms

Algorithms, Design, Experimentation

Keywords

Color Concept, Color Transfer, Scene Analysis

“Forest” conceptualized image“Autumn” conceptualized image

The input images are applied to “spring” and “autumn” concepts respec�vely.

The le� part of each image displays the original input, while the right part is

the output of color conceptualiza�on.

Figure 1: Examples of color conceptualization.

enhancement. By fitting the color histogram of a harmonicscheme, incongruent colors can be replaced by colors thatsatisfy established harmonic rules. However, color harmo-nization is a full automatic approach. A user cannot use itto edit colors based on his/her subjective ideas.





“Forest” conceptualized image“Autumn” conceptualized image

The input images are applied to “spring” and “autumn” concepts respec�vely.

The le� part of each image displays the original input, while the right part is

the output of color conceptualiza�on.

Examples of Color Conceptualization





Coast

Warm

Cold

(a) input image (b) “Forest” conceptualized image




Chosen Works :: #6 :: Sawant & Mitra, 2008

•••••••••••••••••••••••••••••••••••••

Color Harmonization for Videos

Nikhil Sawant Niloy J. Mitra

Dept. of CSE, Indian Institute of Technology, Delhi

{nikhilus85,niloym}@gmail.com

Abstract

Color harmonization is an artistic technique to adjust

the colors of a given image in order to enhance their visual

harmony. In this paper, we present a method to automati-

cally improve the color harmony of images. Harmonization

is performed using a carefully designed optimization in the

hue space, while keeping the saturation and intensity com-

ponents unchanged. Finally, for videos, we pose the prob-

lem as an efficient joint optimization in space and time, thus

minimizing flickering or visual artifacts in the harmonized

output video. We report the performance of our algorithm

on a variety of test images and video sequences.

1. Introduction

tury, Newton gave us the first color wheel. Subsequently,

Maxwell came up with important contributions in the field

of colors [3, 4]. Itten [2] was the first to introduce the color

wheel based on hue information. He proposed a scheme

of color harmony based on relative positions of colors on a

color wheel. He introduced two (complementary colors),

three (equilateral triangle), four (square), six (hexagon)

color harmony schemes. Tokumaru [5] extended this work

for harmony evaluation. He also introduced template-based

harmonization schemes. Such templates attempt to quantify

our perception and understanding of matching colors, al-

lowing us to solve the color harmonization problem, whose

goal is to improve the visual appeal of an image, in an opti-

mization framework.

Our work is inspired by the recent work on image color

harmonization by Cohen-Or and colleagues [1]. They used

the templates proposed by Tokumaru et al. [5] to harmonize

the images along with a graph cut method to ensure contigu-





Figure 1: (Left) Input image. (Right) The color harmonized

image is visually more pleasing. The output depends on the

type of hue-template [5] used, template X in this case.





template i template V template L template I

template T template Y template X template N

Figure 2: Color harmonization can be seen as fitting or ap-

proximating hue information using harmonizing templates,

which were originally proposed by Tokumaru et al. [5].

{i , V , L, I, T , Y , X , N}





Figure 6: A video sequence, when color harmonized by simply processing each frame individually, results in flickering.

Figure 8: Our joint space-time optimization approach results in a flicker-free color harmonized video. A naive approach on

the same sequence results in artifacts as seen in Figure 6.

The method used in this work avoids flicker and other artifacts




Chosen Works :: #7 :: Wang et al., 2008

•••••••••••••••••••••••••••••••••••••

Color Design for Illustrative Visualization

Lujin Wang, Joachim Giesen, Kevin T. McDonnell, Member, IEEE, Peter Zolliker, Member, IEEE,

and Klaus Mueller, Senior Member, IEEE

Abstract—Professional designers and artists are quite cognizant of the rules that guide the design of effective color palettes,

from both aesthetic and attention-guiding points of view. In the field of visualization, however, the use of systematic rules

embracing these aspects has received less attention. The situation is further complicated by the fact that visualization often uses

semi-transparencies to reveal occluded objects, in which case the resulting color mixing effects add additional constraints to the

choice of the color palette. Color design forms a crucial part in visual aesthetics. Thus, the consideration of these issues can be of

great value in the emerging field of illustrative visualization. We describe a knowledge-based system that captures established

color design rules into a comprehensive interactive framework, aimed to aid users in the selection of colors for scene objects and

incorporating individual preferences, importance functions, and overall scene composition. Our framework also offers new

knowledge and solutions for the mixing, ordering and choice of colors in the rendering of semi-transparent layers and surfaces. All

design rules are evaluated via user studies, for which we extend the method of conjoint analysis to task-based testing scenarios.

Our framework’s use of principles rooted in color design with application for the illustration of features in pre-classified data

distinguishes it from existing systems which target the exploration of continuous-range density data via perceptual color maps.

Index Terms—Color design, volume rendering, transparency, user study evaluation, conjoint analysis, illustrative visualization

1 INTRODUCTION

Recent years have seen multifarious efforts to better integrate and exploit properties of human visual perception into visualization de-sign. Illustrative rendering techniques have been developed that ren-der the scene at different levels of abstractions [30] and detail [32] or in different rendering styles [5], with applications ranging from in-formation visualization [20] to full-scale volume rendering. In these approaches, the levels of abstraction are most often controlled by a task- or object-dependent importance parameter [31]. Another per-ception-motivated strategy is to guide viewer attention to salient features [16]. Color can play a major role in these particular efforts. However, there is no illustrative rendering system so far that incor-porates rules from color design directly into the visualization engine. Yet, the working scenario of graphics designers is quite similar to

Professional designers and artists are quite cognizant of rules that guide the design of color palettes, not only from an aesthetic point of view but also from an attention-guiding, salient one. Likewise, visu-alization is not only concerned with providing a pleasing image – it also has a mission, that is, to help users gain quick and accurate in-sight into the visualized data [34]. Our framework captures these known color design rules into a knowledge-based system, which then combines them with scene analysis, user preferences, and importance functions to derive appropriate colorizations. These latter considera-tions are typically not embodied in currently existing frameworks.

The scope of our system is image-centric as well as volume visu-alization. Both often use semi-transparencies to reveal occluded ob-jects. Here, color mixing effects and the perceived depth-ordering of

••••





Fig. 12: Illustrative visualizations of a six-dimensional dataset using illustrative parallel coordinates. (a) Ideal visualization with appropriate

weightings and color choices, and the use of the local model in overlapping areas. (b) Improper weightings are employed. The blue cluster

no longer seems to be in front. (c) The use of improper weightings and the disabling of the local model results in a confusing visualization.

(a) (b) (c)

αα

in the same compositing order, but with opacities assigned in the

Fig. 8. Color mixing in a volume rendered body. (a) Cyan and

α α2 α − α e two α α − α r e opacity

tes am-

es ont

color re heavily

po-ted

d 7d, when two opposite hues are mixed, no false a

prevent

∩

Fig. 6. Colorizations of Transmission Electron Microscopy (TEM)

data. From left to right, classes A, B, or C (the small oblong,

elliptical cells) were most important.

stem suggests hues cyan

e same provides guidelines on

of each class is r the former,

importance, 0.7 for age size) e area to

meter can portant classes

portance a higher

portance) ations

varying




Chosen Works :: #8 :: Wang & Mueller, 2008

•••••••••••••••••••••••••••••••••••••

Harmonic Colormaps for Volume Visualization

Lujin Wang and Klaus Mueller

Center for Visual Computing, Computer Science, Stony Brook University, NY

Abstract

Color design forms a crucial part in visual aesthetics, and it has been shown that a visually aesthetic visuali-

zation will be looked at more carefully. An important role plays here the choice of a colormap that is com-

posed of harmonic colors. This paper presents an interface that allows users to choose harmonic colors in

volume visualization applications. In addition, it describes mechanisms by which non-harmonic colormaps

can be converted to harmonic ones, but keeping lightness constant to preserve the original contrast relation-

ships. Finally, we also show how harmonic colors can be used for the highlighting of important volume fea-

tures.

Categories and Subject Descriptors (according to ACM CSS): I.3.3 [Computer Graphics]: Display Algorithms.

1 Introduction

In visualization and volume graphics, image and volume

datasets typically come in form of 2D and 3D arrays of

scalar densities, which are mostly obtained via simula-

accomplishing perceptually uniform color scales (color-

maps). The research presented in [9] specifically ad-

dressed the lightness component of these colormaps,

devising a simple method to specify such colormaps on

commodity non-calibrated displays. Other work [5] de-

IEEE/ EG Symposium on Volume and Point-Based Graphics (2008)

H.-C. Hege, D. Laidlaw, R. Pajarola, O. Staadt (Editors)





Figure 2: Color harmonization without lightness-preservation (Buckyball dataset). (a) Original rendering with

corresponding hue wheel; (b) Rendering with colormap harmonization using just the hue shifting (hue wheel on the

right); (c) Rendering with hue harmonization, but with our lightness-preservation scheme applied after hue shifting

(same hue wheel than (b)).

(a) (b) (c)

scheme to preserve lightness after hue shifting. The hue Then, when the user employs the hue wheel for a harmo-

Figure 5: Volume rendered segmented frog dataset. (a)

(a) (b) (a) (b)





Figure 6: Composition of shockwave and jet dataset rendering.(a) Non-harmonic trans-

fer function; (b) Harmonic transfer function (T type); (c) Harmonic transfer function (V

type).

(a) (b) (c)

blending weight of the back-

the mixture.

shall assume

to control the

ence of color

mi-transparent

restrict our

discussion on achieving

ving har-

We now present a variety of

ng the capa-

thod for

lume visuali-

e 5a we show

ndered frog

with a non-harmonic

on (see the





Figure 6: Features are highlighted one by one: (a) All features are rendered in neutral colors, no feature

is highlighted, (b)-(d) The outside feature is highlighted by increasing the vividness of its color gradually,

while preserving the lightness, (e) The vividness of the outside feature decreases, (f)-(h) The inside feature

is highlighted gradually, (i)-(j) The vividness of the inside feature decreases.




Chosen Works :: #9 :: Huo & Tan, 2009

•••••••••••••••••••••••••••••••••••••

An Improved Method for Color

Harmonization

Xing Huo

Hefei university of Technology, Math Department

Hefei university of Technology, Computer and

Information institute

Hefei,China

Jieqing Tan

Hefei university of Technology, Math Department

Hefei university of Technology, Computer and

Information institute

Hefei,China

Abstract—Harmonic colors are color sets which have special

internal relationships that are aesthetically pleasing to the

human eye. Color harmonization is to find the best sets of

colors which will make the image more comfortable to human

visions. An improved technique of best scheme searching for

automatic color harmonization is proposed in this paper. By

considering the predominant color in image and its

contribution to the harmonic image, this paper formulates a

ratio method to find the best scheme. Moreover this paper

suggests some change to the strategy of color shifting, thus

makes the algorithm more efficient. In addition, this paper

utilizes the conventional technique but produces better results

through a set of simple processing. It is shown that the

efficiency and accuracy of the new technique is significantly

better than the traditional technique.

Keywords-color harmonization;hue;graph-cut;color space

I. INTRODUCTION

Color harmony originated from the theory of white light

al.[5]

presented a system for designing colors based on several color rules, and applied them to a graphical user inter-face (GUI) building tool, Daniel Cohen

[6] proposed a

new method which could automatically harmonizes a given color palette through an optimization process, and provides a means to automatically recolor an arbitrary image. Color harmony also gives some nudges in color transfer

[7,10,11].

Our method is based on the work of Cohen, it can cope with the image with rich colors ,find the most suitable scheme automatically and keep some regions intact. The method proposed in this paper improved the optimization and recoloring method of Cohen’s work. It takes the predominant hue of the image into account as well as a slight change in the recoloring function which minimizes the time cost.

II. HARMONIC SCHEMES

The harmonic schemes in this paper are brought forth by Matsuda

[4].There are eight schemes in Figure 1, each scheme

f





(a) Original Image (b) Result Image(α = 54) (c) Result Image(α = 60)

(d) Hue Histogram of

Original Image(α = 54,60)

(e) Hue Histogram of

Harmonized Image (α =54)

(f) Hue Histogram of

Harmonized Image (α =60)

Figure 2 Result Images and Hue Histograms by Cohen and Improved algorithm

Figure 3. Harmonized background image in respect to foreground





TABLE I. COMPARE COHEN’S ALGORITHM WITH IMPROVED ALGORITHM

Cohen’s

method

Template i V L I T Y X

F-template 2224714 1072405 493496 422310 189772 220565 40446

Orientation

angle

54

Time cost 33984 ms

Improved

method

Template i V L I T Y X

F-template 2514050 1749780 564215 555936 247651 396070 46311

Orientation

angle

60

Time cost 79 ms

hhh snF •=

α

α πα

α

α

α

α

α

work :

)))()((1(2

)()(' pCpHEw

pCpH −−+=σ

(3)

σ

)),(,( αmXF

∈

•−=

Xp

T pSpEpHmXFm

)()()()),(,( )(αα

Where X is the given image, p stands for any pixel in th

α

α

α

2

2

2)( σ

σ

t

etE−

=

σ




Chosen Works :: #10 :: Shapira et al., 2009

•••••••••••••••••••••••••••••••••••••

Image Appearance Exploration by Model-Based Navigation

L. Shapira1, and A. Shamir2 and D. Cohen-Or1

1Tel-Aviv University, Israel2Interdisciplinary Center Herzliya, Israel

Abstract

Changing the appearance of an image can be a complex and non-intuitive task. Many times the target image colorsand look are only known vaguely and many trials are needed to reach the desired results. Moreover, the effect ofa specific change on an image is difficult to envision, since one must take into account spatial image considera-tions along with the color constraints. Tools provided today by image processing applications can become highlytechnical and non-intuitive including various gauges and knobs.In this paper we introduce a method for changing image appearance by navigation, focusing on recoloring im-ages. The user visually navigates a high dimensional space of possible color manipulations of an image. He caneither explore in it for inspiration or refine his choices by navigating into sub regions of this space to a specificgoal. This navigation is enabled by modeling the chroma channels of an image’s colors using a Gaussian MixtureModel (GMM). The Gaussians model both color and spatial image coordinates, and provide a high dimensionalparameterization space of a rich variety of color manipulations. The user’s actions are translated into transfor-mations of the parameters of the model, which recolor the image. This approach provides both inspiration andintuitive navigation in the complex space of image color manipulations.

Categories and Subject Descriptors(according to ACM CCS): Computer Graphics [I.3.6]: Interaction Techniques—Image Processing [I.4.3]: Enhancement—Image Processing [I.4.9]: Applications—Image Processing [I.4.10]: Im-age Representation - Multidimensional—




Chosen Works :: #10 :: Shapira et al., 2009

L. Shapira & A. Shamir & D. Cohen-Or / Image Appearance Exploration by Model-Based Navigation




Chosen Works :: #11 :: Sunkavalli et al., 2010

•••••••••••••••••••••••••••••••••••••

Multi-scale Image Harmonization

Kalyan Sunkavalli∗

Harvard University

Micah K. Johnson†

MIT

Wojciech Matusik‡

Disney Research, Zurich

Hanspeter Pfister§

Harvard University

(a) Source / Target (b) Seamless Cloning (c) Harmonization (d) Close-ups

Figure 1: In traditional image compositing (a) a user applies geometric transformations to a source image (top) and inserts it into a targetimage (bottom). Tools such as the Photoshop Healing Brush use gradient domain compositing to ensure that the composite is seamless (b) butthe inconsistencies between the two images, make the result look unrealistic: the inserted face is much smoother than the rest of the image.Our method “harmonizes” the images before blending them, producing a composite that is seamless and realistic (c). The close-up images(d) compare traditional gradient-domain blending (top) to the harmonized result (bottom).

Abstract

Traditional image compositing techniques, such as alpha mattingand gradient domain compositing, are used to create compositesthat have plausible boundaries. But when applied to images taken

1 Introduction

Combining regions of multiple photographs or videos into a seam-less composite is a fundamental problem in many vision and graph-ics applications, such as image compositing, mosaicing, scene com-




Chosen Works :: #11 :: Gruber et al., 2010

(a) Source / Target (b) Inserting source into target (c) Close-ups (d) Inserting target into source (e) Close-ups

Figure 10: In this example, the source ((a) top) is smooth while the target ((a) bottom) is noisy. When inserting the source into the target,harmonization adds noise to produce a realistic composite (b). Conversely, when the target image is inserted into the source, harmonizationremoves most of the noise to match the images (d).

(a) Source / Target (b) Seamless Cloning (c) Harmonization (d) Close-ups





Smooth

Histogram

Matching

Harmonized

Pyramid

Final Composite

Alpha, Seamless

BoundariesNoise Pyramid

Target Pyramid

Source Pyramid

Smooth

Histogram

Matching

Noise Image

Target Image

Source Image

Pyramid

Compositing

Multi-scale

Harmonization

Noise

Matching

Figure 2: An overview of the Multi-scale Image Harmonizationframework. The input source and target images, and a uniform ran-





•••••••••••••••••••••••••••••••••••••

Color Harmonization for Augmented Reality

Lukas Gruber1, Denis Kalkofen

2 and Dieter Schmalstieg

3

Graz University of Technology

ABSTRACT 2 REAL-TIME COLOR SHIFTING

In this paper we discuss color harmonization for Augmented Reality. Color harmonization is a technique used to adjust the combination of colors in order to follow aesthetic guidelines. We implemented a system which is able to harmonize the combination of the colors in video based AR systems. The presented approach is able to re-color virtual and real-world items, achieving overall more visually pleasant results. In order to allow preservation of certain colors in an AR composition, we furthermore introduce the concept of constraint color harmonization.

To achieve real-time performance, we implemented the color

shifting on the GPU using pre-computed look-up tables. We

identify and execute the necessary color shifts on a per pixel

basis. Our algorithm consists of the following steps:

1. Pre-compute color shift for all templates and orientations

and store in a look-up table (off-line)

2. Select harmonization template T and its orientation θ, according to [1]

3. Select split border between template sectors, according to

[3]

4. Shift the hue value of every pixel, using a pixel shader 1 INTRODUCTION

For each color template T, we compute a single look-up table

before the application is started. This color template look-up

table (CT-LUT) holds the results of each possible color shift, for

each hue value. We set the granularity to 1 degree of arc (0 to

359), resulting in a texture dimension of 360 by 360 texel. Each

texel encodes the mapping result for a single hue in a single

orientation of the template. The different mapping directions for

a single hue have been assigned to different color channels of

the texture. At runtime, a template and an orientation and its

mapping direction is chosen for an image according to [1].

Finally, a simple texture lookup defines the new color per pixel.

In Augmented Reality (AR) oftentimes virtual objects must be

generated without any knowledge about the real world

environment in which they are going to be deployed. This is

particularly relevant for outdoor AR applications (e. g., Layar4),

where neither the application developer nor the content creator

have reasonable control over the real world background, which

will be blended with the virtual content. Even if the knowledge

about the visual properties of the real world surroundings is

available, the dynamics of the real world environment in all but

the most constrained laboratory conditions make it at least very

difficult to ensure that virtual content will always fit into its real

world environment. It is therefore desirable to investigate online

composition algorithms that can adjust the virtual and real image

components to yield a better visual match.

3 FRAME COHERENT COLOR SHIFTING

Frame coherent color harmonization for AR differs strongly

from offline frame coherent color shifting as discussed by

Sawant et al. [3]. Firstly, in an interactive system it is impossible

to process upcoming frames and secondly, AR systems have to

Among the known properties of appealing images, the combination of colors is accepted as one of the most influential





Figure 1. Histogram scaling versus template search restriction. (A) Restricting the template search to only constrained elements may heavily

change the appearance of the entire visualization. The environment in this example consists of a single constrained object (the red stop

sign) which appears in a single color. By restricting the template search to only the hue values of the stop sign, the hue values of the

entire environment will shift towards red. (B) By scaling the histogram at hue values of constrained pixels we are able to compute the

template and its orientation taking into account the remaining part of the imagery. This technique is able to harmonize the environment

using fewer modifications. Notice the slight changes of hue values on the formerly blue arm of the LEGO figure riding the motor-bike.

This will lead to an increased cognitive load, making the AR sign (Figure 1A).

To achieve real-time performance, this technique implements the colorshifting on the GPU using pre-computed look-up tables.





•••••••••••••••••••••••••••••••••••••

Data-Driven Image Color Theme Enhancement

Baoyuan Wang ∗∗ Yizhou Yu†∗ Tien-Tsin Wong‡ Chun Chen∗ Ying-Qing Xu§

∗Zhejiang University †University of Illinois at Urbana-Champaign‡The Chinese University of Hong Kong §Microsoft Research Asia

Figure 1: An example of image color theme enhancement using our method. Left: Original image; middle: recolored result with the“nostalgic” color theme; and right: recolored result with the “lively” color theme.

Abstract

It is often important for designers and photographers to convey orenhance desired color themes in their work. A color theme is typ-ically defined as a template of colors and an associated verbal de-scription. This paper presents a data-driven method for enhancing adesired color theme in an image. We formulate our goal as a unifiedoptimization that simultaneously considers a desired color theme,texture-color relationships as well as automatic or user-specified

Psychological studies confirm the strong associations between col-ors and semantic themes. For instance, “graceful” is often associ-ated with a combination of pinkish and purple colors while “sad”is typically defined using a combination of blue and grey colors.In general, a color theme is a template of colors and an associatedverbal description. Given a color theme, our goal is to performcolor theme enhancement of an image, which seeks a global trans-formation of the colors in the original image so that it maintainsthe realism of natural images while the new color composition is





(a) input image (b) soft segmentation (h) recolored images(g) color histograms for textures Optimization solverTexture Clustering Build Histograms(e) training examples...ThemeDatabase (c) color themes from communities

TextureLibrary (d)(f)... ...Figure 3: The overall pipeline of our framework. Each sub-image in (b) is an influence map of a soft segment. Each soft segment finds the mostrelevant texture class in (f) and adopt the corresponding color histogram in (g). The histograms in (g) are further converted into continuousprobability density distributions using Gaussian Mixture Model(GMM). The user needs to select a desired color theme from (d), then ouroptimization solver takes the prior knowledge (color probabilities), the desired color theme and the image segments into consideration togenerate final recolored images in (h).





Figure 6: Left: an input image and a color theme. Middle: image generated by the greedy initial assignment in section 5.1, without furtheroptimization. Right: final image with further optimization.

Figure 7: Left: an input image and a target color theme. Middle: result image without E3. Right: result image with E3.

Examples of Harmonization Using Color Themes





Figure 12: A variety of images and their recolored ones with different color theme enhancements. The first column shows the original images.




Chosen Works :: #14 :: Liu et al., 2010

•••••••••••••••••••••••••••••••••••••

Optimizing Photo Composition

Ligang Liu1† Renjie Chen1 Lior Wolf2 Daniel Cohen-Or2

1Zhejiang University, China2Tel-Aviv University, Israel

(a) (b) (c) (d)

0 RT DA VB SZ

1

0.2

0.4

0.6

0.8

Sum

(a)

(b)

(c)

Figure 1: Optimizing the aesthetics of the original photograph in (a) by our approach leads to the new image composition

shown in (c). (b) shows the cropping result of the approach of [Santella et al. 2006]. The aesthetic scores are shown in (d).

Our result in (c) obtains higher aesthetic score than (a). RT(rule of thirds), DA(diagonal), VB(visual balance), and SZ(region

size) are components of the objective function.

Abstract

Aesthetic images evoke an emotional response that transcends mere visual appreciation. In this work we develop

a novel computational means for evaluating the composition aesthetics of a given image based on measuring

several well-grounded composition guidelines. A compound operator of crop-and-retarget is employed to change

the relative position of salient regions in the image and thus to modify the composition aesthetics of the image. We

propose an optimization method for automatically producing a maximally-aesthetic version of the input image. We

validate the performance of the method and show its effectiveness in a variety of experiments.

Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Picture/Image




Chosen Works :: #14 :: Liu et al., 2010

(a) (b) (c) (d)

0 RT DA VB SZ

1

0.2

0.4

0.6

0.8

Sum

(a)

(b)

(c)

Figure 1: Optimizing the aesthetics of the original photograph in (a) by our approach leads to the new image composition

shown in (c). (b) shows the cropping result of the approach of [Santella et al. 2006]. The aesthetic scores are shown in (d).

Our result in (c) obtains higher aesthetic score than (a). RT(rule of thirds), DA(diagonal), VB(visual balance), and SZ(region

size) are components of the objective function.

(a) (b) (c) (d)

0 RT DA VB SZ

1

0.2

0.4

0.6

0.8

Sum

(a)

(b)

(c)

(a)

(b)

(c)

RT DA VB SZ Sum 0

1

0.2

0.4

0.6

0.8

Figure 7: Results of aesthetic composition. (a) The original images; (b) an arbitrary cropping frame of (a); (c) the aesthetic

composition result by our approach; (d) the aesthetic scores of (a),(b),and (c).




Chosen Works :: #15 :: Díaz et al., 2010

•••••••••••••••••••••••••••••••••••••

Cost-effective Feature Enhancement for Volume Datasets

J. Díaz1, J. Marco2 and P. Vázquez1

1MOVING Group, Universitat Politècnica de Catalunya2GESSI Group, Universitat Politècnica de Catalunya

Abstract

Volume models often show high complexity. Local details and overall shape may sometimes be difficult to perceive.

Unsharp masking techniques improve the perception of those small features by increasing the local contrast. In

this paper we present a simple and fast method for feature enhancement based on 3D mipmaps. In contrast to other

approaches, in addition to increasing luminance on the feature details, we also darken the valleys of the volume

thus increasing local contrast and making neighboring details more visible. Our approach is fast and simple, with

small memory requirements thanks to the use of 3D mipmaps. We also propose a color selection strategy, based

on harmonic colors, that further enhances the salient features without abrupt or uncomfortable color changes.

Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Picture/Image

Generation–Display Algorithms

1. Introduction

Contrast enhancement is a popular 2D image processing tool

that improves the image appearance. Concretely, it helps

users to understand complex models by emphasizing small

features. Several experiments have proven that users pre-

• The method is simple and fast, thus maintaining high

framerates for complex models.

• The memory consumption is limited: Auxiliary data struc-

tures consist of a set of 3D mipmaps of the volume

dataset, leading to small memory requirements as com-

pared with the original model.




Chosen Works :: #15 :: Díaz et al., 2010

(a) Ray Casting (b) Protrusion enhancement (c) Sinkage darkening (d) Enhanced view

Mipmap 2, γ = 1.0 Mipmap 2, γ = 1.0

Figure 1: Feature enhancement is achieved by combining a luminance increase for the protruding zones (image c) and darken-

ing for the sinking regions (image d). The final combination is shown in Figure b.

(a) Ray-Casting (b) Unsharped

MM8, γ = 0.4

Figure 4: Feature enhancement of a volumetric model.

Ray-Casting Feature emphasis

with harmonic color

Figure 10: Harmonic color-based feature emphasis for the

pollen grain model.




Chosen Works :: #16 :: Zhang et al., 2011

•••••••••••••••••••••••••••••••••••••

Online Video Stream Stylization

Figure 1: Stylization example. 1: Original video frame; 2 - 4: Three frames of the stylized video with color scheme replacement.

Abstract1

This paper gives an automatic method for online video stream styl-2

ization, producing a temporally coherent output video stream. Our3

system transforms video into an abstract style with large regions of4

constant color and highlighted bold edges. Our system includes two5

novel components. Firstly, to provide coherent and simplified out-6

put, we segment frames, and use optical flow to propagate segmen-7

tation information from frame to frame; an error control strategy8

is used to help ensure that the propagated information is reliable.9

Secondly, to achieve coherent and attractive coloring of the output,10

we use a color scheme replacement algorithm specifically designed11

for an online video stream. We demonstrate real-time performance,12

allowing our approach to be used for live communication, video13

games, and related applications.14

1 Introduction15

Video stream processing has many applications to areas such as live

with user-guided colors, and high contrast. In comparison, Win-43

nemoller [2006] produces a different artistic style based on simpli-44

fied but smoothly shaded contents. Our cartoon-like style means45

that temporal coherence requirements are particularly strict.46

Current stylization methods fall into three categories, each hav-47

ing limitations. Some methods focus on image processing and do48

not readily generalize to video. Others use simple image filters to49

achieve real-time performance, producing simplified and smoothly50

shaded contents, but the output may lack temporal coherence if in-51

put video streams are of low-quality. Yet others require significant52

user interaction to produce high-quality artistic results, and have a53

high computational cost.54

We present a real-time system for a particular style of video styliza-55

tion while providing good coherence. Our approach benefits from56

two novel aspects:57

• a segmentation strategy which uses optical flow to propagate58

segmentation information from frame to frame, with an error59

control strategy to help ensure that the propagated information





Input Video

Edge Detection

Optical Flow

Computation

Coherent Image Segmentation

Color Scheme Replacement

Output Video

Importance MapBoundary Smoothing

& Edge Overlaying

Figure 2: Pipeline. Frame segmentation uses optical flow and edges as inputs, tuned by an importance map, to produce coherent regions.The color scheme replacement step transforms region colors according to a reference video.

Stages of the Method: Harmonization is implemented on the penultimatestage to preserve the coherence between regions





Figure 7: Video stylization results. Columns 1, 3: original video frames; columns 2, 4: corresponding stylization results.

Figure 9: Comparison of stylization effects. From left to right: source image, results using Winnemoller’s method, results of color quantiza-tion, results using our method.




Chosen Works :: #17 :: Haber et al., 2011

•••••••••••••••••••••••••••••••••••••

Computational Aesthetics in Graphics, Visualization, and Imaging (2011)

D. Cunningham and T. Isenberg (Editors)

ColourVis:

Exploring Colour Usage in Paintings Over Time

Jonathan Haber, Sean Lynch, Sheelagh Carpendale

{jmhaber, sglynch, sheelagh} @ucalgary.ca

Department of Computer Science, University of Calgary, Calgary, Alberta, Canada

Figure 1: ColourVis representation of paintings by Vincent van Gogh.

Abstract

The colour palette of painters over history has been of interest to many, including: art historians, archeologists,




Chosen Works :: #17 :: Haber et al., 2011

Figure 11: Picasso paintings visualized using ColourVis.

Figure 12: Representation of 30 years of paintings from the

cubism movement.

Figure 13: ColourVis representation of Realism and Surre-

alism painting movements.

Figure 14: Comparison of the Rembrandt Harmenszoon van

Rijn / Vincent van Gogh / Pablo Picasso painting collections.





•••••••••••••••••••••••••••••••••••••

Example-Based Image Color and Tone Style Enhancement

Baoyuan Wang∗∗ Yizhou Yu†‡ Ying-Qing Xu§

∗Zhejiang University †University of Illinois at Urbana-Champaign‡The University of Hong Kong §Microsoft Research Asia

(a) (b) (c) (d)Figure 1: Style enhancement results. (a) Original photo taken by iPhone 3G, (b) enhanced photo that mimics the color and tone style ofCanon EOS 5D MarkΠ; (c) Original photo, (d) enhanced photo with a style learned from a photographer.

Abstract

Color and tone adjustments are among the most frequent image en-hancement operations. We define a color and tone style as a setof explicit or implicit rules governing color and tone adjustments.Our goal in this paper is to learn implicit color and tone adjust-ment rules from examples. That is, given a set of examples, eachof which is a pair of corresponding images before and after adjust-ments, we would like to discover the underlying mathematical rela-tionships optimally connecting the color and tone of correspondingpixels in all image pairs. We formally define tone and color adjust-ment rules as mappings, and propose to approximate complicatedspatially varying nonlinear mappings in a piecewise manner. Thereason behind this is that a very complicated mapping can still be

ferent photographs. As another example, it is well-known that pho-tographs taken by different digital cameras have varying degrees oftone and color discrepancies. This is because each type of camerahas its own built-in radiance and color response curves. We definea tone and color style as a set of explicit or implicit rules or curvesgoverning tonal and color adjustments.

Manually adjusting the tone and color of a photograph to achievea desired style is often tedious and labor-intensive. However, iftone and color styles can be formulated mathematically in a digitalform, they can be automatically and easily applied to novel inputimages to make them look more appealing. Unfortunately, the rulesgoverning a tone and color style are most often not explicitly avail-able. For instance, it is typically very hard for a photographer tomathematically summarize the rules he uses to achieve a certain





(a) (b) (c) (d)Figure 1: Style enhancement results. (a) Original photo taken by iPhone 3G, (b) enhanced photo that mimics the color and tone style ofCanon EOS 5D MarkΠ; (c) Original photo, (d) enhanced photo with a style learned from a photographer.

Figure 4: (A) is an original photo taken by iPhone 3G while (D) is the reference photo taken by Canon 5D MarkΠ. (B) is the result from anaffine color mapping while (C) is the result from a quadratic color mapping. Between (B) and (C), (C) is overall visually closer to (D).

The technique has two stages: Training and Style Enhancement Stage

Uses a SVM in the first stage for the Color and Gradient Mappings





ImageRegistration

Feature Extraction

Image Pairs

Hierarchical FeatureSub-space Division

Train Gradient Mappings

Train Color Mappings

Features along edges

Hierarchical Feature Sub-space DivisionFeatures away

from edges

Train Binary ClassifierNon-leaf Nodes

Leaf Nodes

Leaf Nodes

...

......

A Gradient MappingTree: Ttree

A Color MappingTree: Ctree

Figure 2: The training stage.Leaf nodes are colored in green while intermediate nodes are colored in yellow.

Input Image

GradientMappings

LocateSubspaces

In TtreeSoft

Segmentation

Color Mappings

OutputImage

ColorSoft Blending

Tone Optimization

LocateSubspaces

In Ctree

Edge Pixels

All Pixels

Figure 3: Image enhancement pipeline.

Two Stages of the Technique: Training and Style Enhancement




Chosen Works :: #19 :: O’Donovan et al., 2012

•••••••••••••••••••••••••••••••••••••

Color Compatibility From Large Datasets

Peter O’Donovan

University of Toronto

Aseem Agarwala

Adobe Systems, Inc.

Aaron Hertzmann

University of Toronto

Abstract

This paper studies color compatibility theories using large datasets,and develops new tools for choosing colors. There are three partsto this work. First, using on-line datasets, we test new and existingtheories of human color preferences. For example, we test whethercertain hues or hue templates may be preferred by viewers. Second,we learn quantitative models that score the quality of a set of fivecolors, called a color theme. Such models can be used to rate thequality of a new color theme. Third, we demonstrate simple proto-types that apply a learned model to tasks in color design, includingimproving existing themes and extracting themes from images.

Links: DL PDF WEB DATA CODE

1 Introduction

Graphic design relies on effective use of color, and choosing col-ors is a difficult but crucial task for both amateur and professionaldesigners. Designers often look for inspiration from many sources,such as art, photography, and color palette books. Color choice isguided largely by intuition and qualitative rules, such as theories ofcomplementary colors and warm versus cool colors. It is generallybelieved that certain color combinations are harmonious and pleas-ing, while others are not. In the past two centuries, many theoriesof color compatibility have been proposed to describe and explainthese phenomena, but there has been little large-scale testing.

On-line communities provide new ways for graphic designers tocreate and share color designs. Two websites, Adobe Kuler andCOLOURLovers, allow users to create color themes, i.e., ordered

Our studies are based on three datasets, each of which comprises acollection of color themes and their ratings. We derived two datasetsfrom Kuler and COLOURLovers, and created the third using Ama-zon Mechanical Turk (“MTurk”). These datasets exhibit differentadvantages and disadvantages. For example, Kuler users have moreexposure to color theory than MTurk workers, while MTurk datais collected in a more controlled fashion. However, taste in colorcan vary widely, and users in these datasets have varying goals,backgrounds, and viewing environments; not surprisingly, there issubstantial variation. Nonetheless, analysis of the data reveals manyregularities and patterns.

We first analyze these datasets to understand which colors peopleuse, and how colors are combined. Our main observations are asfollows. User-created themes are far from random; themes formclusters or manifolds in the space of 5 colors, and themes fartherfrom this manifold tend to be rated worse. People also have strongpreferences for particular colors. The data reveals a preference forwarm hues and cyans in color themes, which is distinct from pref-erences for purples and blues with single colors. Hue templates, themost popular models of color compatibility, are tested in severalways, and no evidence is found that they predict compatible colors.We examine the number of distinct hues people prefer in a theme,and find users generally prefer themes which are neither too sim-ple (i.e., monochromatic), nor too complex (more than 2-3 differenthues). Further MTurk experiments indicate that theme names usu-ally do not affect the rating, though evocative names can have animpact.

We offer a new color compatibility model for predicting ratings,and examine which features of color themes are most important.




Chosen Works :: #19 :: O’Donovan et al., 2012

i V L I

T Y X N

.

i V I

R C N

.

Figure 1: Hue templates implemented in Kuler (left), and thoseproposed by Matsuda [1995] (right). Kuler implements severalcolor selection rules (similar to Matsuda’s i, V, I), as well as others:t(R)iad, (C)ompound. Each theme is described by a color wheel,with gray areas for the hues used by that theme. COLOURLoversimplements the i, V, I, R, Y, X templates (see Supplemental Mate-rial §4). Matsuda uses sectors over the hue wheel, whereas Kulerand COLOURLovers use fixed angle distances, similar to classicaltheory (see Appendix B for discussion).

Figure 2: All-time most popular color themes from Kuler, as ofJanuary 7, 2011. Each color theme consists of five colors. The in-terface shown here also provides some statistics about the ratings,as well as commentary from other users.

This paper studies color compatibility theories using large datasets

First, using on-line datasets, they test new and existing theories of human color preferences.Second, learn quantitative models that score the quality of a set of five colors, called a color

theme. Third, apply a learned model to tasks in color design.




Chosen Works :: #20 :: Guo et al., 2012

•••••••••••••••••••••••••••••••••••••

Improving Photo Composition Elegantly:

Considering Image Similarity During Composition

Optimization

Y. W. Guo1 †, M. Liu1, T. T. Gu1, and W. P. Wang2

1National Key Lab for Novel Software Technology, Nanjing University2Department of Computer Science, The University of Hong Kong

Abstract

Optimization of images with bad compositions has attracted increasing attention in recent years. Previous methods

however seldomly consider image similarity when improving composition aesthetics. This may lead to significant

content changes or bring large distortions, resulting in an unpleasant user experience. In this paper, we present

a new algorithm for improving image composition aesthetics, while retaining faithful, as much as possible, to the

original image content. Our method computes an improved image using a unified model of composition aesthetics

and image similarity. The term of composition aesthetics obeys the rule of thirds and aims to enhance image

composition. The similarity term in contrast penalizes image difference and distortion caused by composition

adjustment. We use an edge-based measure of structure similarity which nearly coincides with human visual

perception to compare the optimized image with the original one. We describe an effective scheme to generate

the optimized image with the objective model. Our algorithm is able to produce the recomposed images with

minimal visual distortions in an elegant and user controllable manner. We show the superiority of our algorithm

by comparing our results with those by previous methods.

Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Picture/ImageGeneration—Display algorithms





(a) (b) (c)

Figure 1: The image (b) with improved composition generated by our method for an input image (a). (c) shows the result of Liu et al.

[LCWCO10]. The result exhibits obvious distortion in the cloud region. Moreover, the scale operation they adopt may cause blurring artifacts,

see the girl sitting in the wheelchair.

(a) The input (b) Result in [BSS10] (c) Our result

Figure 6: Comparison with the method in [BSS10]. (a) The input image in [BSS10]. (b) The result produced by the method of [BSS10]. Note

that, the building is originally located on the grassland in the distance. However, its location is changed to the yellow land near the viewpoint

in the result of [BSS10]. This in fact changes image semantics. (c) Our result.





(a) (b) (c) (d)

Figure 7: More results. (a), (c) The input images. (b), (d) Our results.

This approach uses the image quality assessment index SSIM

The Structural Similarity (SSIM) Index is used during compositionoptimization proccess




Chosen Works :: #21 :: Tang et al., 2012

•••••••••••••••••••••••••••••••••••••

IMAGE COMPOSITION WITH COLOR HARMONIZATION

YANLI WAN*, ZHEN TANG†, ZHENJIANG MIAO‡ and BO LI§

Institute of Information Science, Beijing Jiaotong University

Beijing, 100044, P. R. China*[email protected]†[email protected]‡[email protected]

§[email protected]

Received 22 November 2010Accepted 9 February 2012

Published 18 August 2012

Image composition is a very important technique in computer generated imagery. Besides somefactors such as contrast, texture and noise that a®ect the quality of the composition, color

harmony between fore- and background is also an important factor that would a®ect the quality

of the composition. However, in the previous image composition techniques, color harmony

between fore- and background is seldom considered. In this paper, an optimization method isproposed to deal with the color harmonization problem that used in image composition. A cost

function is derived from the local smoothness of the hue values, and the image is harmonized by

minimizing the cost function. A new matching cost function is proposed to select the bestmatching harmonic schemes. Our approach overcomes several shortcomings of the existing color

harmonization methods. We validate the performance of our method and demonstrate its

e®ectiveness with a variety of experiments.

Keywords : Color harmonization; image composition; harmonic template; optimization;

pre-harmonization.

1. Introduction

Much of what we perceive and feel about an image is experienced through colors.

Although our perception of colors depends on the context, and is culture-related, it is

impossible to look at an image without being a®ected by the harmony of its colors.3

Harmony among colors is greatly related to the relative positions in color space, and

is represented by a set of harmonic templates11,18 as shown in Fig. 1(a). These

templates are de¯ned on the wheel of hue channel of the HSV color space. If a set of

colors falls inside the gray sectors, it will be considered to be harmonic. The N-type

template which corresponds to the gray-scale images is not used in this work. Note

that, Fig. 1(a) only shows one example template for each type of harmonic template.

A series of harmonic templates for each type template can be generated by rotating

International Journal of Pattern Recognitionand Arti¯cial Intelligence

Vol. 26, No. 3 (2012) 1254001 (23 pages)

#.c World Scienti¯c Publishing Company

DOI: 10.1142/S0218001412540018

1254001-1

Int.

J. P

att.

Rec

ogn.

Art

if. I

ntel

l. 20

12.2

6. D

ownl

oade

d fr

om w

ww

.wor

ldsc

ient

ific

.com

by U

NIV

ER

SID

AD

E F

ED

ER

AL

DE

AL

AG

OA

S on

10/

26/1

2. F

or p

erso

nal u

se o

nly.





where # mean \number of results". The comparison result is shown in Fig. 14. It can

be seen that, our algorithm can preserve the coherence much better than Refs. 3

and 14.

4.3. More composition results with color harmonization

In image composition, whether or not the color harmonization process is needed

depends on the image content, the user and the applications. For some cases, the

color of the fore- and background should be adjusted because the composite image

disobeys the law of nature. For example, when we synthesize a foreground object

captured from a spring scene with a background scene describing the autumn, the

foreground color should be adjusted as shown in Fig. 15(c). Figure 15(d) shows

another case of adjusting the autumn foreground object that will be pasted into the

spring scene. For most of the cases, the user will determine whether or not the color

0 10 20 30 40 50 60 70 80 90 100

0.05

0.1

0.15

0.2

Image no.

ER

CohenOr et al. Sawant N. et al. Our method.

Fig. 14. Error rate comparison of our algorithm with the previous two algorithms.3,14

(a)

(b)

Fig. 15. Some other examples of image composition with color harmonization. From left to right of each

set are the original foreground image, background image, the image composition without color harmoni-

zation and image composition with color harmonization.

Y. Wan et al.

1254001-20

Int.

J. P

att.

Rec

ogn.

Art

if. I

ntel

l. 20

12.2

6. D

ownl

oade

d fr

om w

ww

.wor

ldsc

ient

ific

.com

by U

NIV

ER

SID

AD

E F

ED

ER

AL

DE

AL

AG

OA

S on

10/

26/1

2. F

or p

erso

nal u

se o

nly.

Comparisons Between Algorithms of Cohen-Or et al., Sawant et al., andTang et al.

The proposed algorithm preserves the coherence much better than others!







and 14.










0 10 20 30 40 50 60 70 80 90 100

0.05

0.1

0.15

0.2

Image no.

ER



(a)

(b)




Y. Wan et al.

1254001-20

Int.

J. P

att.

Rec

ogn.

Art

if. I

ntel

l. 20

12.2

6. D

ownl

oade

d fr

om w

ww

.wor

ldsc

ient

ific

.com

by U

NIV

ER

SID

AD

E F

ED

ER

AL

DE

AL

AG

OA

S on

10/

26/1

2. F

or p

erso

nal u

se o

nly.



and 14.










0 10 20 30 40 50 60 70 80 90 100

0.05

0.1

0.15

0.2

Image no.

ER



(a)

(b)




Y. Wan et al.

1254001-20

Int.

J. P

att.

Rec

ogn.

Art

if. I

ntel

l. 20

12.2

6. D

ownl

oade

d fr

om w

ww

.wor

ldsc

ient

ific

.com

by U

NIV

ER

SID

AD

E F

ED

ER

AL

DE

AL

AG

OA

S on

10/

26/1

2. F

or p

erso

nal u

se o

nly.

harmonization process is needed according to their applications. Using which type of

harmonic templates to harmonize the image is greatly related to the user's culture,

knowledge and experiences. Figures 15(a) and 15(b) are two examples of changing

the cold or warm foreground colors. Here we only show one harmonization result for

each example. More results can be obtained by using other harmonic template and

rotating the sectors by di®erent angles.

5. Conclusion

A new optimization-based color harmonization algorithm is proposed to harmonize

the composite images. It is based on the observation that pixels in a small window

which have similar inharmonic hues with respect to a certain harmonic scheme

should be harmonized by similar harmonic hues. This observation is formulated into

a quadratic function, and the desired hues are solved by minimizing this cost func-

tion. A new template matching cost function which employs relative distances is also

presented. It outperforms the previous matching cost functions which are based on

the absolute distances. A component-based pre-harmonic scheme is designed to

preserve the hue distribution of the harmonized images. The experiments demon-

strate the e®ectiveness of our algorithm.

Acknowledgments

This work is supported by the National Natural Science Foundation of China

(60973061), National 973 Key Research Program of China (2011CB302203), Ph.D.

Programs Foundation of Ministry of Education of China (20100009110004) and the

Technological Innovation Fund of Excellent Doctoral Candidate of Beijing Jiaotong

University (141047522).

(c)

(d)

Fig. 15. (Continued)

Image Composition with Color Harmonization

1254001-21

Int.

J. P

att.

Rec

ogn.

Art

if. I

ntel

l. 20

12.2

6. D

ownl

oade

d fr

om w

ww

.wor

ldsc

ient

ific

.com

by U

NIV

ER

SID

AD

E F

ED

ER

AL

DE

AL

AG

OA

S on

10/

26/1

2. F

or p

erso

nal u

se o

nly.

Some Results





•••••••••••••••••••••••••••••••••••••

Affective Image Colorization

Xiao-Hui Wang1 (迄裡閤), Jia Jia1 (細羞), Han-Yu Liao2 (壬厳匸), and Lian-Hong Cai1 (囲信醐)

1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China2Academy of Art & Design, Tsinghua University, Beijing 100084, China

E-mail: [email protected]; [email protected]; [email protected]; [email protected]

Received September 5, 2012; revised September 17, 2012.

Abstract Colorization of gray-scale images has attracted many attentions for a long time. An important role of imagecolor is the conveyer of emotions (through color themes). The colorization with an undesired color theme is less useful, evenit is semantically correct. However this has been rarely considered. Automatic colorization respecting both the semanticsand the emotions is undoubtedly a challenge. In this paper, we propose a complete system for affective image colorization.We only need the user to assist object segmentation along with text labels and an affective word. First, the text labels alongwith other object characters are jointly used to filter the internet images to give each object a set of semantically correctreference images. Second, we select a set of color themes according to the affective word based on art theories. With thesethemes, a generic algorithm is used to select the best reference for each object, balancing various requirements. Finally, wepropose a hybrid texture synthesis approach for colorization. To the best of our knowledge, it is the first system which isable to efficiently colorize a gray-scale image semantically by an emotionally controllable fashion. Our experiments show theeffectiveness of our system, especially the benefit compared with the previous Markov random field (MRF) based method.

Keywords image colorization, affective word, color theme

1 Introduction

Color can enhance the expressiveness of an image. Awonderful colorization not only gives a gray-scale imagegood visual sense, but also endows it with much richer

demonstrated by various psychological studies[7-9]. Im-ages with the same content but different color themesmay have totally different emotions. So a proper col-orization can also give images much richer emotions.And the colorization with an undesired color theme

©





Fig.2. Pipeline[10]. (a) Input. (b) Object filtering. (c) Color theme selection. (d) Generic algorithm based reference selection. (e)

Image colorization.

The Overall Pipeline of the System

Uses a Color Theme Database Properly Harmonized





Colorization results. (a) Input images and labels. (b) Affective words andcolor themes. (c) Colorization results. (d) Reference images.




Chosen Works :: #23 :: Xue et al., 2012

•••••••••••••••••••••••••••••••••••••

Understanding and Improving the Realism of Image Composites

Su Xue

Yale University

Aseem Agarwala

Adobe Systems, Inc.

Julie Dorsey

Yale University

Holly Rushmeier

Yale University

Figure 1: Our method can automatically adjust the appearance of a foreground region to better match the background of a composite. Giventhe proposed foreground and background on the left, we show the compositing results of unadjusted cut-and-paste, Adobe Photoshop’s MatchColor, the method of Lalonde and Efros[2007], and our method.

Abstract

Compositing is one of the most commonly performed operationsin computer graphics. A realistic composite requires adjusting theappearance of the foreground and background so that they appearcompatible; unfortunately, this task is challenging and poorly un-derstood. We use statistical and visual perception experiments tostudy the realism of image composites. First, we evaluate a num-ber of standard 2D image statistical measures, and identify thosethat are most significant in determining the realism of a compos-ite. Then, we perform a human subjects experiment to determinehow the changes in these key statistics influence human judgementsof composite realism. Finally, we describe a data-driven algorithmthat automatically adjusts these statistical measures in a foregroundto make it more compatible with its background in a composite.We show a number of compositing results, and evaluate the per-formance of both our algorithm and previous work with a human

in the research community [Smith and Blinn 1996; Wang and Co-hen 2007; Rhemann et al. 2009], the second has not been systemati-cally studied. Professional compositors have several rules of thumb,but in the end, most composites are made realistic by trial-and-erroradjustment of standard image controls such as brightness, contrast,hue, and saturation.

In this paper, we use statistical and visual perception experimentsto study the factors that influence the realism of image composites,and propose an automated method to increase their realism. Thescope of this problem is large: composite realism is influenced bysemantics (e.g., is a polar bear in a rainforest realistic), and fac-tors that require 3D reasoning to analyze (e.g., inter-reflections).However, perception research has shown that the human visual sys-tem is remarkably insensitive to certain lighting inconsistencieswithin an image, such as shadow directions and highlight place-ments [Ostrovsky et al. 2005; Lopez-Moreno et al. 2010]. If a user





Fg

Bg

Figure 7: Top: results of matching highlight, mid-tone, and shadowzones of the luminance histogram for a composite. Bottom: illustra-tion of matching the corresponding zones of the histograms.

Figure 9: The sequence of adjustments for a particular composite.

Adjustments in Local Contrast, Luminance, Correlated Color Temperature(CCT), and Saturation.

The rate of realism is a major concern in this method!





Figure 1: Our method can automatically adjust the appearance of a foreground region to better match the background of a composite. Giventhe proposed foreground and background on the left, we show the compositing results of unadjusted cut-and-paste, Adobe Photoshop’s MatchColor, the method of Lalonde and Efros[2007], and our method.

Automatically Adjust the Appearance of a Foreground Region




Chosen Works :: #24 :: Bayeve & Fargeas, 2012

•••••••••••••••••••••••••••••••••••••

COLOR HARMONIZATION ENHANCEMENT

Yoann BAVEYE, Aureline FARGEAS

ESIR, Universite de Rennes 1

ABSTRACT

The goal of color harmonization is to have an image harmo-

nized in terms of color or hue. It is a concept of art and design

or more specifically of human visual perception which can be

usefull adapted to image processing. In this paper, we explain

how we try to harmonize an image. We want a final image as

much as realistic compare to the original image. For instance,

the color of the skin would probably not be modified. Given a

color image, proposed method finds the best harmony for the

image colors. Results show that the proposed method harmo-

nizes consistently and efficiently any image and returns the

faithfull color image corresponding. Moreover, the harmo-

nization method can be applied locally in order to be con-

sistent with a given part of the picture. The background of

a picture can be harmonized according to the colors of the

foreground (for example, the clothes of a girl). In this exam-

ple, the colors of the girl would not be modified but buildings

in the background would be harmonized with respect to the

foreground.

designer into the scene, the proposed method is able to har-

monized this object with repect to the scene. Most of all,

professional web designers experienced in color theory can

create color palettes that evoke different moods, appropriate

to the product or service being sold.

The paper is organized as follows: In Section 2, we dis-

cuss image color harmonization. In Section 3, we propose

improvements over existing method. Section 4 presents an

optimization to remove artifacts. We present various results

in Section 5 and conclude this paper in section 6.

2. STATE OF THE ART

Generating harmonic colors has been an open problem among

artists and scientists [2]. Modern color theory which was de-

veloped at the beginning of the 20th century, deals mainly

with representations of colors, but it also discusses color har-

mony [3, 4, 5]. It is only in 1960 that scientists began to use a

new kind of color wheel with an emphasis on hue [3]. One of





Fig. 3. Artifacts caused by the fact that hues which are nearly

equally close to both sectors of the template may “choose”

their sector arbitrarily and this causes color discontinuities in

the resulting image.

creases computational time.

Fig. 4. Mean Shift segmentation

Adoption of a New Cost Function

Use of Mean Shift Segmentation





Original pictureHarmonized picture

of [1]Our harmonized

result

Comparison of Harmonized Results: Cohen-Or et al. X Bayeve




Chosen Works :: #25 :: Bayeve et al., 2013

•••••••••••••••••••••••••••••••••••••

Saliency-Guided Consistent Color

Harmonization

Yoann Baveye, Fabrice Urban, Christel Chamaret, Vincent Demoulin,and Pierre Hellier

Technicolor Research and Innovation, Rennes, France{baveyey,urbanf,chamaretc,demoulinv,hellierp}@technicolor.com

https://research.technicolor.com/rennes/

Abstract. The focus of this paper is automatic color harmonization,which amounts to re-coloring an image so that the obtained color paletteis more harmonious for human observers. The proposed automatic algo-rithm builds on the pioneering works described in [3,12] where templatesof harmonious colors are defined on the hue wheel. We bring three con-tributions in this paper: first, saliency [9] is used to predict the mostattractive visual areas and estimate a consistent harmonious template.Second, an efficient color segmentation algorithm, adapted from [4], isproposed to perform consistent color mapping. Third, a new mappingfunction substitutes usual color shifting method. Results show that themethod limits the visual artifacts of state-of-the-art methods and leads





Fig. 2. Harmonious templates Tm on the hue wheel for a given angle [3]. The completecollection of harmonious templates is obtained by rotating all templates. For a giventemplate at a given angle, grey areas enclose hues that form a harmonious set. In otherwords, an image where the histogram is strictly enclosed in the grey area is consideredas harmonious.

Use of new templates!





!"#$%&'((!)*

+!,-'$%.//#)/!0)102#$

30$045#*&#)/'/!0)

64!*!)'$%(!7/-4#

8#&($'/#2#/#4&!)'/!0)

9'4&0)!:#2%(!7/-4#

51 8&

31

Fig. 3. Overview of the proposed algorithm. The first step (top part) consists in esti-mating the best harmonic template. Based on a measure of the salient areas, a templatedetermination is performed so as to minimize a statistical distance between the his-togram and a harmonious template. Prior to the color mapping, a color segmentationtechnique is performed to ensure that consistent colors are finally mapped into thesame harmonious template sector.

Visual Attention Model with Saliency Map (SM)





! " # $ % & ' ()

*)

+)

,)

-)

.)

/)

1)

234567(89:5;<&=:5 =6>9:5;<&=:5?83<3@5A9:5;<&=:5

!"#$%&!'()*+,-.!

/,2)

3"%4

!(

Fig. 5. Distribution of template selection for a database of 154 images regarding theCohen-Or [3], Tang [21] and our proposed energy computation. The Cohen-or techniquetends to select the T and X templates.

Comparisons Between Choice of Templates





OriginalCohen-Ortemplate

Tang template Proposed template

Huehistogram

weightedHue

histogram

Huehistogram

Huehistogram

Huehistogram

weightedHue

histogram

Fig. 6. Influence of distribution similarity computation regarding template choice usingthe proposed color mapping function

Influence of Distribution Similarity




Conclusions

Conclusions in Respect of Papers Presented

The methods presented in optimize the

harmony of a given image

Operate by displacing hue values for a given harmonicdistribution, using various methods (Tokumaru

Templates, Gaussian mixture model, etc.)

Especially useful for artificial colors in composition,combination and restoration of images.




Demonstrative Videos

Demonstrative Videos of Some Techniques Discussed

PLAY FULLSCREEN

VIEW INSIDE

PLAY FULLSCREEN

VIEW INSIDE




One Last Question...

Will we be able?

...to capture and describe the creative process and the sensitivity ofanalytical way through procedures reproducible by machines?




Thanks

Thanks for your attention!

Michel Alves dos Santos - [email protected]

Michel Alves dos Santos - (Alves, M.)MSc Candidate at Federal University of Rio de Janeiro.

E-mail: [email protected], [email protected]: http://lattes.cnpq.br/7295977425362370Home: http://www.michelalves.comPhone: +55 21 2562 8572 (Institutional Phone Number)

http://www.facebook.com/michel.alves.santos

http://www.linkedin.com/profile/view?id=26542507


http://www.ufrj.br

mailto:[email protected]

mailto:[email protected]

http://lattes.cnpq.br/7295977425362370

http://www.michelalves.com



References I

Y. Baveye, F. Urban, C. Chamaret, V. Demoulin, and P. Hellier.Saliency-guided consistent color harmonization.In Proceedings of the 4th International Conference on Computational Color Imaging,CCIW’13, pages 105–118, Berlin, Heidelberg, 2013. Springer-Verlag.ISBN 978-3-642-36699-4.URL http://dx.doi.org/10.1007/978-3-642-36700-7_9.

V. Bochko and J. Parkkinen.A spectral color analysis and colorization technique.Computer Graphics and Applications, IEEE, 26(5):74–82, 2006.

Y. Chang, S. Saito, K. Uchikawa, and M. Nakajima.Example-based color stylization of images.ACM Transactions on Applied Perception, 2(3):322–345, 2006.

D. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand, and Y.-Q. Xu.Color harmonization.ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 25(3):624–630, 2006.

R. Datta, D. Joshi, J. Li, and J. Z. Wang.Studying aesthetics in photographic images using a computational approach.In Computer Vision–ECCV 2006, pages 288–301. Springer, 2006.


http://dx.doi.org/10.1007/978-3-642-36700-7_9



References II

A. A. Gooch, S. C. Olsen, J. Tumblin, and B. Gooch.Color2gray: salience-preserving color removal.ACM Trans. Graph., 24(3):634–639, jul 2005.ISSN 0730-0301.URL http://doi.acm.org/10.1145/1073204.1073241.

W. C. Granville.Color harmony: What is it?Color Research & Application, 12(4):196–201, 1987.ISSN 1520-6378.URL http://dx.doi.org/10.1002/col.5080120407.

W. C. Granville and E. Jacobson.Colorimetric specification of the color harmony manual from spectrophotometricmeasurements.J. Opt. Soc. Am., 34(7):382–393, Jul 1944.URL http://www.opticsinfobase.org/abstract.cfm?URI=josa-34-7-382.

L. Gruber, D. Kalkofen, and D. Schmalstieg.Color harmonization for augmented reality.In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on,pages 227–228, 2010.


http://doi.acm.org/10.1145/1073204.1073241

http://dx.doi.org/10.1002/col.5080120407

http://www.opticsinfobase.org/abstract.cfm?URI=josa-34-7-382



References III

Y. W. Guo, M. Liu, T. T. Gu, and W. P. Wang.Improving photo composition elegantly: Considering image similarity during compositionoptimization.Comp. Graph. Forum, 31(7pt2):2193–2202, Sept. 2012.ISSN 0167-7055.URL http://dx.doi.org/10.1111/j.1467-8659.2012.03212.x.

J. Haber, S. Lynch, and S. Carpendale.Colourvis: exploring colour usage in paintings over time.In Proceedings of the International Symposium on Computational Aesthetics in Graphics,Visualization, and Imaging, CAe ’11, pages 105–112, New York, NY, USA, 2011. ACM.ISBN 978-1-4503-0908-0.URL http://doi.acm.org/10.1145/2030441.2030464.

M. Hascoet.Visual color design.In Information Visualisation (IV), 2012 16th International Conference on, pages 62–67.IEEE, 2012.

L. Holtzschue.Understanding Color.John Wiley & Sons. Inc., New York, 2002.


http://dx.doi.org/10.1111/j.1467-8659.2012.03212.x

http://doi.acm.org/10.1145/2030441.2030464



References IV

X. Hou and L. Zhang.Color conceptualization.In Proceedings of the 15th international conference on Multimedia, MULTIMEDIA ’07,pages 265–268, New York, NY, USA, 2007. ACM.ISBN 978-1-59593-702-5.URL http://doi.acm.org/10.1145/1291233.1291288.

H. Huang, Y. Zang, and C.-F. Li.Example-based painting guided by color features.Vis. Comput., 26(6-8):933–942, June 2010a.URL http://dx.doi.org/10.1007/s00371-010-0498-y.

H. Huang, Y. Zang, and C.-F. Li.Example-based painting guided by color features.The Visual Computer, 26(6-8):933–942, 2010b.

X. Huo and J. Tan.An improved method for color harmonization.In Image and Signal Processing, 2009. CISP ’09. 2nd International Congress on, pages 1–4,2009.

R. Irony, D. Cohen-Or, and D. Lischinski.Colorization by example.In Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, pages201–210. Eurographics Association, 2005.


http://doi.acm.org/10.1145/1291233.1291288

http://dx.doi.org/10.1007/s00371-010-0498-y



References V

J. Itten.The Art of Color.New York: Van Nostrand Reinhold Company., 1960.

D. J., M. J., and V. P.Cost-effective feature enhancement for volume datasets.In 15th International Workshop on Vision, Modeling and Visualization, pages 187–194, 2010.

K. L. Kelly and D. B. Judd.Color: universal language and dictionary of names, volume 440.US Department of Commerce, National Bureau of Standards, 1976.

J. Krause.Color Index: Over 1,000 Color Combinations, CMYK and RGB Formulas, for Print and WebMedia.F & W Publications, Incorporated, 2002.ISBN 1581802366.

J.-F. Lalonde and A. A. Efros.Using color compatibility for assessing image realism.In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pages 1–8.IEEE, 2007.




References VI

A. Levin, D. Lischinski, and Y. Weiss.Colorization using optimization.ACM Trans. Graph., 23(3):689–694, aug 2004.ISSN 0730-0301.URL http://doi.acm.org/10.1145/1015706.1015780.

C. Li and T. Chen.Aesthetic visual quality assessment of paintings.Selected Topics in Signal Processing, IEEE Journal of, 3(2):236–252, 2009.

A. Lindner, A. Shaji, N. Bonnier, and S. Süsstrunk.Joint statistical analysis of images and keywords with applications in semantic imageenhancement.In Proceedings of the 20th ACM international conference on Multimedia, MM ’12, pages489–498, New York, NY, USA, 2012. ACM.ISBN 978-1-4503-1089-5.URL http://doi.acm.org/10.1145/2393347.2393417.

L. Liu, R. Chen, L. Wolf, and D. Cohen-Or.Optimizing photo composition.In Computer Graphics Forum, volume 29, pages 469–478. Wiley Online Library, 2010.


http://doi.acm.org/10.1145/1015706.1015780

http://doi.acm.org/10.1145/2393347.2393417



References VII

Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum.Natural image colorization.In Proceedings of the 18th Eurographics conference on Rendering Techniques, pages309–320. Eurographics Association, 2007.

Y. Luo and X. Tang.Photo and video quality evaluation: Focusing on the subject.In Computer Vision–ECCV 2008, pages 386–399. Springer, 2008a.

Y. Luo and X. Tang.Photo and video quality evaluation: Focusing on the subject.In Computer Vision–ECCV 2008, pages 386–399. Springer, 2008b.

Y. Matsuda.Color Design.Asakura Shoten (in Japanese), Tokio, Japan, 1995.

B. J. Meier.Ace: a color expert system for user interface design.In Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software,UIST ’88, pages 117–128, New York, NY, USA, 1988. ACM.ISBN 0-89791-283-7.URL http://doi.acm.org/10.1145/62402.62424.


http://doi.acm.org/10.1145/62402.62424



References VIII

P. Moon and D. E. Spencer.Geometric formulation of classical color harmony.JOSA, 34(1):46–50, 1944.

J. Morovic and M. R. Luo.The fundamentals of gamut mapping: A survey.pages 283–290, 2001.

B. S. Morse, D. Thornton, Q. Xia, and J. Uibel.Image-based color schemes.In Image Processing, 2007. ICIP 2007. IEEE International Conference on, volume 3, pagesIII–497. IEEE, 2007.

A. H. Munsell.A grammar of color: a basic treatise on the color system.Van Nostrand Reinhold Co. GoetheBook, 1969.

F. Nack, A. Manniesing, and L. Hardman.Colour picking: the pecking prder of form and function.In Proceedings of the eleventh ACM international conference on Multimedia, MULTIMEDIA’03, pages 279–282, New York, NY, USA, 2003. ACM.ISBN 1-58113-722-2.URL http://doi.acm.org/10.1145/957013.957071.


http://doi.acm.org/10.1145/957013.957071



References IX

A. Neumann and L. Neumann.Color style transfer techniques using hue, lightness and saturation histogram matching.In Computational Aesthetics’05, pages 111–122, 2005.

L. Neumann, A. Nemcsics, and A. Neumann.Computational color harmony based on coloroid system.In Proceedings of the First Eurographics conference on Computational Aesthetics inGraphics, Visualization and Imaging, Computational Aesthetics’05, pages 231–240,Aire-la-Ville, Switzerland, Switzerland, 2005. Eurographics Association.ISBN 3-905673-27-4.URL http://dx.doi.org/10.2312/COMPAESTH/COMPAESTH05/231-240.

P. Obrador.Automatic color scheme picker for document templates based on image analysis and dualproblem.In Electronic Imaging 2006, pages 607609–607609. International Society for Optics andPhotonics, 2006.

P. Obrador, M. A. Saad, P. Suryanarayan, and N. Oliver.Towards category-based aesthetic models of photographs.In Proceedings of the 18th international conference on Advances in Multimedia Modeling,MMM’12, pages 63–76, Berlin, Heidelberg, 2012. Springer-Verlag.ISBN 978-3-642-27354-4.URL http://dx.doi.org/10.1007/978-3-642-27355-1_9.


http://dx.doi.org/10.2312/COMPAESTH/COMPAESTH05/231-240

http://dx.doi.org/10.1007/978-3-642-27355-1_9



References X

P. O’Donovan, A. Agarwala, and A. Hertzmann.Color compatibility from large datasets.In ACM SIGGRAPH 2011 papers, SIGGRAPH ’11, pages 63:1–63:12, New York, NY, USA,2011. ACM.ISBN 978-1-4503-0943-1.URL http://doi.acm.org/10.1145/1964921.1964958.

W. Ostwald and F. Birren.The Color Primer.Van Nostrand Reinhold Company, New York, USA, 1969.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery.Numerical Recipes in C: The Art of Scientific Computing.Cambridge University Press, New York, NY, USA, 2nd edition, 1992.ISBN 0521437148.

K. Rasche, R. Geist, and J. Westall.Re-coloring images for gamuts of lower dimension.Computer Graphics Forum, 24(3):423–432, 2005.ISSN 1467-8659.URL http://dx.doi.org/10.1111/j.1467-8659.2005.00867.x.


http://doi.acm.org/10.1145/1964921.1964958

http://dx.doi.org/10.1111/j.1467-8659.2005.00867.x



References XI

E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley.Color transfer between images.IEEE Comput. Graph. Appl., 21(5):34–41, sep 2001.ISSN 0272-1716.URL http://dx.doi.org/10.1109/38.946629.

Y. Sato and J. Tajima.A color scheme supporting method in a color design system.In SPIE, volume 2411, pages 25–33, 1995.

C. Sauvaget and V. Boyer.Harmonic colorization using proportion contrast.In Proceedings of the 7th International Conference on Computer Graphics, Virtual Reality,Visualisation and Interaction in Africa, AFRIGRAPH ’10, pages 63–69, New York, NY, USA,2010a. ACM.ISBN 978-1-4503-0118-3.URL http://doi.acm.org/10.1145/1811158.1811169.

C. Sauvaget and V. Boyer.Harmonic colorization using proportion contrast.In Proceedings of the 7th International Conference on Computer Graphics, Virtual Reality,Visualisation and Interaction in Africa, pages 63–69. ACM, 2010b.


http://dx.doi.org/10.1109/38.946629

http://doi.acm.org/10.1145/1811158.1811169



References XII

N. Sawant and N. Mitra.Color harmonization for videos.In Computer Vision, Graphics Image Processing, 2008. ICVGIP ’08. Sixth Indian Conferenceon, pages 576–582, 2008.

S. Seo, Y. Park, and V. Ostromoukhov.Image recoloring using linear template mapping.Multimedia Tools Appl., 64(2):293–308, May 2013.ISSN 1380-7501.URL http://dx.doi.org/10.1007/s11042-012-1024-1.

L. Shapira, A. Shamir, and D. Cohen-Or.Image appearance exploration by model-based navigation.In Computer Graphics Forum, volume 28, pages 629–638. Wiley Online Library, 2009.

N. Silvestrini and E. P. Fischer.Colorsystem: Colour order systems in art and science., September 2013.URL http://www.colorsystem.com/?lang=en.

K. Suganuma, J. Sugita, and T. Takahashi.Colorization using harmonic templates.In ACM SIGGRAPH 2008 posters, SIGGRAPH ’08, pages 62:1–62:1, New York, NY, USA,2008. ACM.ISBN 978-1-60558-466-9.URL http://doi.acm.org/10.1145/1400885.1400952.


http://dx.doi.org/10.1007/s11042-012-1024-1

http://www.colorsystem.com/?lang=en

http://doi.acm.org/10.1145/1400885.1400952



References XIII

M. Sun, Q. Sun, and X. Xu.Color harmony based on fitting functions.In Information Technology and Applications, 2009. IFITA’09. International Forum on,volume 1, pages 165–167. IEEE, 2009.

K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister.Multi-scale image harmonization.ACM Trans. Graph., 29(4):125:1–125:10, July 2010a.ISSN 0730-0301.URL http://doi.acm.org/10.1145/1778765.1778862.

K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister.Multi-scale image harmonization.ACM Transactions on Graphics (TOG), 29(4):125, 2010b.

G. Tanaka, N. Suetake, and E. Uchino.Color transfer based on normalized cumulative hue histograms.JACIII, 14(2):185–192, 2010.

Z. Tang, Z. Miao, and Y. Wan.Image composition with color harmonization.In Image and Vision Computing New Zealand (IVCNZ), 2010 25th International Conferenceof, pages 1–8, 2010.


http://doi.acm.org/10.1145/1778765.1778862



References XIV

Z. Tang, Z. Miao, Y. Wan, and F. F. Jesse.Colour harmonisation for images and videos via two-level graph cut.Image Processing, IET, 5(7):630–643, 2011a.ISSN 1751-9659.

Z. Tang, Z. Miao, Y. Wan, and Z. Wang.Color harmonization for images.Journal of Electronic Imaging, 20(2):023001–023001, 2011b.

M. Tokumaru, N. Muranaka, and S. Imanishi.Color design support system considering color harmony.In Fuzzy Systems, 2002. FUZZ-IEEE’02. Proceedings of the 2002 IEEE InternationalConference on, volume 1, pages 378–383, 2002.

J. W. von Goethe.Goethe’s Color Theory. Translated by Rupprecht Matthei.Van Nostrand Reinhold Co., New York, USA, 1971.

B. Wang, Y. Yu, T.-T. Wong, C. Chen, and Y.-Q. Xu.Data-driven image color theme enhancement.In ACM SIGGRAPH Asia 2010 papers, SIGGRAPH ASIA ’10, pages 146:1–146:10, NewYork, NY, USA, 2010. ACM.ISBN 978-1-4503-0439-9.URL http://doi.acm.org/10.1145/1866158.1866172.


http://doi.acm.org/10.1145/1866158.1866172



References XV

B. Wang, Y. Yu, and Y.-Q. Xu.Example-based image color and tone style enhancement.ACM Trans. Graph., 30(4):64:1–64:12, July 2011.ISSN 0730-0301.URL http://doi.acm.org/10.1145/2010324.1964959.

C. Wang, R. Zhang, and F. Deng.Image composition with color harmonization.Chinese Optics Letters, 7(6):483–485, 2009.

L. Wang and K. Mueller.Harmonic colormaps for volume visualization.In Proceedings of the Fifth Eurographics / IEEE VGTC conference on Point-Based Graphics,SPBG’08, pages 33–39, Aire-la-Ville, Switzerland, Switzerland, 2008a. EurographicsAssociation.ISBN 978-3-905674-12-5.URL http://dx.doi.org/10.2312/VG/VG-PBG08/033-039.

L. Wang and K. Mueller.Harmonic colormaps for volume visualization.In Proceedings of the Fifth Eurographics/IEEE VGTC conference on Point-Based Graphics,pages 33–39. Eurographics Association, 2008b.


http://doi.acm.org/10.1145/2010324.1964959

http://dx.doi.org/10.2312/VG/VG-PBG08/033-039



References XVI

L. Wang, J. Giesen, K. T. McDonnell, P. Zolliker, and K. Mueller.Color design for illustrative visualization.IEEE Transactions on Visualization and Computer Graphics, 14(6):1739–1754, Nov. 2008.ISSN 1077-2626.URL http://dx.doi.org/10.1109/TVCG.2008.118.

X. Wang, J. Jia, H. Liao, and L. Cai.Affective image colorization.Journal of Computer Science and Technology, 27(6):1119–1128, 2012a.

X. Wang, J. Jia, H. Liao, and L. Cai.Image colorization with an affective word.In Proceedings of the First international conference on Computational Visual Media,CVM’12, pages 51–58, Berlin, Heidelberg, 2012b. Springer-Verlag.ISBN 978-3-642-34262-2.URL http://dx.doi.org/10.1007/978-3-642-34263-9_7.

X. Wang, J. Jia, H. Liao, and L. Cai.Image colorization with an affective word.In Computational Visual Media, pages 51–58. Springer, 2012c.

T. Welsh, M. Ashikhmin, and K. Mueller.Transferring color to greyscale images.ACM Trans. Graph., 21(3):277–280, jul 2002.URL http://doi.acm.org/10.1145/566654.566576.


http://dx.doi.org/10.1109/TVCG.2008.118

http://dx.doi.org/10.1007/978-3-642-34263-9_7

http://doi.acm.org/10.1145/566654.566576



References XVII

S. Westland, K. Laycock, V. Cheung, P. Henry, and F. Mahyar.Colour harmony.JAIC-Journal of the International Colour Association, 1, 2012.

S. Xue, A. Agarwala, J. Dorsey, and H. Rushmeier.Understanding and improving the realism of image composites.ACM Trans. Graph., 31(4):84:1–84:10, July 2012.URL http://doi.acm.org/10.1145/2185520.2185580.

L. Yao, P. Suryanarayan, M. Qiao, J. Z. Wang, and J. Li.Oscar: On-site composition and aesthetics feedback through exemplars for photographers.Int. J. Comput. Vision, 96(3):353–383, Feb. 2012.URL http://dx.doi.org/10.1007/s11263-011-0478-3.

A. G. Zajonc.Goethe’s theory of color and scientific intuition.American Journal of Physics, 44:327, 1976.

S.-H. Zhang, X.-Y. Li, S.-M. Hu, and R. R. Martin.Online video stream abstraction and stylization.IEEE Transactions on Multimedia, pages 1286–1294, 2011.


http://doi.acm.org/10.1145/2185520.2185580

http://dx.doi.org/10.1007/s11263-011-0478-3

a gentle introduction to color harmonization techniques

Education

janeiro ufrj

janeiro cep

janeiro ilha

whats color harmony

sistemas e computao

gentle introduction

concept of color harmony

whats harmony