qualifying exam - image-based reconstruction with color harmonization

Federal University of Rio de Janeiro - UFRJ - Campus Cidade Universitária - Rio de Janeiro - Ilha do Fundão - COPPE/PESC/LCG

Thirty Minute Speech :: Overview of Activities Developed in Disciplines and Guided Studies :: Laboratory Seminars and Meetings

Thirty Minute SpeechAn Overview of Activities Developed in Disciplines and Guided Studies

Michel Alves dos SantosGraduate Program in Systems Engineering and ComputingGraduate Program in Systems Engineering and ComputingFederal University of Rio de Janeiro - UFRJ - COPPEFederal University of Rio de Janeiro - UFRJ - COPPE

Advisors: D.Sc. Ricardo Marroquim & Ph.D. Cláudio Esperança

{michel.mas, michel.santos.al}@gmail.com

Feb, 2015Feb, 2015«Image-Based Reconstruction With Color Harmonization»

Michel Alves: Laboratory of Computer Graphics/LCG Graduate Program in Systems Engineering and Computing



Introduction - Work Inspiration


[Bundler, http://www.cs.cornell.edu/∼snavely/bundler/, 2015]

[Coliseum Reconstruction - Structure from Motion (SfM) for Unordered Image Collections]Image-Based

Recon

struction



Introduction - Usage and Importance


[3DVisA Index of 3D Projects, http://3dvisa.cch.kcl.ac.uk/project86.html, 2015]Image-Based

Recon

struction



Proposal for Work

Join Image-Based Reconstruction and Color Harmonization

But this ideia have some challenges...


[Seamless montage, Gal et al., 2010] [Color Harmonization, Cohen-Or et al., 2006]



Challenges

I Creation of textures without discard of original image set;I Employment of color harmonization techniques in texturing.


[Splat-based Surface Reconstruction from Defect-Laden Point Sets, Campos et al., 2013]



Applications

I Product Design;I Restorations/Simulations;I Architectural Support;I Games/Scenery Modeling, etc.


[Automated 3D Model Reconstruction from Photographs, Paul Bourke, 2013]



Steps of Proposed Work

I Acquisition of point cloud from the image set;I Reconstruction from the point cloud;I Texture Montage (stitching & seamless montage);I Color Correction (using image blending);I Employment of color harmonization techniques.


[Creating 3D Models With a Simple Webcam - ProFORMA, Qi Pan, 2009]



Image-Based Reconstruction


[Building Rome in a Day, Agarwal et al., 2009]

[The Colosseum, 2.106 images, 819.242 points - http://grail.cs.washington.edu/rome/]



Image-Based Reconstruction - Steps


[Digital Pygmalion project: from photographs to 3D computer model]



Image-Based Reconstruction - Survey


[A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, Seitz et al., 2006]

[http://vision.middlebury.edu/mview]Dino Dataset

Temple Dataset



Texturing - Atlas

� Generally, the base/original image set is discarded.

� We will use a form without discard of original image set.


[Least Squares Conformal Maps for Automatic Texture Atlas Generation, Lévy et al., 2002]



Image Stitching

I Images could create a visible seam (illumination and background changing);


[Geometrical registration and stitch line in panorama creation, 2015]

[Fast Poisson Blending using Multi-Splines, Szeliski et al., 2013]



Seamless Montage

simple back-projection [Lempintsky & Ivanov] [Gal et. Al]

I Method to avoid seams in texture montage.


[Seamless Montage for Texturing Models, Gal et al., 2010]



Image Blending

I Correction of color in seams;I Used to compensate exposure differences and other misalignments.


[A Multiresolution Spline With Application to Image Mosaics, Burt & Adelson, 1983]

[Fast Poisson Blending using Multi-Splines, Szeliski et al., 2013]

630 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 37, NO. 4, OCTOBER 2012

TABLE IBLENDING TECHNIQUES: A COMPARISON

TS: Transition Smoothing; OS: Optimal Seam; L: Luminance; W: Wavelet; G: Gradient; R: Radiance; GS: Gray Scale; CW: Color Wise; SC: Single Channel.

step [18] does not offer good enough results when registrationproblems occur. Second, the exposure correction mechanismselected may lead to a global exposure degeneration. Last, thebehavior of the method when faced with large input image se-quences is unknown.Few approaches in the literature have specifically faced the

problem of underwater imagery mosaicing. Gu and Rzhanov[43], similarly to Agarwala et al. [36], proposed a graph-cuttechnique to select the optimal seam between two images, andthe application around this boundary of a pure gradient-domainfusion. The method claims to overcome the short comings ofpure graph-cut techniques, which show noticeable seams in thecase of changing illumination conditions, and gradient-domainfusion, which produce blurring in case of misalignment. The au-thors do not define criteria for selecting the contributing imagein the case of multiple images overlapping in the same region.Thus, Gu and Rzhanov [43] limited it to “panoramic” mosaicswhere only two images overlap in the same area. Furthermore,a conventional graph-cut approach, in the case of gigapixel mo-saics, may lead to nonoptimal seams in the presence of differentexposures or illuminations.

III. CLASSIFICATION OF TECHNIQUES

The list of papers that conform the state of the art on imageblending is long, and the main requirements for conventionalpanorama image generation have been satisfyingly addressedby several of them. Unfortunately, blending in underwater pho-tomosaicing is a specific application, which has not been treatedthoroughly in the literature. Consequently, despite the numerousimage mosaicing methods, not all of them are adequate to dealwith large-scale underwater photomosaicing. To highlight the

properties, benefits, and drawbacks of the current methods, andwith the aim of evaluating their application to underwater mo-saicing, a classification is proposed.There are several criteria that determine the behavior and

performance of a given blending algorithm, including theircapability of dealing with high-resolution underwater photo-mosaics. Table I provides a comprehensive comparison of themost relevant blending techniques proposed in the literature.The specially important categories for underwater applications(mostly working with monochrome images) are exposurecorrection and elimination of ghosting and double contouring,concerning image quality and scalability in large-scale photo-mosaicing.

A. Basic Principle

Two main groups of algorithms can be found in the literaturein the context of image blending [35]: transition smoothing(TS) and optimal seam finding (OS) techniques. TS methods,also known as feathering [31] or alpha blending methods [46],attempt to minimize the visibility of seams by smoothing thecommon overlapping regions of the combined images. TSmethods often suffer from ghosting, a blurring of the finestdetails (i.e., low-frequency image components), and doublecontouring, consisting of practice on a partial duplication ofcertain scene structures (i.e., high-frequency image compo-nents), if registration is not accurate enough or the scenarioconsiderably violates the planar scene assumption for 2-Dmosaicing. OS methods place the seam between images wherephotometric differences in their joining boundaries are minimal[29], [47]. OS methods are not able to deal with images withdifferent exposures, as is often the case in underwater imagery

Image Blending: Techniques

[A Novel Blending Technique for Underwater Gigamosaicing, Prados et al., 2012]



Color Harmonization


Original Optmized for α = 30 ◦ Optmized for α = 330 ◦

[Color Harmonization Project, Michel Alves, 2013]



Color Harmonization: About

ground object to the background, so that together they form a har-monic color set (see Figure 1). In general, our algorithm is usefulfor enhancing colors in images that are comprised of a collection ofparts originating from different sources and whose colors requireharmonization.

2 Background and Related Work

The study of color harmony is historically intertwined with thestudy of the physical nature of light and color. Early discover-ies in the theory of color harmony were made by such mastersas Newton, Goethe, Young, and Maxwell. Modern color theory,which was developed at the beginning of the 20th century, dealsmainly with representations of colors, but it also discusses colorharmony [Munsell 1969; Ostwald and Birren 1969; Itten 1960].Moon and Spencer [1944] introduced a quantitative representa-tion of color harmony based on the Munsell color system [Munsell1969]. At the same time, Granville and Jacobson [1944] presented aquantitative representation of color harmony based on the Ostwaldcolor system [Ostwald and Birren 1969]. To a large degree, theseworks define harmony as order.

Itten [1960] introduced a new kind of color wheel in which he de-scribed color harmony, with an emphasis on hue. Itten’s color har-mony theory is based on the relative positions of the hues on thecolor wheel. For example, from the three primary colors of cyan,magenta, and yellow, Itten designed a hue wheel of twelve colors.He referred to complementary colors as a two-color harmony. Ittenalso recognized the three-color harmony of hues that form an equi-lateral triangle, the four-color harmony of hues forming a square,the six-color harmony of a hexagon, etc. His schemes have beenwidely adopted by artists and designers. Based on Itten’s schemesand extensive psychophysical research, Matsuda [1995] introduceda set of 80 color schemes, defined by combining several types ofhue and tone distributions. These schemes were used in [Tokumaruet al. 2002] for harmony evaluation and color design. Our colorharmonization method is also based on these schemes.

There are various interactive tools that provide designers with har-monic sets (e.g., [Color Schemer 2000; Color Wheel Expert 2000;Nack et al. 2003]). Such applications provide the user with a set ofharmonic colors that accommodates the user’s requirements spec-ified by a color seed and possibly a number of other parameters.Meier et al. [1988] presented a system for designing colors basedon several color rules, and applied them to a graphical user inter-face (GUI) building tool. The primary goal of their system was totest whether an automated mechanism would be a viable solution tothe problem of choosing effective and tasteful colors. None of theabove systems offers a means to harmonize a given arbitrary colorimage. The method we introduce in this paper automatically har-monizes a given color palette through an optimization process, andprovides a means to automatically recolor an arbitrary image.

Our work is also related to general recoloring methods [Reinhardet al. 2001; Welsh et al. 2002; Levin et al. 2004; Gooch et al.2005; Ironi et al. 2005; Rasche et al. 2005]. Automatic recoloringtechniques require the user to provide a reference image. The rela-tionship between the colors of the input and the reference imagesare learned and transferred to recolor the given image. One of thechallenges in these techniques is to recolor the image in a coherentway [Ironi et al. 2005]. In other words, contiguous spatial regionsin the input image should remain contiguous after the recoloring.Our color harmonization process uses a graph-cut optimization toenforce contiguous modification of colors in image space.

i type V type L type I type

T type Y type X type N type

Figure 2: Harmonic templates on the hue wheel. A collection ofcolors that fall into the gray areas is considered to be harmonic.The templates may be rotated by an arbitrary angle. The sizes ofthe sectors are specified in the Appendix.

3 Harmonic Schemes

The notion of color harmony in this work is based on the schemesdeveloped by Matsuda [Matsuda 1995; Tokumaru et al. 2002],which descend from Itten’s notions of harmony [Itten 1960], widelyaccepted in applicable fields involving colors. Figure 2 illustratesthe eight harmonic types defined over the hue channel of the HSVcolor wheel. Each type is a distribution of hue colors that definesa harmonic template: colors with hues that fall in the gray wedgesof the template are defined as harmonic according to this template.We refer to these distributions as templates, since they define theradial relationships on the color wheel, rather than specific colors(meaning that any template may be rotated by an arbitrary angle).The harmonic templates may consist of shades of the same col-ors (types i, V and T), possibly with complementary colors (seetemplates I, Y, X) or more complex combinations (template L andits mirror image). The sectors of these templates are the domainsover which simple membership functions are defined. Color har-mony is mainly affected by the hue channel; however, Tokumaru etal. [2002] also addressed tone distribution functions for the valuesof the S and V channels, and fuzzy rules for the correlation betweenthe hue templates and the tone distributions. For details, the readeris referred to [Tokumaru et al. 2002].

The type-N template corresponds to gray-scale images and thusis not dealt with in this work. Note that each of the remainingseven templates consists of one or two sectors. Each hue h onthe color wheel is then associated with one of these sectors. Thesimplest way is to associate h with the closest (in terms of arclength) sector. Thus, we define ETm(α)(p) as the sector borderhue of template Tm with orientation α that is closest to the hue ofpixel p (m ∈ {i, I,L,T,V,X ,Y}).

Given an image, we fit a harmonic template Tm to the hue his-togram of the image. We define a distance between the histogramand a template, and determine the template that best fits our im-age by solving an optimization problem. A template Tm togetherwith an associated orientation α defines a harmonic scheme, de-noted by (m,α). Given a harmonic scheme (m,α), we define afunction F(X ,(m,α)) which measures the harmony of an image Xwith respect to the scheme (m,α):

F(X ,(m,α)) = ∑p∈X

∥∥∥H(p)−ETm(α)(p)∥∥∥ ·S(p) , (1)

where H and S denote the hue and the saturation channels, respec-tively; the hue distance ‖ · ‖ refers to the arc-length distance on thehue wheel (measured in radians); hues that reside inside the sec-tors of Tm are considered to have zero distance from the template.

625

Saturation

HSV System

Use of the Tokumaru Templates

Optimization of Potential Functions and Color Mapping

Works Only on HSV Hue Channel


[Color Harmonization, Cohen-Or et al., 2006]



Color Harmonization: Technique in Action

V type

T type

I Distance between image and template;I Optimization of the position α of template sector;I Appropriate color mapping.




Color Harmonization: Main Authors

(*) Color Mapping (**) Potential Functions (***) Segmentation

� Color Harmonization/Cohen-Or et al., 2006 (*), (**)

� Color Harmonization for Videos/Sawant & Mitra, 2008 (*), (**), (***)

� Improved Color Harmonization/Huo & Tan, 2009 (*), (**)

� Color Harmonization for Images/Tang et al., 2010 (**)

� Color Harmonization Enhancement/Baveye & Fargeas, 2012 (***)

� Saliency-Guided Color Harmonization/Baveye et al., 2013 (*), (***)




Color Harmonization: Focus on Color Mapping

I Cohen-Or et al.:H ′(p) = C(p) + w

2 · (1− Gσ(||H(p)− C(p)||)) Gσ(x) = 1σ√2π · e

− x22σ2

I Sawant & Mitra:H ′(p) = C(p) + sgn · θ2 · Lσ(H(p)− E (p)) Lσ(x) = σ · x

I Huo & Tan:H ′(p) = C(p) + sgn · w

2 · (1− Eσ(||H(p)− C(p)||)) Eσ(x) = e−x22σ2

I Baveye et al.:

H ′(p) = C(p) + sgn · w2 · Stgh

(||H(p)−C(p)||

w/2

)Stgh(x) = e2x−1

e2x+1




Color Harmonization: Previous Results

I Now, we will expose some previous results in color harmonization...




Color Harmonization: Previous Results/Misc

Original Template T, α = 35 ◦ Template V, α = 5 ◦

Original Template L, α = 40 ◦ Template Y, α = 320 ◦


[Color Harmonization Project, Michel Alves, 2013]



Color Harmonization: Previous Results/Paint


[Girl with a Pearl Earring, Johannes Vermeer, 1665]

Original Template V, α = 335 ◦ Template L, α = 275 ◦



Color Harmonization: Previous Results/Eyes


Original Template i, α = 15 ◦

Template L, α = 10 ◦ Template V, α = 0 ◦



Color Harmonization: Previous Results/Room


Original Template Y, α = 170 ◦





Original Template L, α = 50 ◦





Original Template Y, α = 10 ◦



Color Harmonization: Proposal of Short Paper

«..Employment of Kernels and Sigmoids on Harmonized Color Mapping..»




ThanksThanks for your attention!

Michel Alves dos Santos - http://www.michelalves.com

Michel Alves dos Santos - (Alves, M.)Federal University of Rio de JaneiroE-mail: [email protected], [email protected]ésumé: http://lattes.cnpq.br/7295977425362370Personal Page: http://www.michelalves.com

http://www.facebook.com/michel.alves.santos

http://www.linkedin.com/profile/view?id=26542507


http://www.michelalves.com

http://www.ufrj.br

mailto:[email protected]

mailto:[email protected]

http://lattes.cnpq.br/7295977425362370

http://www.michelalves.com



Main Bibliography I3DVisA.3dvisa index of 3d projects: Museum applications, 2006.URL http://3dvisa.cch.kcl.ac.uk/project86.html.

S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and R. Szeliski.Building rome in a day.In International Conference on Computer Vision, Kyoto, Japan, September 2009. IEEE.

Y. Baveye, F. Urban, C. Chamaret, V. Demoulin, and P. Hellier.Saliency-guided consistent color harmonization.7786:105–118, 2013.

P. Bourke.Workshop: Automated 3d model reconstruction from photographs, April 2013.URL http://paulbourke.net/papers/cgat2013workshop/.

P. Bourke.Automated 3d model reconstruction from photographs, March 2014.URL http://paulbourke.net/papers/dha2014/.

P. J. Burt and E. H. Adelson.A multiresolution spline with application to image mosaics.ACM Trans. Graph., 2(4):217–236, Oct. 1983.

R. Campos, R. Garcia, P. Alliez, and M. Yvinec.Splat-based surface reconstruction from defect-laden point sets.Graphical Models, 75(6):346 – 361, 2013.


http://3dvisa.cch.kcl.ac.uk/project86.html

http://paulbourke.net/papers/cgat2013workshop/

http://paulbourke.net/papers/dha2014/



Main Bibliography IID. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand, and Y.-Q. Xu.Color harmonization.ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 25(3):624–630, 2006.

C. H. Esteban.Research on 3d object modelling, 2008.URL http://carlos-hernandez.org//research.html.

R. Gal, Y. Wexler, E. Ofek, H. Hoppe, and D. Cohen-Or.Seamless montage for texturing models.Comput. Graph. Forum, 29(2):479–486, 2010.

L. Gruber, D. Kalkofen, and D. Schmalstieg.Color harmonization for augmented reality.In Mixed and Augmented Reality, 2010 9th IEEE International Symposium on, pages 227–228, 2010.

C. Hernández.Stereo and Silhouette Fusion for 3D Object Modeling from Uncalibrated Images Under Circular Motion.PhD thesis, Ecole Nationale Supŕieure des Télécommunications, May 2004.

X. Hou and L. Zhang.Color conceptualization.In Proceedings of the 15th international conference on Multimedia, pages 265–268. ACM, 2007.

X. Huo and J. Tan.An improved method for color harmonization.In Image and Signal Processing, 2009. CISP’09. 2nd International Congress on, pages 1–4, 2009.


http://carlos-hernandez.org//research.html



Main Bibliography IIIB. Lévy, S. Petitjean, N. Ray, and J. Maillot.Least squares conformal maps for automatic texture atlas generation.ACM Trans. Graph., 21(3):362–371, jul 2002.

B. S. Morse, D. Thornton, Q. Xia, and J. Uibel.Image-based color schemes.In Image Processing. ICIP 2007. IEEE International Conference on, volume 3, pages III–497. IEEE, 2007.

P. O’Donovan, A. Agarwala, and A. Hertzmann.Color compatibility from large datasets.In ACM Transactions on Graphics (TOG), volume 30, page 63. ACM, 2011.

Q. Pan, G. Reitmayr, and T. Drummond.Proforma: Probabilistic feature-based on-line rapid model acquisition, 2008.URL http://www.bmva.org/bmvc/2009/Papers/Paper297/Paper297.pdf.

Q. Pan, G. Reitmayr, E. Rosten, and T. Drummond.Rapid 3d modelling from live video.In MIPRO, 2010 Proceedings of the 33rd International Convention, pages 252–257, May 2010.

P. Perez, M. Gangnet, and A. Blake.Poisson image editing.ACM Trans. Graph., 22(3):313–318, July 2003.

R. Prados, R. Garcia, N. Gracias, J. Escartin, and L. Neumann.A novel blending technique for underwater gigamosaicing.Oceanic Engineering, IEEE Journal of, 37(4):626–644, Oct 2012.


http://www.bmva.org/bmvc/2009/Papers/Paper297/Paper297.pdf



Main Bibliography IVN. Sawant and N. J. Mitra.Color harmonization for videos.In Indian Conference on Computer Vision, Graphics and Image Processing, pages 576–582, 2008.

S. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski.A comparison and evaluation of multi-view stereo reconstruction algorithms.In Computer Vision and Pattern Recognition, Conference on, volume 1, pages 519–528, June 2006a.

S. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski.Bundler: Structure from motion (sfm) for unordered image collections, 2006b.URL http://vision.middlebury.edu/mview.

N. Snavely.Bundler: Structure from motion (sfm) for unordered image collections, 2008.URL http://www.cs.cornell.edu/~snavely/bundler/.

K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister.Multi-scale image harmonization.ACM Transactions on Graphics (Proc. ACM SIGGRAPH), 29(4):125:1–125:10, 2010.

R. Szeliski.Image alignment and stitching: A tutorial.Technical report, Microsoft Research, Redmond, WA 98052, 2006.

R. Szeliski.Computer Vision: Algorithms and Applications.Texts in Computer Science. Springer, 2010.


http://vision.middlebury.edu/mview

http://www.cs.cornell.edu/~snavely/bundler/



Main Bibliography V

R. Szeliski, M. Uyttendaele, and D. Steedly.Fast poisson blending using multi-splines.In Computational Photography (ICCP), 2011 IEEE International Conference on, pages 1–8, April 2011.

Z. Tang, Z. Miao, Y. Wan, and Z. Wang.Color harmonization for images.Journal of Electronic Imaging, 20(2):023001–023001, 2011.

Y. Wan, Z. Tang, Z. Miao, and B. Li.Image composition with color harmonization.IJPRAI, 26(3), 2012.

B. Wang, Y. Yu, T.-T. Wong, C. Chen, and Y.-Q. Xu.Data-driven image color theme enhancement.In ACM Transactions on Graphics (TOG), volume 29, page 146. ACM, 2010.

L. Wang and K. Mueller.Harmonic colormaps for volume visualization.In Proceedings of the Fifth Eurographics VGTC conference on Point-Based Graphics, pages 33–39, 2008.

L. Wang, J. Giesen, K. T. McDonnell, P. Zolliker, and K. Mueller.Color design for illustrative visualization.IEEE Transactions on Visualization and Computer Graphics, 14(6):1739–1754, 2008.


qualifying exam - image-based reconstruction with color harmonization

Education