“miscellaneous topics” - york universitymbrown/eecs6323/lectures/07_eecs63… · 26 the hsv/hsi...
TRANSCRIPT
![Page 1: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/1.jpg)
1
Computational Photography:
“Miscellaneous Topics”Part 1
Brown
![Page 2: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/2.jpg)
2
This lecture’s topic
We will discuss the following:
• Seam Carving for Image Resizing
– An interesting new way to consider resizing images
– This paper made a “wave” at SIGGRAPH’07
• Color Harmonization
– An automatic approach to harmonize colors
– With an overview of HSV colorspce
• Image Inpainting
– Seminal paper by Bertalmio et al on image restoration
![Page 3: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/3.jpg)
3
Three Papers DiscussedThree papers in this lecture:
1. Shai Avidan and Ariel Shamir SIGGRAPH 07
“Seam Carving for Content-Aware Image Resizing”
2. Daniel Cohen-Or et al, SIGGRAPH06
“Color Harmonization”
(With a pre-discussion on color)
3. Bertalmio et al SIGGRAPH’00
“Image Inpainting”
![Page 4: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/4.jpg)
4
“Seam Carving”
• SIGGRAPH 07Shai Avidan and Ariel Shamir
Shai was at MERL when paper was published,left for Adobe, then toTel-Aviv University
Ariel is a professor at the Efi Arazi School of Computer Science (Israel)
Both got their PhD’s from Hebrew University in 1999
• IdeaTypical image operation: scale-up or down and image
- Don’t necessarily want the image to be scaled pixel by pixel
- Instead want the content to fit into something smaller (call this re-targeting)
- How can we do this in a clever way?
ArielShai
![Page 5: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/5.jpg)
5
Idea Resizing
Here is an image and a plot of each pixels importance
(blue=less important, yellow=more important)
We want to reduce
the size of this image,
to make it smaller.
Let’s say we want to reduce the width of this
image by K columns.
![Page 6: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/6.jpg)
6
Cropping
Cropping
Current solution – crop the image:
Several automatic techniques exist to crop important parts of the
image (and remove the less important). Crop a region
such that a total of k-columns are remove from the left
and right.
![Page 7: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/7.jpg)
7
Column Removal
Column Removal(note artifacts)
Another option is to remove whole columns, where columns
with less energy are removed first. So, remove the K columns
with minimum energy. This results in strange artifacts.
![Page 8: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/8.jpg)
8
Pixel Removal
Pixel Removal
We can remove the K less important pixels per row.
This produces very ugly results.
![Page 9: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/9.jpg)
9
Seam Carving
Seam Carving
We can carve K seams (where a seam runs from top to bottom)
with the least energy.
![Page 10: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/10.jpg)
10
Comparison
![Page 11: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/11.jpg)
11
Seam Carving
Horizontal seam
A connected
line from left
to right that can
move at most
1-pixel per
row
Vertical seam
(connected line from top to bottom, that can move
at most 1-pixel per column)
![Page 12: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/12.jpg)
12
Seams and Energy
• If we want to reduce by either width or height, remove the seam with the least “energy”
• Optimal horizontal seam (right to left), where e is some energy function
• Define (vertical seam) bottom-to-top similarly
• A pixel (i,j) can only be part of one seam. So the goal then is to final the optimal “seams”, s*, such that:
*
![Page 13: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/13.jpg)
13
What Seam Energy to Use?
• The authors discuss several choices, but
these are the two they found the most
useful:
Simply gradient energy (not this is not magnitude, but an approximation)
Where, HoG is the histogram of oriented gradients. Using an 11x11 window about pixel (x,y), compute
gradients orientation and then build a histogram of the orientation (in this case an 8-bin histogram).
Take the max value of the histogram. A value with a large max means here is a strong edge in the
11x11 window. This energy attracts seams to edges, but not to cross the edge (because of the
numerator).
![Page 14: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/14.jpg)
14
e1 and eHoG example
![Page 15: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/15.jpg)
15
Image Expanding
• What if we want to expand the image?
• Picking the minimum energy seam and
duplicating it, gives a strange effect!
![Page 16: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/16.jpg)
16
Image Expanding
• Say we want to expand by K pixels
• Pick the K minimum seam
• Duplicate these seams by linear-interpolation
![Page 17: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/17.jpg)
17
A little help from the user
A failure case, seam carving is not aware of the
content’s meaning, only energy.
![Page 18: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/18.jpg)
18
A little help from the user
No problem, have the user assign “maximum”
energy to regions.
![Page 19: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/19.jpg)
19
Object Removal
Assign 0 energyAssign max energy
![Page 20: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/20.jpg)
20
No hope cases
Structure of content is not suitable for seam-carving.
OK, go back to regular resizing!
![Page 21: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/21.jpg)
21
Seam Carving Summary
• Very, very cool idea
– So obvious after you have seen it that you wish you had this idea!
• You’d be rich, Adobe would hire you and stuff your pockets with cash
• This paper is part of a move to “re-think” image editing
– “Content-aware” image editing
– Resizing while considering the content
– User assistant for hard cases
![Page 22: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/22.jpg)
22
“Color Harmonization”
• SIGGRAPH 07Daniel Cohen-Or et al
Daniel is a professor at the Tel Aviv University in Israel
He has many SIGGRAPH papers every year.
• Idea
Harmonization is the result of choosing colors that are pleasing to humans
- Can we provide a way to do this for images?
- Retarget colors to be harmonized?
Daniel
![Page 23: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/23.jpg)
Recall
• Our discussion on RGB color
• CIE XYZ perceptual space
• And sRGB color space
• Here, we will discuss HSV/HSI space
23
![Page 24: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/24.jpg)
CIE XYZ to CIE xy Chromaticity
[X, Y, Z]
(X=Y=Z)
CIE XYZ
x = X / (X+Y+Z)
y = Y / (X+Y+Z)
z = Z / (X+Y+Z)
CIE xy
![Page 25: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/25.jpg)
Standard RGB (sRGB)
R
G
In 1996, Microsoft and HP
defined the “standard”
RGB primaries.
![Page 26: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/26.jpg)
26
The HSV/HSI Colospace
• This is a different way at looking at the RGB color cube. Let’s first consider the Hue,
Saturation, and Value (HSV) color space which is a variation on the HLS.
• For HSV, we modify the RGB cube such that the greyscale line is the vertical axis.
We now modify the cube to be a cone. A color in this cone is expressed by three
values: 1) its position along the vertical axis (Value), 2) an angle (Hue) about that axis
(with reference point Hue=0o=red); 3) and the distance to the edge of the cone
(Saturation). We sometimes refer to hue as the “Color Wheel”.
RGB
Transformation Visualization between RGB and HSV
http://www.flashandmath.com/advanced/color/RGBtoHSV.html
Example Applet Showing RBG and HSV
http://dcssrv1.oit.uci.edu/~wiedeman/cspace/me/rgbhsv.html
![Page 27: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/27.jpg)
27
HVS and the related HLS
• Adobe uses the HLS space, which is similar to HSV. Instead of
Value, however, HLS uses the term “Lightness”. In, HSV, a value=1
does not result in a pixel that is completely white, it would only be
completely white if the saturation=0.
• However, for HLS, as the lightness value increases, the range of
colors decreases, thus everything becomes white with a
lightness=1.
![Page 28: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/28.jpg)
28
Color Enhancement via HSV Manipulation
• We can manipulate each component of hue, saturation and intensity for all pixels simultaneously
• Normally for image-editing applications:1. Convert colors in RGB representation to HSI
2. Manipulate HSI components, typically by• Hue transformation involves adding a user-specified
constant to the hues of all pixels (equivalent to rotating chromaticity plane about intensity axis)
• Saturation and intensity transformation involves scaling these values by a constant factor
3. Convert colors in HSI representation back to RGB
![Page 29: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/29.jpg)
29
HSI Manipulation Examples
Hue
Saturation
Intensity
![Page 30: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/30.jpg)
30
Another example
original
Hue modification
Saturation modifications
Lightness modifications
![Page 31: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/31.jpg)
31
Back to Color Harmony Paper
![Page 32: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/32.jpg)
32
What is color harmony?
• Harmonic colors are pleasing to the eye.
• They engage the human observer and
give a sense of order and balance in the
visual experience.
[slides from Cohen-Or’s SIGGRAPH talk]
![Page 33: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/33.jpg)
33
• Mathematical formulation has been developing
together with color theory
– Newton, Goethe, Young, Maxwell
• Itten [1960]: harmony means relationships on
the hue wheel:
2-color harmony:
complementary
colors
3-color harmony:
equilateral triangle
N-color harmony:
equilateral N-gon
Formal definition of color harmony?
![Page 34: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/34.jpg)
34
• Matsuda [1995]: extensive empirical studies,
derived 8 hue templates
• Tokumaru et al. [2002] developed a fuzzy
system to evaluate the harmony of color
schemes
i type V type L type I type
T type Y type X type N type
Formal definition of color harmony?
![Page 35: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/35.jpg)
35
Harmonic scheme
• The templates can be arbitrarily rotated
• Harmonic scheme is template type Tm +
specific orientation α
i type V type L type I type
T type Y type X type N type
Type “N” is not considered in this paper, this is for grayscale
images.
![Page 36: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/36.jpg)
36
Harmony score
• To evaluate the harmony of an input
image X we analyze its hue histogram:
– Every pixel p contributes its saturation S(p) to
the bin of the hue H(p)
![Page 37: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/37.jpg)
37
Harmony function
• The harmony of image X w. r. t. harmonic
scheme (Tm , α) :
( ), ( , ) ( ) ( ) ( )m Tm
p X
F X T H p E p S p
H(p)
ETm(α)(p)
This term is
the closest edge
of the template
(oriented at angle
alpha)
![Page 38: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/38.jpg)
38
Best template
• We compute α that minimizes F(X, (Tm , α))
for each template Tm using Brent’s
algorithm
• The best-fitting harmonic scheme:
0 0
( ,α)
( ,α ) argmin ,( ,α)m mm
T F X T
So, given an image, they compute the best fit template from
the different types (see 49) at the best orientation alpha.
![Page 39: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/39.jpg)
39
Harmonization
• Given (Tm , α) we shift the hues so that the
hue histogram is contained in (Tm , α)
![Page 40: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/40.jpg)
40
Color shifting
• The hue of pixel p is shifted to its associated sector
ETm(α)(p)
• The amount of squeezing is controlled by a Gaussian
fall-off function
![Page 41: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/41.jpg)
41
Color Coherency Problem
The problem, no way
to force neighboring pixels
to similar colors.
Here, similar colors (blue) move to
two different regions (green, purple).
![Page 42: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/42.jpg)
42
Color coherency
• Another example
Problem!
This would be better!
![Page 43: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/43.jpg)
43
Graph-cut optimization
• To make the coloring more coherent we
assign ETm(α)(p) by optimizing the labeling
V
1 2( ) ( ) ( )E V E V E V
Favors short
distance to the
template sector
Favors coherent
labeling of
neighboring pixels
[Back to MRF – each pixel now has a data-cost E1 and a neighbor cost E2.
This is similar to the lazy-snapping MRF formulation. ]
![Page 44: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/44.jpg)
44
Graph-cut optimization
• To make the coloring more coherent we
assign ETm(α)(p) by optimizing the labeling
V
1 2( ) ( ) ( )E V E V E V
1( ) ( ) ( ( )) ( )p
E V H p H V p S p
1
2 max
{ , }
( ) ( ), ( ) ( ) ( ) ( , )p q N
E V V p V q H p H q S p q
![Page 45: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/45.jpg)
45
Results
![Page 46: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/46.jpg)
46
Overcoming segmentation
problems• The graph-cut may fail when an object in
the image has several connected
components
![Page 47: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/47.jpg)
47
Overcoming segmentation problems
• User-assisted fix: scribbling on the erroneously
labeled area
• Re-compute the labeling
![Page 48: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/48.jpg)
48
Results – choosing colors
![Page 49: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/49.jpg)
49
Results – cut and paste
• The background is harmonized according to the best-
fitting harmonic template of the pasted foreground
original harmonized harmonized
![Page 50: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/50.jpg)
50
Results
• Text over a poster
![Page 51: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/51.jpg)
51
Results
• Text over a poster
![Page 52: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/52.jpg)
52
Results
Images harmonized
to different “flags”
colors.
Find the flags template,
force the image to this.
![Page 53: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/53.jpg)
53
Discussion
• Nature is already harmonic
original best-fitting
template
poorly-fitting
template
![Page 54: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/54.jpg)
54
Discussion
• Cannot improve good artwork!
Wassily Kandinsky, Composition VII, 1913
![Page 55: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/55.jpg)
55
Discussion
• Grayish colors will remain such
![Page 56: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/56.jpg)
56
Harmony Summary
• Provides a method to enhances the harmony of colors in a given image
• Operates by fitting the image hues into a given harmonic distribution
– Several different harmonic chooses are pre-defined (based on color theory)
• Especially useful for ‘artificial’ colors, cut-and-paste settings and collages that combine imagery from different sources
![Page 57: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/57.jpg)
57
“Image Inpainting”
• SIGGRAPH’00Marcelo Bertalmio et al
While Marcelo was a PhD student at U. Of Minnesota
Now a professor at Universitat Pompeu Fabra (Spain)
• Idea
Find a good way to fill in missing image information
- The idea comes from how artists fix paintings when the paint chips away:
they “in paint”
Marcelo
![Page 58: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/58.jpg)
58
Real “Inpainting”
http://www.cac-accr.ca/english/e-CAC-publications-CAC-pub-sel.asp
![Page 59: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/59.jpg)
59
Image Inpainting
Assume you have an image that has been
corrupted. Above you want to fill in the “white”
pixels with the surrounding content.
Or, another scenario, you
the user draw the region
you want to correct.
![Page 60: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/60.jpg)
60
The idea
Image I
∂Ω
Ω
Ω
∂Ω
is a “hole” you want to repair
is a the border of the hole
with pixel intensities
Idea: propagate information from ∂Ω inside to
Ω. But, do this in a clever manner.
![Page 61: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/61.jpg)
61
The problem
Problem: Need to
propagate the
information along image
gradients! Otherwise
the result will be
incorrect.
Hole to be filled
Incorrect
![Page 62: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/62.jpg)
62
Solution
The paper write up is very complicated, but can be followed if you read
it slowly. Basic idea is that at each step you try to propagate the boundary
∂Ω pixels in image gradient direction. The boundary will slowly shrink.
This slowly fills in the hole (and propagates gradient) thus maintaining
the direction.
(start) (after many iterations)
(more iterations) (and yet more iterations)
![Page 63: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/63.jpg)
63
Examples
Inpainting result.
The key here is that image
gradient was incorporated.
![Page 64: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/64.jpg)
64
Limitations
Not perfect, because it can’t reproduce “texture”
![Page 65: “Miscellaneous Topics” - York Universitymbrown/EECS6323/lectures/07_EECS63… · 26 The HSV/HSI Colospace • This is a different way at looking at the RGB color cube. Let’s](https://reader035.vdocuments.us/reader035/viewer/2022070816/5f100f527e708231d4473f0d/html5/thumbnails/65.jpg)
65
Inpainting Summary
• This idea spawned a great deal of further work– Google Scholar has 1902 citations to this paper
• The idea is simple– Fill in the hole by propagating information
– The approach is clever
• But the problem is good– The user helps by giving the region to correct
– Again, we are seeing user-assistance in the procedure
• The algorithm itself is automated
• Similar ideas:
– Bayesian matting/Possion Matting/PIE/and so on