colorization using optimizationcsip/csip2007-colorization.pdf · colorization using optimization...
TRANSCRIPT
Anat Levin Dani Lischinski Yair Weiss
School of Computer Science and EngineeringThe Hebrew University of Jerusalem, Israel
Colorization Using Optimization
1Tuesday, June 19, 2007
Colorization: a computer-assisted process of adding color to a monochrome image or movie.
(Invented by Wilson Markle, 1970)
Colorization
2Tuesday, June 19, 2007
• Colorizing black and white movies?
• Maybe not...
• But what about black and white TV shows?
Motivation
3Tuesday, June 19, 2007
• Colorizing black and white movies / TV shows
• Recoloring color images for special effects
Motivation
4Tuesday, June 19, 2007
5Tuesday, June 19, 2007
Color 101• Typically represented by points in a three
dimensional color space:• (Red, Green, Blue)• (Cyan, Magenta, Yellow)
• Some color spaces separate between the grayscale and the chromatic components of a color:
6Tuesday, June 19, 2007
Typical Colorization Process
Images from: “Yet Another Colorization Tutorial”
http://www.worth1000.com/tutorial.asp?sid=161018
7Tuesday, June 19, 2007
Images from: “Yet Another Colorization Tutorial”
http://www.worth1000.com/tutorial.asp?sid=161018
Typical Colorization Process• Delineate region boundary
8Tuesday, June 19, 2007
Images from: “Yet Another Colorization Tutorial”
http://www.worth1000.com/tutorial.asp?sid=161018
Typical Colorization Process• Delineate region boundary• Choose region color from palette
9Tuesday, June 19, 2007
Images from: “Yet Another Colorization Tutorial”
http://www.worth1000.com/tutorial.asp?sid=161018
Typical Colorization Process• Delineate region boundary• Choose region color from palette
10Tuesday, June 19, 2007
Images from: “Yet Another Colorization Tutorial”
http://www.worth1000.com/tutorial.asp?sid=161018
Typical Colorization Process• Delineate region boundary• Choose region color from palette
11Tuesday, June 19, 2007
Video Colorization Process• Delineate region boundary• Choose region color from palette
• Track regions across video frames
12Tuesday, June 19, 2007
Process Limitations
• Time consuming and labor intensive
• Difficult to delineate detailed or fuzzy boundaries
• Failures in tracking
13Tuesday, June 19, 2007
Hertzmann et al. 2001, Welsh et al. 2002
A : A’
B : B’
?
Colorization by Analogy
14Tuesday, June 19, 2007
Colorization by Analogy - Discussion
• Indirect artistic control
• No spatial continuity constraint
15Tuesday, June 19, 2007
Our Approach
16Tuesday, June 19, 2007
Artist scribbles desired colors inside regions
Our Approach
17Tuesday, June 19, 2007
Colors are propagated to all pixels
“Nearby pixels with similar intensities should have the same color”
Our Approach
18Tuesday, June 19, 2007
Intensity channel Color channels
Assumption: Neighboring pixels with similar intensities should have similar colors
Propagation using Optimization
19Tuesday, June 19, 2007
Intensity channel Color channels
Propagation using Optimization
Minimize the sum of affinity-weighed squared distances between the colors of adjacent pixels:
J(U) = !r,s
wrs (U(r)!U(s))2
20Tuesday, June 19, 2007
Affinity Functions
proportional to local variancewrs ! e!(Y (r)!Y (s))2/2!2
r
21Tuesday, June 19, 2007
Space-time Affinity
wrs ! e!(Y (r)!Y (s))2/2!2r
22Tuesday, June 19, 2007
Since the cost function is quadratic, its minimum may be found by solving sparse system of linear equations.
Minimize the cost function:
Subject to labeling constraints
Optimization
J(U) = !r,s
wrs (U(r)!U(s))2
!U(rk) = Uk
"
23Tuesday, June 19, 2007
Optimization
Minimize by solving the equations:
!J(U) = 0!J(V ) = 0
U! = argminU
J(U)
V ! = argminV
J(V )
24Tuesday, June 19, 2007
DerivationJ(U) = !
r,swrs (Ur!Us)2
= !r,s
wrs!U2
r !2UrUs +U2s"
= !r,s
wrsU2r !2!
r,swrsUrUs +!
r,swrsU2
s
= !r
DrrU2r !2!
r,swrsUrUs +!
sDssU2
s
= 2UT (D!W )U
25Tuesday, June 19, 2007
Derivation
J(U) = 2UT (D!W )U
Unconstrained system:
(D!W )U = 0
Note: has a trivial solution (any constant vector U).
!J(U) = 4(D!W )U = 0
26Tuesday, June 19, 2007
DerivationAdd constraints:
J(U) = UT (D!W )U +! !k
!Uk!Uk
"2
J(U) = UT (D!W )U +!!U!U
"T C!U!U
"
!J(U) = 2(D!W )U +2!C!U!U
"= 0
Constrained system:
(D!W +!C)U = !CU
27Tuesday, June 19, 2007
Color Interpolation
28Tuesday, June 19, 2007
Colorizing Stills
29Tuesday, June 19, 2007
30Tuesday, June 19, 2007
Original Colorized
Colorizing Stills
31Tuesday, June 19, 2007
Colorizing Stills
32Tuesday, June 19, 2007
Colorizing Stills
33Tuesday, June 19, 2007
Challenging Case
34Tuesday, June 19, 2007
Our result
Segmentation aided colorization
NCuts Segmentation (Shi & Malik 97)
Segmentation?
35Tuesday, June 19, 2007
Affinity between pixels – based on intensity AND color similarities.
Recoloring
36Tuesday, June 19, 2007
Recoloring
37Tuesday, June 19, 2007
Recoloring
38Tuesday, June 19, 2007
13 out of 92 frames
Colorizing Video
39Tuesday, June 19, 2007
16 out of 101 frames
Colorizing Video
40Tuesday, June 19, 2007
Red channel<->matte
Matting
41Tuesday, June 19, 2007
Matting
42Tuesday, June 19, 2007
Future Work
• Import image segmentation advantages: affinity functions, optimization techniques.
• Alternative color spaces, propagating hue and saturation differently
43Tuesday, June 19, 2007
★ Interface: User scribbles color on a small number of pixels
★ Colors propagate in space-time volume respecting intensity boundaries
★ Convincing colorization with a small amount of user effort
Code & examples available: www.cs.huji.ac.i/~yweiss/Colorization/
Summary
44Tuesday, June 19, 2007