![Page 1: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/1.jpg)
Image Editing in the Gradient Domain
Shai AvidanTel Aviv University
![Page 2: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/2.jpg)
Slide Credits (partial list)
• Rick Szeliski
• Steve Seitz• Alyosha Efros
• Yacov Hel-Or• Marc Levoy
• Bill Freeman• Fredo Durand
• Sylvain Paris
![Page 3: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/3.jpg)
![Page 4: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/4.jpg)
Image Composition
Target ImageSource Images
![Page 5: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/5.jpg)
Basics
• Images as scalar fields
– R2 -> R
![Page 6: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/6.jpg)
Vector Field
• A vector function G: R2 →R2
• Each point (x,y) is associated with a vector (u,v)
G(x,y)=[ u(x,y) , v(x,y) ]
![Page 7: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/7.jpg)
Gradient Field• Partial derivatives of scalar field • Direction
– Maximum rate of change of scalar field
• Magnitude– Rate of change
• Not all vector fields are gradients of an image.– Only if they are curl-free (a.k.a conservative)– What’s the difference between 1D and 2D Gradient field?
,y
I
x
II
∂∂
∂∂
=∇
),( yxI
![Page 8: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/8.jpg)
Continues v.s. Discrete
Image I(x,y)
Ix Iy
Continues case → derivative
Discrete case → Finite differences
[ ]
[ ] Iy
I
Ix
I
T ∗−→∂∂
∗−→∂∂
110
110
![Page 9: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/9.jpg)
Interpolation
Ω
Ω
Ω∂Ω
over function scalar unknown :f
\over function scalar known :f
boundary with , ofsubset closed a :
ofsubset closed a :
*
2
S
S
RS
I
![Page 10: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/10.jpg)
Intuition – hole filling
• 1D:
• 2D:
x1 x2
![Page 11: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/11.jpg)
Membrane Interpolation2
min ∫∫Ω
∇ff
Solve the following minimization problem:
Subject to Dirichlet boundary conditions:Ω∂Ω∂
= *ff
Variational Methods to the Rescue!
Calculus:
When we want to minimize g(x) over the space of real values xWe derive and set g’(x)=0
What’s the derivative of a function?
Variational Methods:Express your problem as an energy minimization over a space of functions
![Page 12: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/12.jpg)
Derivative Definition
1D Derivative: ( ) ( ) ( )ε
εε
xfxfxf
−+=′
→0lim
Multidimensional derivative for some direction vector w
( ) ( ) ( )ε
εε
xfwxfxfDw
rrrr
r−+
=→0
lim
We want to minimize ( ) ( ) ( )∫ ==′2
1
112 and with
x
x
bxfaxfdxxf
Assume we have a solution f and try to define some notion of 1D derivativewrt to a 1D parameter ε in a given direction of functional space:
For a perturbation function η(x) that also respects boundary conditions (i.e.η(x_1)=η(x_2) = 0) and a scalar ε the integral
( ) ( )( )∫ ′′+′2
1
alone n bigger tha be should 2x
x
fdxxxf ηε
![Page 13: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/13.jpg)
And we are left with:
But since this must be true for every η, it holds that f’’(x) = 0 everywhere.
Calculus of VariationsLets open the parenthesis: ( ) ( ) ( ) ( ) dxxηεxfxηεxf
x
x∫ ′+′′+′
2
1
222 2
The third term is always positive and is negligible when ε is going to zero.
So derive the rest with respect to ε and set to zero: ( ) ( )∫ =′′2
1
02x
x
dxxfxη
Integrate by parts: ( ) ( ) ( ) ( )[ ] ( ) ( )∫ ∫ ′′′−′=′′2
1
2
1
2
1
x
x
x
x
xx dxxfxη xfxηdxxfxη
Where: ( ) ( )[ ] ( ) ( ) ( ) ( )agafbgbfxgxf ba −=
And since η(x_1)=η(x_2) = 0 then the expression in the squared brackets is equal to zero
( ) ( ) 02
1
=′′′∫x
xdxxfxη
![Page 14: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/14.jpg)
Intuitioninterval over the integrated slove theis ofmin The ∫ ′f
Locally, if the second derivative was not zero, this would mean that the First derivative is varying which is bad since we want to be minimized∫ ′f
Recap:
We start with the functional we need to minimizeIntroduce the perturbation functionUse Calculus of variationSet to zeroIntegrate by partsAnd obtain the solution.
![Page 15: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/15.jpg)
Euler-Lagrange Equation
( )∫=2
1
,,x
xx
dxffxFJ
A fundamental equation of calculus of variations, that states that if J is defined by an integral of the form
Then J has stationary value if the following differential equation is satisfied
0=∂∂
−∂∂
xf
F
dx
d
f
FEquation (2)
Equation (1)
2min ∫∫
Ω
∇ff
Recall, we want to solve the following minimization problem:
Subject to Dirichlet boundary conditions:Ω∂Ω∂
= *ff
![Page 16: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/16.jpg)
Membrane Interpolation( )222
:caseour In yx fffF +=∇=
0 :becomes (2)equation Then =∂∂
−∂∂
−∂∂
yx f
F
dy
d
f
F
dx
d
f
F
( )
( )
( )2
222
2
222
22
22
22
0
y
ff
dy
d
f
ff
dx
d
f
F
dy
d
x
ff
dx
d
f
ff
dx
d
f
F
dx
d
f
ff
f
F
yy
yx
y
xx
yx
x
yx
∂∂
==∂
+∂=
∂∂
∂∂
==∂
+∂=
∂∂
=∂
+∂=
∂∂
0 :Laplacian get the weand2
2
2
2
=∆=∂∂
+∂∂
fy
f
x
f
![Page 17: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/17.jpg)
Smooth image completion
Ω∂Ω∂Ω=∇∫∫ *2
.. minarg :Lagrange-Euler fftsff
The minimum is achieved when:
Ω∂Ω∂=Ω=∆ * .. 0 fftsoverf
![Page 18: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/18.jpg)
Discrete Approximation (Membrane interpolation)
Ω∂Ω∂=Ω=∆ * .. over 0 fftsf
2
2
2
2
y
f
x
ff
∂∂
+∂∂
=∆
yxyxyxyxyx fffx
fff
x
f,1,,12
2
,,1 2 −++ +−≅∂∂
−≅∂∂
( )04
22,
,1,1,,1,1
1,,1,,1,,1
=−+++=
+−++−≅∆
−+−+
−+−+
yxyxyxyxyx
yxyxyxyxyxyx
fffff
ffffffyxf
![Page 19: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/19.jpg)
Discrete Approximation
=
−
−
−
b
x
x
0
0
0
1
11411
1411
11411
2
1
Each fx,y is an unknown variable xi, there are N unknown (the pixel values)
This reduces to a sparse linear system of equations:
We have A_x * I = 0 A_y * I = 0 A_boundary * I = boundary so We can combine all and get Ax = b
Gradient constraints
Boundary conditions
![Page 20: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/20.jpg)
What’s in the picture?
![Page 21: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/21.jpg)
What’s in the picture?
![Page 22: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/22.jpg)
What’s in the picture?
![Page 23: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/23.jpg)
Editing in Gradient Domain
• Given vector field G=(u(x,y),v(x,y)) (pasted gradient) in a bounded region Ω. Find the values of f in Ω that optimize:
Ω∂∗
Ω∂Ω=−∇∫∫ ffwithGf
f
2min
G=(u,v)
f*
fI
f*
Ω
![Page 24: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/24.jpg)
Intuition - What if G is null?
• 1D:
• 2D:
x1 x2
Ω∂∗
Ω∂Ω=∇∫∫ ffwithf
f
2min
![Page 25: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/25.jpg)
What if G is not null?
• 1D case
Seamlessly paste onto
- Add a linear function so that the boundary condition is respected
- Gradient error is equally distributed all over Ω in order to respect the boundary condition
![Page 26: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/26.jpg)
2D case
From Perez et al. 2003
![Page 27: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/27.jpg)
2D case
From Perez et al. 2003
![Page 28: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/28.jpg)
2D case
![Page 29: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/29.jpg)
Poisson Equation( ) ( )222
:caseour In yyxx GfGfGfF −+−=−∇=
0=∂∂
−∂∂
−∂∂
yx f
F
dx
d
f
F
dx
d
f
F
∂
∂−
∂
∂=
−
∂
∂=
∂∂
∂∂
−∂∂
=
−∂∂
=∂∂
y
G
y
fG
y
f
dy
d
f
F
dy
d
x
G
x
fG
x
f
dx
d
f
F
dx
d
yyy
y
y
xxx
x
x
2
2
2
2
22
22
divGf
y
G
x
Gf
y
G
x
G
y
f
x
f
yx
yx
=∆
∂
∂+
∂∂
=∆
=∂
∂−
∂∂
−∂∂
+∂∂
0 :get weand2
2
2
2
![Page 30: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/30.jpg)
Discrete Approximation (Poisson Cloning)
Ω∂Ω∂=Ω=∆ * .. fftsoverdivGf
2
2
2
2
y
f
x
ff
∂∂
+∂∂
=∆
yxyxyxyxyx fffx
fff
x
f,1,,12
2
,,1 2 −++ +−≅∂∂
−≅∂∂
( )04
22,
,1,1,,1,1
1,,1,,1,,1
=−+++=
+−++−≅∆
−+−+
−+−+
yxyxyxyxyx
yxyxyxyxyxyx
fffff
ffffffyxf
( ) ( ) ( ) ( )1,,,1, −−+−−≅∂
∂+
∂∂
= yxGyxGyxGyxGy
G
x
GdivG yyxx
yx
![Page 31: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/31.jpg)
Alternative Derivation (discrete notation)
=
v
uI
y
x
D
D
−
−
−
−
−
−
−
=
11
11
11
11
11
11
11
xD[ ]*110* −=∂∂x
IxDIx
=∗∂∂
• Let Dx - Toeplitz matrix
![Page 32: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/32.jpg)
2
min x
Iy
D
D
−
uI
v
( ) ( )
=
v
uI T
yTx
y
xTy
Tx DD
D
DDD
• Normal equation:
( ) vuI Ty
Txy
Tyx
Tx DDDDDD +=+
[ ] [ ]*011*110, −=−⇒ flipDNote Tx
![Page 33: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/33.jpg)
Numerical Solution
• Discretize Laplacian
=+≡∇ yTyx
Tx DDDD2 [ ] ∗
−=∗
−+−
010
141
010
1
2
1
121
−
−
−
−
−
−
−
1411
1411
11411
11411
11411
1141
1141
Sparse Toeplitz Matrix
![Page 34: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/34.jpg)
Comments:– A is a sparse.– A is symmetric and can be inverted.
– If Ω is rectangular A is a Toeplitz matrix.– Size of A is ∼NxN.– Impractical to form or store A.– Impractical to invert A
( ) vuI Ty
Txy
Tyx
Tx DDDDDD +=+
bI =A
![Page 35: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/35.jpg)
Iterative Solution: Conjugate Gradient
• Solves a linear system Ax=b (in our case x=I)• A is square, symmetric, positive semi-definite.
• Advantages:– Fast!– No need to store A but calculating Ax – In our case Ax can be calculated using a single
convolution.– Can deal with constraints.
![Page 36: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/36.jpg)
Conjugate Gradient as a minimization problem
• Minimizes
And since A is symmetric
![Page 37: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/37.jpg)
Steepest Descent Method
• Pick gradient direction r(i)
• Find optimum along this direction x(i)+αr(i)
Gradient direction
Energy along the gradient
![Page 38: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/38.jpg)
Behavior of gradient descent
• Zigzag or goes straight depending if we’re lucky– Ends up doing multiple steps in the same direction
![Page 39: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/39.jpg)
Conjugate gradient
• For each step i: – Take the residual d(i)=b-Ax(i) (= -gradient)
– Make it A-orthogonal to the previous ones
– Find minimum along this direction
• Needs at most N iterations.
• Matlab command:
x=cgs(A,b)
A can be a function handle afun
such that afun(x) returns A*x
![Page 40: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/40.jpg)
Solving Poisson equation with boundary conditions
Ω
*ΩS T
( ) ( )ΩΩ
+=+ SDDDDDDDD yTyx
Txy
Tyx
Tx I s.t. ∗∗ ΩΩ
= TI
Ω∇S
• Define a circumscribing square Π=Ω∪Ω*– Let Ω⊂ Π denotes the edited image area.– Let Ω*= Π-Ω denotes the surrounding area.
![Page 41: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/41.jpg)
yTyx
Txk ∂∗∂+∂∗∂=
( ) ∗ΩΩ∗= IIkAI U
Ω
*ΩS T( ) *ΩΩ∗= TSkb U
x=cgs(A,b)
• The above requirements can be expressed as a linear set of equations:
Ω∇S
[ ] [ ]bAI
T
SDDDD
I
IDDDD yyT
xxT
yyT
xxT
=⇒
+=
+⇒
∗∗ Ω
Ω
Ω
Ω
![Page 42: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/42.jpg)
Image stitching
![Page 43: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/43.jpg)
Gradient Domain CompositionGradient Domain Composition
![Page 44: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/44.jpg)
Cut & Paste Cut & Paste Paste in Gradient Domain Paste in Gradient Domain
![Page 45: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/45.jpg)
![Page 46: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/46.jpg)
Another example
![Page 47: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/47.jpg)
Transparent CloningTransparent Cloning
I SΩ Ω∇ = ∇2
S TI Ω Ω
Ω
∇ + ∇∇ = ( )max ,I S TΩ Ω Ω∇ = ∇ ∇
![Page 48: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/48.jpg)
Transparent Cloning
![Page 49: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/49.jpg)
Another example
![Page 50: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/50.jpg)
Another example
![Page 51: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/51.jpg)
Changing local illumination
![Page 52: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/52.jpg)
Defect concealment
![Page 53: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/53.jpg)
High Dynamic Range Compression
Small exposure: Dark inside
![Page 54: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/54.jpg)
High Dynamic Range Compression
Large exposure: Outside Saturated
![Page 55: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/55.jpg)
Manipulate gradients
α is set to 0.1 of average gradient magnitudeβ is set between 0.8 and 0.9
Where the gradient is given by:
![Page 56: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/56.jpg)
High Dynamic Range Compression
Desired Image
![Page 57: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/57.jpg)
Software Tone Mapping
Short Exposure
Long Exposure
High Dynamic Range Compression
![Page 58: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/58.jpg)
Shadow Removal
![Page 59: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/59.jpg)
Color2Grey Algorithm
Optimization:
min Σ Σ ( (gi - gj) - δi,j )2
j=i-µ
If δij == ∆L then ideal image is gOtherwise, selectively modulated by ∆Cij
i
i+µ
![Page 60: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/60.jpg)
Original Color2GreyPhotoshop
Grey
Results
Color2Grey+ Color
![Page 61: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/61.jpg)
Color2Grey+ColorOriginal PhotoshopGrey Color2Grey
![Page 62: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/62.jpg)
Original PhotoshopGrey Color2Grey
![Page 63: Image Editing in the Gradient Domain - Tel Aviv Universityipapps/Slides/lecture06.pdf · Image Editing in the Gradient Domain Shai Avidan ... – Only if they are curl-free ... •](https://reader031.vdocuments.us/reader031/viewer/2022030511/5abafdd67f8b9ab1118c7f88/html5/thumbnails/63.jpg)
Original PhotoshopGrey Color2Grey