convex analysis michael moeller chapter 1 · convex analysis michael moeller basics convexity...
TRANSCRIPT
![Page 1: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/1.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Chapter 1Convex AnalysisNonlinear Multiscale Methods for Image and Signal AnalysisSS 2015
Michael MoellerComputer Vision
Department of Computer ScienceTU Munchen
![Page 2: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/2.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.
![Page 3: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/3.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.
![Page 4: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/4.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Convexity
![Page 5: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/5.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Let us repeat some basics things to talk about
u = arg minu∈Rn
E(u).
Definition
• For E : Rn → R ∪ {∞}, we call
dom(E) := {u ∈ Rn | E(u) <∞}
the domain of E .• We call E proper if dom(E) 6= ∅.
![Page 6: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/6.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Definition: Convex Function
We call E : Rn → R ∪ {∞} a convex function if
1 dom(E) is a convex set, i.e. for all u, v ∈ dom(E) and allθ ∈ [0,1] it holds that θu + (1− θ)v ∈ dom(E).
2 For all u, v ∈ dom(E) and all θ ∈ [0,1] it holds that
E(θu + (1− θ)v) ≤ θE(u) + (1− θ)E(v)
We call E strictly convex, if the inequality in 2 is strict for allθ ∈]0,1[, and v 6= u.
![Page 7: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/7.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Example: Inpainting
u = arg minu∈Rn
∥∥∥∥√(Dxu)2 + (Dy u)2
∥∥∥∥1, such that ui = fi ∀i ∈ I
with index set I of uncorrupted pixels.→ Discuss convexity.
![Page 8: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/8.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Existence
![Page 9: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/9.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
When doesu = arg min
u∈RnE(u)
exist?
• E is lower semi-continuous, i.e. for all u
lim infv→u
E(v) ≥ E(u)
holds.• There exists an α such that
{u | E(u) ≤ α}
is non-empty and bounded.
Proof: Board.
![Page 10: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/10.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Fundamental Theorem of Optimization
If E : Rn → R ∪ {∞} is lower semi-continuous and has anonempty bounded sublevelset, then there exists
u = arg minu∈Rn
E(u)
Remark: For a proper convex function, lower semi-continuity isthe same as the closedness of the sublevelsets.
Examples on the board:• A convex continuous function that does not have a
minimizer• A convex function with bounded sublevelsets that does not
have a minimizer
![Page 11: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/11.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Continuity of Convex Functions
If E : Rn → R ∪ {∞} is convex, then E is locally Lipschitz (andhence continuous) on int(dom(E)).
Proof: Exercise (in 1d)
Board: Considering the interior is important!
Conclusion
If E : Rn → R is convex, then E is continuous.
![Page 12: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/12.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Definition
We call E : Rn → R ∪ {∞} coercive, if all sequences (un)n with‖un‖ → ∞ meet E(un)→∞.
Theorem
If E : Rn → R is convex and coercive, then there exists
u = arg minu∈Rn
E(u).
![Page 13: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/13.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
When isu = arg min
u∈RnE(u)
unique?
Theorem
If E : Rn → R ∪ {∞} is convex, then any local minimum is aglobal minimum. If E is strictly convex, the global minimum isunique.
![Page 14: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/14.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Subdifferential Calculus
![Page 15: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/15.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational ProblemsWhat is an optimality condition for
u = arg minu∈Rn
E(u)?
Definition: Subdifferential
We call
∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0, ∀v ∈ Rn}
the subdifferential of E at u.• Elements of ∂E(u) are called subgradients.• If ∂E(u) 6= ∅, we call E subdifferentiable at E .• By convention, ∂E(u) = ∅ for u 6= dom(E).
Theorem: Optimality condition
Let 0 ∈ ∂E(u). Then u ∈ arg minu E(u).
![Page 16: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/16.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Examples for non-differentiable functions:• The `1 norm.• Functional
E(u) ={
0 if u ≥ 0∞ else.
Subdifferential and derivatives
Let the convex function E : Rn → R ∪ {∞} be differentiable atx ∈ dom(E). Then
∂E(x) = {∇E(x)}.
![Page 17: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/17.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Is any convex E subdifferentiable at x ∈ dom(E)?
Answer: Almost...
Definition: Relative Interior
The relative interior of a convex set M is defined as
ri(M) := {x ∈ M | ∀y ∈ M, ∃λ > 1, s.t. λx + (1− λ)y ∈ M}
Theorem: Subdifferentiability1
If E is a proper convex function and u ∈ ri(dom(E)), then∂E(u) is non-empty and bounded.
1Rockafellar, Convex Analysis, Theorem 23.4
![Page 18: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/18.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Sum rule2
Let E1, E2 be convex functions such that
ri(dom(E1)) ∩ ri(dom(E2)) 6= ∅,
then it holds that
∂(E1 + E2)(u) = ∂E1(u) + ∂E2(u).
Example: Minimize (u − f )2 + ιu≥0(u).
Example: Minimize 0.5(u − f )2 + α|u|.
2Rockafellar, Convex Analysis, Theorem 23.8
![Page 19: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/19.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Chain rule3
If A ∈ Rm×n, E : Rm → R ∪ {∞} is convex, andri(dom(E)) ∩ range(A) 6= ∅, then
∂(E ◦ A)(u) = A∗∂E(Au)
Example: Minimize ‖Au − f‖22.
3Rockafellar, Convex Analysis, Theorem 23.9
![Page 20: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/20.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Summary (without assumptions):
• ∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0,∀v ∈ Rn}
• If E differentiable: ∂E(x) = {∇E(x)}
• Sum rule ∂(E1 + E2)(x) = ∂E1(x) + ∂E2(x)
• Cain rule ∂(E ◦ A)(u) = A∗∂E(Au)
![Page 21: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/21.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
![Page 22: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/22.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
![Page 23: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/23.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV minimization
![Page 24: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/24.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
What is TV again?
For u ∈ Rm×n let us consider the anisotropic total variation
TVa(u) =m∑
i=2
n∑j=1
|ui,j − ui−1,j |+m∑
i=1
n∑j=2
|ui,j − ui,j−1|
For doing math, it is often easier to consider ~ui+m(j−1) = u(i , j)and write
TVa(u) = ‖K~u‖1
for a suitable matrix K that discretizes the gradient.
![Page 25: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/25.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Our problem becomes
u(α) = arg minu∈Rnm
12‖u − f‖2
2 + α‖Ku‖1.
Let us try to apply all the learned theory. The minimizer isobtained at
0 ∈ u(α)− f + αK T q
with q ∈ ∂‖Ku(α)‖1, i.e.
qi
= 1 if (Ku(α))i > 0= −1 if (Ku(α))i < 0∈ [−1,1] if (Ku(α))i = 0
Seems extremely difficult to find...
![Page 26: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/26.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Crazy idea:
minu
12‖u − f‖2
2 + α‖Ku‖1 =minu
12‖u − f‖2
2 + α sup‖q‖∞≤1
〈Ku,q〉
=minu
sup‖q‖∞≤1
12‖u − f‖2
2 + α〈Ku,q〉
Can we exchange min and sup?
![Page 27: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/27.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Saddle point problems4
Let C and D be non-empty closed convex sets in Rn and Rm,respectively, and let S be a continuous finite concave-convexfunction on C × D. If either C or D is bounded, one has
infv∈D
supq∈C
S(v ,q) = supq∈C
infv∈D
S(v ,q).
We can therefore compute
minu
12‖u − f‖2
2 + α‖Ku‖1 =minu
sup‖q‖∞≤1
12‖u − f‖2
2 + α〈Ku,q〉
= sup‖q‖∞≤1
minu
12‖u − f‖2
2 + α〈Ku,q〉.
4Rockafellar, Convex Analysis, Corollary 37.3.2
![Page 28: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/28.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Now the inner minimization problem obtains its optimum at
0 = u − f + αK T q,
⇒u = f − αK T q.
The remaining problem in q becomes
sup‖q‖∞≤1
12‖f − αK T q − f‖2
2 + α〈K (f − αK T q),q〉
= sup‖q‖∞≤1
12‖αK T q‖2
2 + α〈Kf ,q〉 − ‖αK T q‖22
= sup‖q‖∞≤1
−12‖αK T q − f‖2
2
![Page 29: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/29.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Since we prefer minimizations over maximizations, we write
q =arg max‖q‖∞≤1
−12‖αK T q − f‖2
2
= arg min‖q‖∞≤1
12
∥∥∥∥K T q − fα
∥∥∥∥2
2
Idea: Gradient descent + project onto feasible set.
qk+1 = π‖q‖∞≤1
(qk − τK
(K T qk − f
α
))
![Page 30: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/30.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
Gradient projection algorithm5
The algorithm
qk+1 = π‖·‖∞≤1
(qk − τK
(K T qk − f
α
))with uk = f − αqk , for TV minimization converges for τ < 1
4 .
Remark: The 1/4 is two over the Lipschitz constant of thegradient of the smooth objective.
5Levitin, Polyak, Constrained minimization problems, 1966. Goldstein,Convex programming in Hilbert space, 1964.
![Page 31: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/31.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
TV Minimization
1 img = im2double(imread('cameraman.tif'));2 [m,n] = size(img);3
4 e = ones(m,1);5 yDerMat = spdiags([-e e], 0:1, m, m);6 yDerMat(end) =0;7
8 e = ones(n,1);9 xDerMat = spdiags([-e e], 0:1, n, n)';
10 xDerMat(end)=0;11
12 yDer = yDerMat*img;13 xDer = img*xDerMat;14
15 gradientMatrixForVectorizedImage = ...[kron(xDerMat',speye(m,m)); kron(speye(m,m), ...yDerMat)];
16
17 imgGradient = gradientMatrixForVectorizedImage*img(:);18 figure, imagesc(reshape(imgGradient(1:n*m), ...
[n,m])), colorbar
Note that AXB = C ⇔ kron(B′,A)~X = ~C!
![Page 32: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/32.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Duality
![Page 33: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/33.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
We replaced ‖Ku‖1 (not differentiable) with sup‖q‖∞≤1〈q,Ku〉.
Numerics became easier.
Is there a systematic concept behind this idea?
![Page 34: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/34.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Very important concept: Duality!
Definition: Convex Conjugate
We define the convex conjugate of the functionE : Rn → R ∪ {∞} to be
E∗(p) = supu∈Rn
(〈u,p〉 − E(u)) .
Board: E∗ is convex.
![Page 35: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/35.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Examples:
• E(u) = u2 leads to E∗(p) = p2
• E(u) = ‖u‖2 leads to E∗(p) ={
0 if ‖p‖2 ≤ 1,∞ else.
• E(u) = ‖u‖1 leads to E∗(p) ={
0 if ‖p‖∞ ≤ 1,∞ else.
• E(u) = ‖u‖∞ leads to E∗(p) ={
0 if ‖p‖1 ≤ 1,∞ else.
• E(u) ={
0 if ‖u‖2 ≤ 1,∞ else. leads to E∗(p) = ‖p‖2.
• E(u) ={
0 if ‖u‖∞ ≤ 1,∞ else. leads to E∗(p) = ‖p‖1.
• E(u) ={
0 if ‖u‖1 ≤ 1,∞ else. leads to E∗(p) = ‖p‖∞.
Suspicion: E∗∗ = E?
![Page 36: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/36.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Fenchel-Young Inequality6
Let E be proper, convex and lower semi-continuous,u ∈ dom(E) ⊂ Rn, and p ∈ Rn, then
E(u) + E∗(p) ≥ 〈u,p〉.
Equality holds if and only if p ∈ ∂E(u).
Theorem: Biconjugate7
Let E be proper, convex and lower semi-continuous, thenE∗∗ = E .
Now we understand what we did for TV minimization...
6Borwein, Zhu Techniques of variational analysis, Proposition 4.4.17Rockafellar, Convex Analysis, Theorem 12.2
![Page 37: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/37.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
Theorem: Subgradient of convex conjugate8
Let E be proper, convex and lower semi-continuous, then thefollowing two conditions are equivalent:
• p ∈ ∂E(u)• u ∈ ∂E∗(p)
Board: Example with proximity operator.
8Rockafellar, Convex Analysis, Theorem 23.5
![Page 38: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/38.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational Problems
What is this good for? Easier analysis and optimization!
Fenchel’s Duality Theorem9
The following problems yield the same result:
infu H(u) + R(Ku) ”Primal”
infu supq H(u) + 〈q,Ku〉 − R∗(q)”Saddle point”
supq infu H(u) + 〈q,Ku〉 − R∗(q)
supq −H∗(−K ∗q)− R∗(q) ”Dual”
Assuming that H and R are proper, lower semi-continuous,convex functions with ri(dom(H)) ∩ ri(dom(R ◦ K )) 6= ∅.
9C.f. Rockafellar, Convex Analysis, Section 31, or Borwein, Zhu Techniquesof variational analysis, Theorem 4.4.3
![Page 39: Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity Existence Uniqueness The Subdifferential TV minimization Duality updated 07.05.2015 Chapter](https://reader031.vdocuments.us/reader031/viewer/2022022010/5afa26707f8b9a44658ead3f/html5/thumbnails/39.jpg)
Convex Analysis
Michael Moeller
BasicsConvexity
Existence
Uniqueness
The Subdifferential
TV minimization
Duality
updated 07.05.2015
Variational ProblemsConsider the isotropic total variation
TV (u) =m∑
i=2
n∑j=2
√(ui,j − ui−1,j)2 + (ui,j − ui,j−1)2 = ‖Ku‖2,1
How do we minimize
12‖u − f‖2
2 + α‖Ku‖2,1 ?
Dual problem:
12
∥∥∥∥K T q − fα
∥∥∥∥2
2+ (‖ · ‖2,1)
∗(q)
Similar to previous examples:
(‖ · ‖2,1)∗(q) =
{0 if ‖q‖2,∞ ≤ 1 ∀i ,∞ else.
→ Gradient projection algorithm!